Difference between revisions of "UWAR:Tools:IO"

From PublicWiki
Jump to: navigation, search
 
Line 1: Line 1:
The <tt>com.intel.research.uwar.io</tt> package provides a Java library for reading, writing, and processing data streams in the [[UWAR:DataFormat]].  In addition to handling UWAR data at an abstract, sensor-agnostic level, it also provides interfaces for interacting with specific sensors.  The library can be broken down in the following areas:
+
[[Category:UWAR]]
 
 
* ''Streaming'' - Interfaces for reading and writing sensor streams as individual sensor data events
 
* ''Trace Inspector'' - Interfaces for reading and writing sensor streams as aggregate series of sensor data
 
* ''Converter'' - Interfaces for processing streams in pipeline fashion, applying transformations on the sensor data
 
* ''Matlab IO'' - Interfaces for reading sensor data into Matlab-friendly data structures
 
 
 
== Streaming ==
 
 
 
The streaming data interfaces are all defined in the <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/streaming/package-summary.html com.intel.research.uwar.io.streaming]</tt> package.  They provide the ability to handle individual sensor data events as they are encountered in a sensor stream.  A sensor data event is usually something like a discrete sensor reading for a given sensor, such as the reading of a compass at a given point in time or an GPS estimate of location at another.
 
 
 
The streaming data interfaces are roughly inspired by the SAX processing model used for parsing XML streams.  Such a model has the advantage of not requiring all the underlying data to be resident in memory at the same time.  Considering some sensor traces can be many gigabytes in aggregate size, this is an important feature.  As in SAX, the streaming data interfaces allow the program to register various listeners for sensor data events.  They programmer then starts processing the stream, and appropriate data-handlers are called as sensor data is encountered in the trace.
 
 
 
To get a feel for what this all looks like, we consider the following example:
 
 
 
    TraceFactory factory = TraceFactory.getFactory();
 
    InputStream is = new FileInputStream("trace.uwar");
 
   
 
    IInputTrace trace = factory.loadTrace(is);
 
    trace.addListener( new MSBTraceListenerV1() {
 
   
 
        public void traceFloat(String sensorId, long tick, float value) {
 
            System.out.println("MSB: time=" + tick + " sensor=" + sensorId + " value=" + value );
 
        }
 
   
 
        public void traceAudio(long tick, byte[] data) {
 
   
 
        }
 
    });
 
   
 
    trace.run();
 
           
 
We begin by getting an instance of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/streaming/TraceFactory.html TraceFactory]</tt>.  <tt>TraceFactory</tt> provides a number of methods for creating input and output traces.  We specifically call the <tt>loadTrace()</tt> method to loading our trace input stream.  The method returns an instance of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/streaming/IInputTrace.html IInputTrace]</tt>.  You'll notice that the <tt>IInputTrace</tt> interface doesn't define a lot of methods.  It provides a method for querying the UWAR data stream version and an <tt>addListener()</tt> method, which can be used for registering instances of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/streaming/ITraceEventSink.html ITraceEventSink]</tt>.  These are our sensor data event handlers, and there are a number of sub-interfaces for specific sensor data.  For instance, there are interfaces for handling MSB data, wifi, annotations, and other sensor data.  The programmer simply needs implement the handler for the sensor data type they are interested in and add it to the trace.  In the above example, we register a MSB data handler for V1 of the UWAR data format.  When we call the <tt>run()</tt> method on the input trace, it proceeds to sequentially process all sensor data in the trace, firing MSB data events to the registered handler as the data is encountered.
 
 
 
While reading trace data is important, it might be desirable to write data streams as well.  Consider the following example:
 
 
 
    TraceFactory factory = TraceFactory.getFactory();
 
   
 
    IInputTrace inputTrace = factory.loadTrace( new FileInputStream("trace.uwar") );
 
    final IOutputTrace outputTrace = factory.saveTrace(new FileOutputStream("trace-processed.uwar"), SensorConstantsV1.UWAR_STREAM_VERSION);
 
   
 
    final Map packetHandlers = new HashMap();
 
       
 
    inputTrace.addListener(new IHeaderTraceListenerV1() {
 
        public void trace(IHeaderV1 header) {
 
            RawPacketSourceV1 source = new RawPacketSourceV1(header);
 
            outputTrace.addEventSource(source);
 
            packetHandlers.put( new Short(header.getStreamSymbol()),source);
 
        }
 
    });
 
   
 
    inputTrace.addListener(new IPacketTraceListenerV1() {
 
        public void trace(IPacketV1 packet) {
 
            // Insert your data-transformation code here
 
           
 
            // Save the packet to the output stream
 
            RawPacketSourceV1 source = (RawPacketSourceV1) packetHandlers.get( new Short(packet.getStreamSymbol()));
 
            source.trace(packet);
 
        }
 
    });
 
   
 
    // Do run-run-run...
 
    inputTrace.run();
 
    outputTrace.close();
 
 
 
In the example, we use our <tt>TraceFactory</tt> to load an input stream as before, but also an instance of <tt>IOutputStream</tt>, which we'll use for writing an output stream.  Notice that we specify we'll be writing a V1 UWAR data stream.  We register two event-handlers with the input stream using the same mechanism as before.  However, instead of handling specific sensor data, we'll be listening for raw UWAR sensor headers and data packets.  We first register a header listener that create an instance of <tt>RawPacketSourceV1</tt>.  <tt>RawPacketSourceV1</tt> implements the interface <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/streaming/ITraceEventSource.html ITraceEventSource]</tt>, which specifies the hierarchy of classes that produce sensor data (as opposed to consuming it, like our header listener).
 
 
 
Our <tt>RawPacketSource</tt> automatically handles writing a sensor's data header to the output stream as well as writing packets to the stream, as we see above in our packet listener.  Just like we registered data listeners with the input trace, we register data sources with the output trace.  In our case, we register a packet source for each of the sensor streams present in the source stream.  We keep a map of the packet source objects indexed by the stream-symbol of each stream in the input so that we can pull up the appropriate packet source when writing a packet later on.  Finally, after we have registered all our handlers, we run the input trace, setting of a chain of reads on the input stream and writes on the output stream.  Our example code is effectively copying the source stream verbatim into the output stream.  So when the input trace is done processing, the output should contain the copied result and we close the output to finish up the process.
 
 
 
Obviously, this seems a pretty complicated way of copying a stream and is meant mostly as an illustrative example of the APIs.  It's up to the programmer to insert meaningful data transformations into the pipeline.  Since a number of these transformations are quite common, we have standardized them in the [[#Converter|Converter]] interfaces described below.
 
 
 
== Trace Inspector ==
 
 
 
While the streaming interfaces described above might be sufficient for basic data input and output in a piece-wise fashion, there will be occasions when one wishes to work with full blocks of sensor data at the same time.  Just as the streaming interfaces were inspired the SAX processing model for XML data, the Trace Inspector interfaces were inspired by the DOM model for XML data.  Just like SAX-vs-DOM, the Trace Inspector interfaces allow for reading full portions of stream into memory for aggregate processing.
 
 
 
While the stream interfaces provide listeners for handling individual sensor types, the Trace Inspector attempts to abstract sensor data into common interfaces that can be processed in consistent ways.  The core of this abstraction is the <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/traceinspector/ISensor.html ISensor]</tt> interface, which is modeled directly on a sensor stream in a trace.  Each sensor is identified by a unique string id (eg. <tt>msb/x_accel</tt> or <tt>wifi/00:06:05:00:0f:0c</tt>.  Each sensor also includes a <tt>getRelation()</tt> method, which returns an instance of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/traceinspector/relations/ITimeRelation.html ITimeRelation]</tt>.  A <tt>ITimeRelation</tt> is basically a collection of data points sorted against a time access.  It might be a series of doubles for acceleration readings or points for a GPS sensor.  In any case, here we see the promised ''all the data at once'' features of the Trace Inspector interfaces.
 
  
How do we load instances of <tt>ISensor</tt> from a trace? Let's consider yet another example:
+
The <tt>edu.washington.cs.uwar.io</tt> package provides a Java library for reading, writing, and processing data streams in the [[UWAR:DataFormat]]. The library provides methods for handling UWAR data at an abstract, sensor-agnostic level.
  
    TraceInspectionFactory factory = TraceInspectionFactory.getFactory();
+
Check out the <tt>edu.washington.cs.uwar.io</tt> project from UWAR CVS (see [[UWAR#Development]]) for more info.
    ITraceInspector inspector = factory.createInspector(new File("trace.uwar"));
 
   
 
    // Examine available sensors
 
    inspector.run();
 
   
 
    // Our trace happens to contain fifteen wifi sensors.  We can list them:
 
    String[] ids = inspector.getAllSensorIds();
 
    for( String id : ids )
 
        System.out.println(id);
 
   
 
    // Let's examine one
 
    ISensor sensor = inspector.getSensorById("wifi/00:0F:24:DC:B9:60");
 
   
 
    // By default, data collection is not performed for a sensor, allowing one
 
    // to first determine what sensors are available in a trace and then load
 
    // data only for the relevant sensors
 
    sensor.enableDataCollection(true);
 
   
 
    // Re-run the inspector to load the data for our sensor
 
    inspector.run();
 
   
 
    // We can now examine the data for the sensor
 
    ITimeRelation relation = sensor.getRelation();
 
    SortedMap data = relation.getData();
 
    for( Iterator it = data.entrySet().iterator(); it.hasNext(); ) {
 
        Map.Entry entry = (Map.Entry) it.next();
 
        Long timestamp = (Long) entry.getKey();
 
        Integer rssi = (Integer) entry.getValue();
 
        System.out.println("wifi: time=" + timestamp + " rssi=" + rssi );
 
    }
 
 
 
In our example, we use a factory class for creating an instance of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/traceinspector/ITraceInspector.html ITraceInspector]</tt> for our given trace.  Trace Inspection generally works best with traces in file form.  While the streaming interfaces described above can handle input streams, only a file can be repeatedly examined, as is required by the trace inspector.
 
 
 
Once we have our instance of the inspector, we call its <tt>run</tt> method each time we wish to perform an inspection.  You will likely run the inpsector at least once to determine what sensors are available in the trace.  Subsequent runs can be used to load data for specific sensors or specific time intervals.  In our example, we enable data collection for a secific wifi-access-point sensor.  When we re-inspect the trace, the sensor's data is now properly loaded into as a relation.  We can iterate over the data as demonstrated.
 
 
 
The <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/traceinspector/ITraceInspector.html ITraceInspector]</tt> and <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/traceinspector/ISensor.html ISensor]</tt> interfaces provider a number of additional methods for querying the data in a trace and in each sensor.
 
 
 
== Converter ==
 
 
 
Defines a generally framework for applying transformations to UWAR data streams. The main interface is <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/conversion/IConverter.html IConverter]</tt>, which accepts an <tt>IInputTrace</tt> and an <tt>IOutputTrace</tt> as an argument and applies some transformation on the input stream and saves it to the output stream.  This API is not well developed yet, but there are some example converters that convert traces from UWAR V1 to V2 and others examples.
 
 
 
== Matlab IO ==
 
 
 
For real data processing, a heavy duty tool is more appropriate than regular Java code.  As such, we provide methods for loading UWAR sensor data into Matlab data structures.  The loader code is still written in Java and is basically a wrapper around the [[#Trace Inspector|Trace Inspector]] interfaces.  Matlab has nicely-integrated Java support, which makes loading UWAR data pretty easy, no matter which OS you are running Matlab under.  Consider the following example, where we load a trace and examine the sensors contained within as well as the time interval of contained data.
 
 
 
    >> mio = com.intel.research.uwar.io.matlab.MatlabIO.load('Trace-2005_03_23-11_58_46.uwar')
 
    >> mio.getSensorIds()
 
   
 
    ans =
 
   
 
    java.lang.String[]:
 
   
 
        'latlon'
 
        'msb/amb_light'
 
        'msb/audio'
 
        'msb/bar_press'
 
        ...
 
   
 
    >> mio.getTraceTimeInterval()
 
   
 
    ans =
 
   
 
          732034
 
        1077518
 
 
We now consider loading data from the trace. We wish to create matrix of data for the three accelerometers annotated with timestamps of the readings. Both will be over the full time interval of the trace. The following call to 'mio.getData()' achieves the desired result, with an array of sensor ids being passed as the first argument. Note how the sensor ids for the three accelerometer traces are combined into one comman-delimited string, indicating that the data for these sensors should be interleaved together.
 
 
 
    >> accel_result = mio.getData('msb/x_accel,msb/y_accel,msb/z_accel');
 
 
The call to 'mio.getData()' returns an <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/matlab/ISensorDataTable.html ISensorDataTable]</tt> results containing the accelerometer data.  The data table contains four instances of <tt>[http://ubicomp.cs.washington.edu/uwar/apis/javadoc/com/intel/research/uwar/io/matlab/ISensorData.html ISensorData]</tt> data entries: one for the timestamps (<tt>getTimestamps()</tt>) and three for the accelerometers (<tt>getSensorData()</tt>). The ISensorData.getSensorId() and ISensorData.getSensorField() methods can be used to interrogate the source of a given sensor data set, whilte the ISensorData.getData() method finally gets us the actual data.
 
 
 
    >> accel_result.getTimestamps().getSensorId()
 
   
 
    ans =
 
   
 
    timestamp
 
   
 
    >> accel_result.getTimestamps().getData()
 
   
 
    ans =
 
   
 
          735838
 
          735934
 
          735935
 
          736023
 
            ...
 
        1077501
 
        1077502
 
        1077507
 
        1077516
 
        1077518
 
 
 
In the following code snippets, we create one large matrix with the first column containing timestamps and the remaining three colums containing the accelerometer data:
 
 
 
    >> accel = accel_result.getSensorData();
 
    >> accel(1).getSensorId()
 
   
 
    ans =
 
   
 
    msb/x_accel
 
   
 
    >> x_accel = accel(1).getData();
 
    >> y_accel = accel(2).getData();
 
    >> z_accel = accel(3).getData();
 
    >> all_accel = [ts, x_accel, y_accel, z_accel ];
 
    >> size(all_accel)
 
   
 
    ans =
 
   
 
          13953          4
 
 
 
We've successfully produced our 4-column matrix with a row for each sensor reading, ready for processing.  Let's now consider a result for a sensor with multiple columns of data.  In our previous example, each sensor only have one vector of data, but other sensors might have multiple vectors of readings, such as a GPS location reading (lat,lon,altitude,etc.).
 
 
 
    >> latlon_result = mio.getData('latlon');
 
    >> data = latlon_result.getSensorData()
 
   
 
    data =
 
     
 
    com.intel.research.uwar.io.matlab.ISensorData[]:
 
        [com.intel.research.uwar.io.matlab.SensorDataImpl]
 
        [com.intel.research.uwar.io.matlab.SensorDataImpl]
 
     
 
    >> timeVsLatLon = [ latlon_result.getTimestamps().getData(), data(1).getData(), data(2).getData() ];
 
    >> timeVsLatLon
 
   
 
    timeVsLatLon =
 
   
 
        1.0e+06 *
 
   
 
        0.7410    0.0000  -0.0001
 
        0.7623    0.0000  -0.0001
 
        0.7689    0.0000  -0.0001
 
        1.0634    0.0000  -0.0001
 
        1.0675    0.0000  -0.0001
 
        1.0758    0.0000  -0.0001
 
   
 
    >> timeVsLatLon(1,2:3)
 
   
 
    ans =
 
   
 
        47.6534 -122.3054
 
 
 
[[Category:UWAR]]
 

Latest revision as of 07:10, 1 November 2006


The edu.washington.cs.uwar.io package provides a Java library for reading, writing, and processing data streams in the UWAR:DataFormat. The library provides methods for handling UWAR data at an abstract, sensor-agnostic level.

Check out the edu.washington.cs.uwar.io project from UWAR CVS (see UWAR#Development) for more info.