IDE File Basics =============== What's an IDE file? ------------------- An IDE file is a read-only, `EBML `_-based hierarchical data format that stores recording information generated by an enDAQ sensor device. It contains both time-indexed data from several different kinds of sensors (like acceleration, pressure, temperature, etc.), as well as metadata about the recording device (like device serial number, model number, device name, etc.) and recording settings. .. _dataset_desc: Accessing an IDE file --------------------- The top-level interface for an IDE file is the :class:`Dataset` object, through which one can access all of the above-listed information. When you open a file for reading, for example, this is the type of object that is returned. Opening an IDE File ~~~~~~~~~~~~~~~~~~~ You can open an IDE file like so:: filename = "/path/to/your/file.IDE" with idelib.importFile(filename) as ds: print(type(ds)) Note: a :class:`Dataset` object perfoms *lazy-loading*, meaning that it only loads information as is needed. As a result, it internally retains a handle to the source file which needs to be closed after use. This can be accomplished by either: * using :class:`Dataset` as a *context manager* (as seen above; this is the recommended method), or * by using :class:`Dataset` as a normal object and calling the ``close()`` method manually:: filename = "/path/to/your/file.IDE" ds = idelib.importFile(filename) # use `ds` here ds.close() # remember to close the file after use! Getting recording data ---------------------- Channels & subchannels ~~~~~~~~~~~~~~~~~~~~~~ IDE files organize recording data into *channels* and *subchannels*. A channel encapsulates data recorded by a particular individual sensor on the device (e.g., XYZ acceleration from the ADXL375 DC Accelerometer); a subchannel, if present, specifies a particular data stream within a channel (e.g., the X-coordinate acceleration from the ADXL375 DC Accelerometer). At the top-level, a :class:`Dataset` object has a ``channels`` member, which is a dict of all channels recorded in the file. The dict is keyed by channel ID numbers, with :class:`Channel` objects in the values. Each :class:`Channel` object has a ``subchannels`` member, which is a list of :class:`Subchannel` objects. If the channel has no subchannels, this member will be ``None``. The channels, channel ID numbers and subchannels that may appear in a given recording file depend on the physical sensors available on the recording device, which are indicated by the device's *product number*. Below are some product number abbreviations used herein: ========================= ============================================================================================== ======================= (Abbreviated) Product No. Description Example Product Nos. ========================= ============================================================================================== ======================= S/W-D enDAQ S-series & W-series devices with a single digital accelerometer S3-D16, W5-D40 ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- S/W-DD enDAQ S-series & W-series devices with dual digital accelerometers S1-D100D40, S2-D25D16 ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- S/W-ED enDAQ S-series & W-series devices with an analog electrocapacitive and a digital accelerometer W8-E25D40, S4-E100D40 ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- S/W-RD enDAQ S-series & W-series devices with an analog piezoresistive and a digital accelerometer S4-R500D40, W8-R2000D40 ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- SSX Mide Slam Stick X data recorders SSX ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- SSC Mide Slam Stick C data recorders SSC ------------------------- ---------------------------------------------------------------------------------------------- ----------------------- SSS Mide Slam Stick S data recorders SSS ========================= ============================================================================================== ======================= The below table lists channel ID numbers used in a recording file based on the recording device's product number (device series numbers and accelerometer sensitivity ranges are omitted when applicable to all such devices): ===================== ========== ======================================== ===================================== Sensor Channel ID Valid Devices Subchannels ===================== ========== ======================================== ===================================== Main Accelerometer 8 S/W-RD, S/W-ED, SSS, SSX X-, Y-, Z-axis Acceleration --------------------- ---------- ---------------------------------------- ------------------------------------- 16/200g Accelerometer 32 S/W-DD, SSX, SSS, SSC, S/W-D16, S/W-D200 X-, Y-, Z-axis Acceleration --------------------- ---------- ---------------------------------------- ------------------------------------- 8/40g Accelerometer 80 S/W-RD, S/W-DD, S/W-ED, S/W-D40, S/W-D8 X-, Y-, Z-axis Acceleration --------------------- ---------- ---------------------------------------- ------------------------------------- IMU Gyroscope 47 All [1]_ X-, Y-, Z-axis Rotation --------------------- ---------- ---------------------------------------- ------------------------------------- Absolute Orientation 65 All [1]_ X-, Y-, Z-, W-axis Quaternion; Acc --------------------- ---------- ---------------------------------------- ------------------------------------- Relative Orientation 70 All [1]_ X-, Y-, Z-, W-axis Quaternion --------------------- ---------- ---------------------------------------- ------------------------------------- MPL3115 36 All [1]_ Pressure, Temperature [2]_ --------------------- ---------- ---------------------------------------- ------------------------------------- MS8607 59 All [1]_ Pressure, Temperature, Humidity [3]_ --------------------- ---------- ---------------------------------------- ------------------------------------- SI1133 76 All [1]_ Lux, UV ===================== ========== ======================================== ===================================== .. [1] excluding early SSC/SSS/SSX models .. [2] 1 Hz Internal Measurements .. [3] 10 Hz Control Pad Measurements To simply use all recording data, we can iterate through each subchannel in a dataset like so:: for ch in ds.channels.values(): for sch in ch.subchannels: print(sch) .. _eventarray_desc: EventArrays & raw data ~~~~~~~~~~~~~~~~~~~~~~ Each :class:`Channel` and :class:`SubChannel` object has a ``getSession()`` method, which returns an :class:`EventArray` object. :class:`EventArray` is a wrapper around a channel's underlying recording data that loads data on demand from the source file. You can index an :class:`EventArray` (e.g., ``eventarray[i]`` for some index ``i``) to get a numpy ``ndarray`` of data. Data is organized in an n-dimensional array. For subchannels, this will always be a 2-by-n array, where n is the number of samples recorded; ``eventarray[1]`` indexes the samples, ``eventarray[0]`` indexes the respective timestamps. For channels, this will be a (c+1)-by-n array, where n is the number of samples recorded and c is the number of subchannels; ``eventarray[1:]`` indexes the samples, ``eventarray[0]`` indexes the respective timestamps. Getting metadata ---------------- .. testsetup:: * import pathlib import idelib ds = idelib.importFile(pathlib.Path("testing/test3.IDE")) ds.filename = '/home/enDAQ/recordings/test.IDE' eventarray = ds.channels[80].subchannels[0].getSession() :class:`Dataset` makes available some basic metadata. Some useful pieces of information are stored directly as members: .. doctest:: >>> ds.filename '/home/enDAQ/recordings/test.IDE' Other data is stored in the dict member ``recorderInfo``: .. doctest:: >>> ds.recorderInfo['RecorderSerial'] 10118 >>> ds.recorderInfo['PartNumber'] 'W8-E100D40' :class:`EventArray` also stores some sample-specific metadata, like the data's units: .. doctest:: >>> eventarray.units ('Acceleration', 'g') .. testcleanup:: ds.close()