# Bundle Data In order to provide processing reliability and consistent data organization for multi-flight time-series datasets, GRYFN Processing Tool incorporates a data bundling process which organizes supplied data into a standard structure with metadata. These bundled datasets (denoted with a “.graw” folder naming convention) contain all the specified raw data, system calibration, and a processing extent (if supplied). In addition to the required processing inputs, the bundler also optionally saves information on pilot, location, and other parameters. GRAW bundles are the input to a processing pipeline. In the Bundle Data tab, the user can choose to create a new bundle, or add data to an existing GRAW bundle. GRAW data bundles make no changes to raw data files, only adding additional metadata files and altering files created directly through the GRYFN Processing Tool (such as targets.yaml, elm\_coefficients.json, etc.). Therefore, GRAW bundles are a definitive archival source for raw data, allowing consistency across datasets. ## Inputs A table describing all inputs to the data bundler is provided below.
NameTypeFormatRequired
System CalibrationGRYFN System CalibrationYAMLYes
Processing ExtentProcessing ExtentGeoJSON/KML/SHPNo
GNSSRaw SBG/APX GNSS-INS data logs, and/or Custom GRYFN Export00#/T04/TXT*, **
LiDARVLP/OS LiDAR Raw DataPCAP*, ***
RGB3-Band ImageryJPG/TIFF*
VNIRHeadwall/Specim Data FolderENVI*
SWIRHeadwall Data FolderENVI*
LogsHardware logs and misc. filesAnyNo
PilotUAV pilot informationText/YAMLNo
LocationMission location informationText/YAMLNo
ConditionsWeather and other notesText/YAMLNo
\* At least one sensor type is required to bundle \*\* SBG data processing also requires the .json metadata file \*\*\* Ouster data processing also requires the .json metadata file ### System Calibration GRYFN system calibration files are a human readable YAML file specifying a unique identifier (id) for each sensor with various parameters that define that sensor’s type, intrinsic optical characteristics, and extrinsic position and orientation relative to a common trajectory reference frame. This calibration file may also contain optional information such as sensor model, serial number, and calibration date. An example of GRYFN’s System Calibration file structure with examples for each sensor type and definitions of the standard parameters is given below. ``` model: Mojave serial: GMOJ2301 sensors: - id: Sony RX1RM2 model: DSC-RX1RM2 serial: 6314615 type: RGB date: 2023-07-21 lever_arm: x: 0.216 y: -0.049 z: -0.008 boresight: omega: 180.358151 phi: 0.897287 kappa: -90.345798 camera: type: frame model: SMAC xp: 11.715605 yp: -6.0726024 c: 7556.16 image_width: 7952 image_height: 5304 k1: -1.1199271e-09 k2: -2.7774539e-17 p1: 3.7102533e-08 p2: -1.2328174e-07 - id: VLP16 Hi-Res model: Puck Hi-Res serial: 11406210245816 type: LiDAR date: 2023-07-21 lever_arm: x: -0.093 y: -0.042 z: 0.006 boresight: omega: 1.918458 phi: -0.077572 kappa: 0.403461 nominal_boresight: omega: 90.0 phi: -90.0 kappa: 0.0 - id: nHS-206 model: Nano-Hyperspec type: VNIR date: 2023-07-21 timedelay0: 0.0 lever_arm: x: 0.136 y: -0.039 z: 0.016 boresight: omega: 0.461230 phi: 0.608521 kappa: 0.299187 camera: type: pushbroom pixel_size: 0.0074 f: 8.256 image_width: 640 - id: APX-15 model: APX-15Lv3 serial: 6035C00140 type: GNSS ``` * **id**: Unique sensor ID string * **model**: Manufacturer sensor model * **type**: RGB, LiDAR, VNIR, SWIR, or GNSS * **date**: Date of data collection used in calibration process. * **time\_delay0**: Offset in seconds to apply from trajectory to imagery. Only used for certain legacy hardware solutions. * **lever\_arm**: Offset in meters from trajectory reference point to sensor origin. X, Y, and Z are respectively forward, right, and down. * **boresight**: Rotation in degrees between trajectory coordinate system and sensor coordinate system. * **nominal\_boresight**: Nominal rotation in degrees between trajectory coordinate system and sensor coordinate system. Omega, phi, and kappa are rotations around X, Y, and Z respectively. * **camera**: Parameters defining the intrinsic properties of optical sensors such as RGB frame cameras and hyperspectral line scanners. * **type**: frame or pushbroom * **model**: camera IOP model * **xp**: perspective center offset in sensor X direction * **yp**: perspective center offset in sensor Y direction * **c/f**: focal length in pixels(frame) or mm(pushbroom) * **image\_width**: width of image sensor in pixels * **image\_height**: height of image sensor in pixels * **pixel\_size**: size of pixel in mm ### Raw Data Source The GRYFN bundling process allows two methods of navigation to raw data products to be bundled, single-source and multi-source. In single-source mode, the bundler will parse the input directory and its’ child directories for all possible data types. For example, all files with a “.pcap” extension will be organized into a folder for the sensor id with type “LiDAR” and a Headwall VNIR data folder and its’ associated dark data folder (if available) will be copied into the GRAW in a folder for the sensor id with type “VNIR”. Multi-source bundling allows the user to specify individual source directories for each data type. Data source directories can be added on a per-flight basis. This allows GRAWs to contain data from multiple UAV flights. This feature is used when, for example, a large field requires multiple individual flights to obtain a complete dataset. These individual flight datasets, or flights, can be bundled together and processed simultaneously for a combined output. {% hint style="warning" %} The multi-flight bundling feature is not meant to contain time-series data of the same area. {% endhint %} {% hint style="danger" %} Bundling is not currently supported for multiple sensors of the same type. {% endhint %} A detailed description of the required files for each data type is given below.
NameType
APX-15Directory containing export “export*.txt”, “event*.dat” (if frame camera), “.t02” and “.t04” file backups, and directories named "project"
Quanta MicroAny directory containing a file named "lastProcessing.json", a directory named "geodesy", or files with the ".0*" extension.
LiDARFiles with extension ".pcap” (data) or files matching "os-*.json" (Ouster metadata)
RGBAll images with “.jpg” or “.tif” extension
VNIR (Headwall)Directory containing VNIR acquisition data folder and a dark data folder (may contain raw or radiance data with default Headwall file names and extensions “_rd” but no additional files/folders)
SWIR (Headwall)Directory containing SWIR acquisition data folder and a dark data folder (may contain raw, radiance, or nuc data with default Headwall file names and extensions “rd_rdk" | "_rd" | “_nuc” but no additional folders)
VNIR (Specim)Directory containing a "manifest.xml" file.
LogsDirectory containing any miscellaneous files to add to bundle logs directory
#### Radiometric Calibration For Headwall Nano HP and SWIR data, GRYFN Processing Tool can automatically convert raw DNs to radiance data on the fly during processing when the [Default Radiometric Calibration Location](/gryfn-software/user-manual/settings.md#default-radiometric-calibration-location) is setup correctly. Using sensor serial numbers located in the bundled SystemCalibration file, GRYFN Processing Tool will automatically search the **Default Radiometric Calibration Location** for matching serial numbers. Radiometric calibration files are provided for all combinations of gain mode and exposure level. GRYFN Processing Tool parses the settings.txt file from the raw data source to appropriately select the necessary radioCal files for each supported sensor to add to the GRAW bundle based on gain mode and exposure time. ### GNSS Data The GRYFN bundling and processing pipeline accepts both post-processed GNSS-INS data in the form of a custom text file and raw GNSS data from SBG Quanta Micro or Applanix APX-15 sensors. The custom export file contains a necessary header block that specifies the location and filename of any related .dat event timetag files, UTC time offset and week number, and mapping frame information. A data column header is also given. An example header and five lines of trajectory data are given below. ``` GRYFN Export Trajectory v0.3 Project Info: Project File: Event Timetag File: C:\Users\GRYFN\Documents\POSPac UAV\Unnamed(1)\Mission 1\Extract\event1_Mission 1.dat Reference to Primary GNSS Lever Arm: X: 0.045 Y: -0.080 Z: -0.340 Reference to IMU Mounting Angles: X: -90.000 Y: 90.000 Z: 0.000 UTC Mission Start Date: 20210630 Wednesday, June 30, 2021 UTC Mission Start Time: 16h:09m:00s UTC Time Offset: 18s GPS Week: 2164 Mapping Fame Info: Datum: WGS84 Grid: Universal Transverse Mercator Zone: UTM North 15 (96W to 90W) Ellipsoid: WGS84 Target Epoch: 2021.493151 GPS_TIME(SoD) EASTING(m) NORTHING(m) ELLIPSOID_HEIGHT(m) ROLL(deg) PITCH (deg) HEADING(deg) EASTING_SD(m) NORTHING_SD(m) HEIGHT_SD(m) ROLL_SD(deg) PITCH_SD(deg) HEADING_SD(deg) ANGLERATE_X ANGLERATE_Y ANGLERATE_Z 58631.009323 652307.123299 4860894.206837 198.803703 -0.842261 -13.399428 265.182502 0.034731 0.032690 0.059805 0.167360 0.192645 0.522946 0.091385 0.020312 -0.011949 58631.014323 652307.088036 4860894.202650 198.803688 -0.819643 -13.314868 265.504095 0.034722 0.032695 0.059787 0.167342 0.192549 0.522770 0.075679 0.035670 0.004107 58631.019323 652307.015917 4860894.247530 198.867820 -0.807195 -13.295999 265.751157 0.034713 0.032700 0.059769 0.167324 0.192453 0.522595 0.049151 0.062895 -0.021024 58631.024323 652306.980530 4860894.243746 198.867781 -0.785800 -13.277930 265.754958 0.034705 0.032704 0.059751 0.167305 0.192357 0.522419 0.077773 0.061499 0.013531 58631.029324 652306.945127 4860894.239966 198.867734 -0.782700 -13.255140 265.744932 0.034696 0.032709 0.059733 0.167287 0.192261 0.522244 0.002727 0.078603 -0.033241 ``` ### Logs In addition to raw data, the bundler can also bundle an additional user-specified directory in the GRAW. All files in this user-specified log directory will be copied to the GRAW on a per-flight basis. The logs directory should be a directory containing extra files (and no sub-directories) that you want to bundle with the raw data. ### Optional Parameters Each GRAW contains a “mission\_data.yaml” metadata file. This file contains additional information about each bundle including the calibration file name, data checksums, and sensor types. Optional information such as UAV pilot credentials, location, and conditions can be added to the mission\_data.yaml at the time of bundle submission. #### Processing Extent Processing extent files are geospatial files containing a polygon feature which can be used to trim output data products to a limited area. Using these processing extents will remove unwanted data, reduce the amount of space needs for product storage, and reduce processing times. An example extent file in GeoJSON is given below. KML and SHP files are supported as well. When Headwall data is collected, the HSI Polygon from the gpsMonitor file can also be used via the VNIR Processing Extent pipeline product option. {% hint style="info" %} If multiple polygons are present: * GeoJson, KML, SHP: The first polygon listed will be used as the processing extent * gpsMonitor.json: Area for each polygon will be calculated and the larger polygon area will be selected as the processing extent. {% endhint %} ``` { "type": "Feature", "geometry": { "type": "Polygon", "coordinates": [ [[-86.995222417348046, 40.471169271289895], [-86.995222471001114, 40.470412174363126], [-86.994041227816268, 40.470412119503784], [-86.994041160897481, 40.471169216429075], [-86.995222417348046, 40.471169271289895]] ] }, "properties": { "name": "Name" } } ``` ## Output When creating a GRAW bundle, the user must specify a name and output directory location. Later in processing, product names can be prepended with the GRAW name, and the processing directory (GPRO) can be stored in the same directory alongside the GRAW. The GRAW output structure is organized with system calibration, processing extent, mission metadata, and a copy of the bundling pipeline in the GRAW root directory. Sensor data from each unique sensor id is organized by flight inside each sensor id folder. Log data is organized by flight into a common “logs” folder. Upon submission of a processing job, several additional files may be added: * SBG Quanta Micro or APX-15 processing folders containing: * Project data * Geodesy information * **export\_\*.txt** SBET in custom TXT format * **event\_\*.txt** Event output in custom TXT format * When hyperspectral reflectance target selection occurs: * **Targets folder** containing the selected DHR target calibration files * **targets.yaml** contains information regarding which pixels have been selected for each target and where the target file locations are. In the GRYFN processing GUI, pre-selected target pixels may be re-loaded upon selecting a GRAW for processing which contains this “targets.yaml” file. * **elm\_coefficients.json** containing ELM data for reflectance conversion --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://gryfn.gitbook.io/gryfn-software/user-manual/bundle-data.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.