Bundle Data
In order to provide processing reliability and consistent data organization for multi-flight time-series datasets, GRYFN Processing Tool incorporates a data bundling process which organizes supplied data into a standard structure with metadata. These bundled datasets (denoted with a “.graw” folder extension) contain all the specified raw data, system calibration, and a processing extent (if supplied). In addition to the required processing inputs, the bundler also optionally saves information on pilot, location, and other parameters. GRAW bundles are the input to a processing pipeline. In the Bundle Data tab, the user can choose to create a new bundle, or add data to an existing GRAW bundle.
Inputs
A table describing all inputs to the data bundler is provided below.
System Calibration
GRYFN System Calibration
YAML
Yes
Processing Extent
Processing Extent
GeoJSON/KML/SHP
No
GNSS
Custom GRYFN Export
TXT
*
LiDAR
VLP/OS LiDAR Raw Data
PCAP
*, **
RGB
3-Band Imagery
JPG/TIFF
*
VNIR
Headwall/Specim Data Folder
ENVI
*
SWIR
Headwall Data Folder
ENVI
*
Logs
Hardware logs and misc. files
Any
No
Pilot
UAV pilot information
Text/YAML
No
Location
Mission location information
Text/YAML
No
Conditions
Weather and other notes
Text/YAML
No
* At least one sensor type is required to bundle
** Ouster data processing also requires the .json metadata file
System Calibration
GRYFN system calibration files are a human readable YAML file specifying a unique identifier (id) for each sensor with various parameters that define that sensor’s type, intrinsic optical characteristics, and extrinsic position and orientation relative to a common trajectory reference frame. This calibration file may also contain optional information such as sensor model, serial number, and calibration date. An example of GRYFN’s System Calibration file structure with examples for each sensor type and definitions of the standard parameters is given below.
id: Unique sensor ID string
model: Manufacturer sensor model
type: RGB, LiDAR, VNIR, SWIR, or GNSS
date: Date of data collection used in calibration process.
time_delay0: Offset in seconds to apply from trajectory to imagery.
lever_arm: Offset in meters from trajectory reference point to sensor origin. X, Y, and Z are respectively forward, right, and down.
boresight: Rotation in degrees between trajectory coordinate system and sensor coordinate system.
nominal_boresight: Nominal rotation in degrees between trajectory coordinate system and sensor coordinate system. Omega, phi, and kappa are rotations around X, Y, and Z respectively.
camera: Parameters defining the intrinsic properties of optical sensors such as RGB frame cameras and hyperspectral line scanners.
type: frame or pushbroom
model: camera IOP model
xp: perspective center offset in sensor X direction
yp: perspective center offset in sensor Y direction
c/f: focal length in pixels(frame) or mm(pushbroom)
image_width: width of image sensor in pixels
image_height: height of image sensor in pixels
pixel_size: size of pixel in mm
Processing Extent
Processing extent files are geospatial files containing a polygon feature which can be used to trim output data products to a limited area. Using these processing extents will remove unwanted data, reduce the amount of space needs for product storage, and reduce processing times. An example extent file in GeoJSON is given below. KML and SHP files are supported as well. When Headwall data is collected, the HSI Polygon from the gpsMonitor file can also be used via the VNIR Processing Extent pipeline product option.
Raw Data Source
The GRYFN bundling process allows two methods of navigation to raw data products to be bundled, single-source and multi-source. In single-source mode, the bundler will parse the input directory and its’ child directories for all possible data types. For example, all files with a “.pcap” extension will be organized into a folder for the sensor id with type “LiDAR” and a Headwall VNIR data folder and its’ associated dark data folder (if available) will be copied into the GRAW in a folder for the sensor id with type “VNIR”.
Multi-source bundling allows the user to specify individual source directories for each data type.
Data source directories can be added on a per-flight basis. This allows GRAWs to contain data from multiple UAV flights. This feature is used when, for example, a large field requires multiple individual flights to obtain a complete dataset. These individual flight datasets, or flights, can be bundled together and processed simultaneously for a combined output.
The multi-flight bundling feature is not meant to contain time-series data of the same area.
Bundling is not currently supported for multiple sensors of the same type.
A detailed description of the required files for each data type is given below.
APX-15
Directory containing export “export*.txt”, “event*.dat” (if frame camera), “.t02” and “.t04” file backups, and directories named "project"
Quanta Micro
Any directory containing a file named "lastProcessing.json", a directory named "geodesy", or files with the ".0*" extension.
LiDAR
Files with extension ".pcap” (data) or files matching "os-*.json" (Ouster metadata)
RGB
All images with “.jpg” or “.tif” extension
VNIR (Headwall)
Directory containing VNIR acquisition data folder and a dark data folder (may contain raw or radiance data with default Headwall file names and extensions “_rd” but no additional files/folders)
SWIR (Headwall)
Directory containing SWIR acquisition data folder and a dark data folder (may contain raw, radiance, or nuc data with default Headwall file names and extensions “rd_rdk" | "_rd" | “_nuc” but no additional folders)
VNIR (Specim)
Directory containing a "manifest.xml" file.
Logs
Directory containing any miscellaneous files to add to bundle logs directory
Radiometric Calibration
For Headwall Nano HP and SWIR data, GRYFN Processing Tool can automatically convert raw DNs to radiance data on the fly during processing when the Default Radiometric Calibration Location is setup correctly.
Using sensor serial numbers located in the bundled SystemCalibration file, GRYFN Processing Tool will automatically search the Default Radiometric Calibration Location for matching serial numbers. Radiometric calibration files are provided for all combinations of gain mode and exposure level. GRYFN Processing Tool parses the settings.txt file from the raw data source to appropriately select the necessary radioCal files for each supported sensor to add to the GRAW bundle.
GNSS Data
The GRYFN bundling and processing pipeline accepts both post-processed GNSS-INS data in the form of a custom text file and raw GNSS data from SBG Quanta Micro or Applanix APX-15 sensors.
The custom export file contains a necessary header block that specifies the location and filename of any related .dat event timetag files, UTC time offset and week number, and mapping frame information. A data column header is also given. An example header and five lines of trajectory data are given below.
Logs
In addition to raw data, the bundler can also bundle an additional user-specified directory in the GRAW. All files in this user-specified log directory will be copied to the GRAW on a per-flight basis. The logs directory should be a directory containing extra files (and no sub-directories) that you want to bundle with the raw data.
Optional Parameters
Each GRAW contains a “mission_data.yaml” metadata file. This file contains additional information about each bundle including the calibration file name, data checksums, and sensor types. Optional information such as UAV pilot credentials, location, and conditions can be added to the mission_data.yaml at the time of bundle submission.
Processing Extent
Processing extent files are geospatial files containing a polygon feature which can be used to trim output data products to a limited area. Using these processing extents will remove unwanted data, reduce the amount of space needs for product storage, and reduce processing times. An example extent file in GeoJSON is given below. KML and SHP files are supported as well. When Headwall data is collected, the HSI Polygon from the gpsMonitor file can also be used via the VNIR Processing Extent pipeline product option.
Output
When creating a GRAW bundle, the user must specify a name and output directory location. Later in processing, product names can be prepended with the GRAW name, and the processing directory (GPRO) can be stored in the same directory alongside the GRAW.
The GRAW output structure is organized with system calibration, processing extent, mission metadata, and a copy of the bundling pipeline in the GRAW root directory. Sensor data from each unique sensor id is organized by flight inside each sensor id folder. Log data is organized by flight into a common “logs” folder.
Upon submission of a processing job including hyperspectral reflectance target selection, an additional “targets.yaml” file is saved to the GRAW. This file contains information regarding which pixels have been selected for each target and where the target file locations are. In the GRYFN processing GUI, pre-selected target pixels may be re-loaded upon selecting a GRAW for processing which contains this “targets.yaml” file.
Last updated