Bundle Data

In order to provide processing reliability and consistent data organization for multi-flight time-series datasets, GRYFN Processing Tool incorporates a data bundling process which organizes supplied data into a standard structure with metadata. These bundled datasets (denoted with a “.graw” folder extension) contain all the specified raw data, system calibration, and a processing extent (if supplied). In addition to the required processing inputs, the bundler also optionally saves information on pilot, location, and other parameters. GRAW bundles are the input to a processing pipeline. In the Bundle Data tab, the user can choose to create a new bundle, or add data to an existing GRAW bundle.

Inputs

A table describing all inputs to the data bundler is provided below.

Name
Type
Format
Required

System Calibration

GRYFN System Calibration

YAML

Yes

Processing Extent

Processing Extent

GeoJSON/KML/SHP

No

GNSS

Custom GRYFN Export

TXT

*

LiDAR

VLP/OS LiDAR Raw Data

PCAP

*, **

RGB

3-Band Imagery

JPG/TIFF

*

VNIR

Headwall/Specim Data Folder

ENVI

*

SWIR

Headwall Data Folder

ENVI

*

Logs

Hardware logs and misc. files

Any

No

Pilot

UAV pilot information

Text/YAML

No

Location

Mission location information

Text/YAML

No

Conditions

Weather and other notes

Text/YAML

No

* At least one sensor type is required to bundle

** Ouster data processing also requires the .json metadata file

System Calibration

GRYFN system calibration files are a human readable YAML file specifying a unique identifier (id) for each sensor with various parameters that define that sensor’s type, intrinsic optical characteristics, and extrinsic position and orientation relative to a common trajectory reference frame. This calibration file may also contain optional information such as sensor model, serial number, and calibration date. An example of GRYFN’s System Calibration file structure with examples for each sensor type and definitions of the standard parameters is given below.

platform: Freefly ALTA X
model: Mojave
serial: GMOJ2301
sensors:
- id: Sony RX1RM2
  model: DSC-RX1RM2
  serial: 6314615
  type: RGB
  date: 2023-07-21
  lever_arm:
    x: 0.216
    y: -0.049
    z: -0.008
  boresight:
    omega: 180.358151
    phi: 0.897287
    kappa: -90.345798
  camera:
    type: frame
    model: SMAC
    xp: 11.715605
    yp: -6.0726024
    c: 7556.16
    image_width: 7952
    image_height: 5304
    k1: -1.1199271e-09
    k2: -2.7774539e-17
    p1: 3.7102533e-08
    p2: -1.2328174e-07

- id: VLP16 Hi-Res
  model: Puck Hi-Res
  serial: 11406210245816
  type: LiDAR
  date: 2023-07-21
  lever_arm:
    x: -0.093
    y: -0.042
    z: 0.006
  boresight:
    omega: 1.918458
    phi: -0.077572
    kappa: 0.403461
  nominal_boresight:
    omega: 90.0
    phi: -90.0
    kappa: 0.0
- id: nHS-206
  model: Nano-Hyperspec
  type: VNIR
  date: 2023-07-21
  timedelay0: 0.0
  lever_arm:
    x: 0.136
    y: -0.039
    z: 0.016
  boresight:
    omega: 0.461230
    phi: 0.608521
    kappa: 0.299187
  camera:
    type: pushbroom
    pixel_size: 0.0074
    f: 8.256
    image_width: 640
- id: APX-15
  model: APX-15Lv3
  serial: 6035C00140
  type: GNSS

  • id: Unique sensor ID string

  • model: Manufacturer sensor model

  • type: RGB, LiDAR, VNIR, SWIR, or GNSS

  • date: Date of data collection used in calibration process.

  • time_delay0: Offset in seconds to apply from trajectory to imagery.

  • lever_arm: Offset in meters from trajectory reference point to sensor origin. X, Y, and Z are respectively forward, right, and down.

  • boresight: Rotation in degrees between trajectory coordinate system and sensor coordinate system.

  • nominal_boresight: Nominal rotation in degrees between trajectory coordinate system and sensor coordinate system. Omega, phi, and kappa are rotations around X, Y, and Z respectively.

  • camera: Parameters defining the intrinsic properties of optical sensors such as RGB frame cameras and hyperspectral line scanners.

    • type: frame or pushbroom

    • model: camera IOP model

    • xp: perspective center offset in sensor X direction

    • yp: perspective center offset in sensor Y direction

    • c/f: focal length in pixels(frame) or mm(pushbroom)

    • image_width: width of image sensor in pixels

    • image_height: height of image sensor in pixels

    • pixel_size: size of pixel in mm

Processing Extent

Processing extent files are geo-spatial files containing a polygon feature which can be used to trim output data products to a limited area. Using these processing extents will remove unwanted data, reduce the amount of space needs for product storage, and reduce processing times. An example extent file in GeoJSON is given below. KML and SHP files are supported as well. When Headwall data is collected, the HSI Polygon from the gpsMonitor file can also be used via the VNIR Processing Extent pipeline product option.

{
    "type": "Feature",
    "geometry": {
        "type": "Polygon",
        "coordinates": [
            [[-86.995222417348046, 40.471169271289895],
            [-86.995222471001114, 40.470412174363126],
            [-86.994041227816268, 40.470412119503784],
            [-86.994041160897481, 40.471169216429075],
            [-86.995222417348046, 40.471169271289895]]
        ]
    },
    "properties": {
        "name": "Name"
    }
}

Raw Data Source

The GRYFN bundling process allows two methods of navigation to raw data products to be bundled, single-source and multi-source. In single-source mode, the bundler will parse the input directory and its’ child directories for all possible data types. For example, all files with a “.pcap” extension will be organized into a folder for the sensor id with type “LiDAR” and a Headwall VNIR data folder and its’ associated dark data folder (if available) will be copied into the GRAW in a folder for the sensor id with type “VNIR”.

Multi-source bundling allows the user to specify individual source directories for each data type.

Data source directories can be added on a per-flight basis. This allows GRAWs to contain data from multiple UAV flights. This feature is used when, for example, a large field requires multiple individual flights to obtain a complete dataset. These individual flight datasets, or flights, can be bundled together and processed simultaneously for a combined output. The multi-flight bundling feature is not meant to contain time-series data of the same area.

Bundling is not currently supported for multiple sensors of the same type.

A detailed description of the required files for each data type is given below.

Name
Type

GNSS

Directory containing POSPac export “export*.txt”, “event*.dat” (if frame camera), and optional “.t02” and “.t04” file backups. (POSPac project folder)

LiDAR

Directory containing “.pcap” files and ".json" Ouster metadata

RGB

Directory containing “.jpg” or “.tif”

VNIR

Directory containing VNIR acquisition data folder and a dark data folder (may contain raw or radiance data with default Headwall file names and extensions “_rd” but no additional files/folders)

SWIR

Directory containing SWIR acquisition data folder and a dark data folder (may contain raw, radiance, or nuc data with default Headwall file names and extensions “_rd” and “_nuc”, but no additional folders)

Logs

Directory containing any miscellaneous files to add to bundle logs directory

Radiometric Calibration

For Nano HP data bundling, the file path specified in the Default Radiometric Calibration Location and the VNIR raw data directory identified during bundling will be traversed for the correct radiometric calibration files for the dataset. If successfully found, the necessary files will be copied to the GRAW so that radiance conversion can be performed at processing time.

GNSS Data

The GRYFN bundling and processing pipeline accepts post-processed GNSS-INS data in the form of a custom text file. This file contains a necessary header block that specifies the location and filename of any related .dat event timetag files, UTC time offset and week number, and mapping frame information. A data column header is also given. An example header and five lines of trajectory data are given below.

GRYFN Export Trajectory v0.3

Project Info:
	Project File: 
	Event Timetag File: C:\Users\GRYFN\Documents\POSPac UAV\Unnamed(1)\Mission 1\Extract\event1_Mission 1.dat
	Reference to Primary GNSS Lever Arm:
		X: 0.045
		Y: -0.080
		Z: -0.340
	Reference to IMU Mounting Angles:
		X: -90.000
		Y: 90.000
		Z: 0.000
	UTC Mission Start Date: 20210630	Wednesday, June 30, 2021
	UTC Mission Start Time: 16h:09m:00s
	UTC Time Offset: 18s
	GPS Week: 2164

Mapping Fame Info:
	Datum: WGS84
	Grid: Universal Transverse Mercator 
	Zone: UTM North 15 (96W to 90W) 
	Ellipsoid: WGS84 
	Target Epoch: 2021.493151


GPS_TIME(SoD)	EASTING(m)	NORTHING(m)	ELLIPSOID_HEIGHT(m)	ROLL(deg)	PITCH (deg)	HEADING(deg)	EASTING_SD(m)	NORTHING_SD(m)	HEIGHT_SD(m)	ROLL_SD(deg)	PITCH_SD(deg)	HEADING_SD(deg)	ANGLERATE_X	ANGLERATE_Y	ANGLERATE_Z

58631.009323	652307.123299	4860894.206837	198.803703	-0.842261	-13.399428	265.182502	0.034731	0.032690	0.059805	0.167360	0.192645	0.522946	0.091385	0.020312	-0.011949
58631.014323	652307.088036	4860894.202650	198.803688	-0.819643	-13.314868	265.504095	0.034722	0.032695	0.059787	0.167342	0.192549	0.522770	0.075679	0.035670	0.004107
58631.019323	652307.015917	4860894.247530	198.867820	-0.807195	-13.295999	265.751157	0.034713	0.032700	0.059769	0.167324	0.192453	0.522595	0.049151	0.062895	-0.021024
58631.024323	652306.980530	4860894.243746	198.867781	-0.785800	-13.277930	265.754958	0.034705	0.032704	0.059751	0.167305	0.192357	0.522419	0.077773	0.061499	0.013531
58631.029324	652306.945127	4860894.239966	198.867734	-0.782700	-13.255140	265.744932	0.034696	0.032709	0.059733	0.167287	0.192261	0.522244	0.002727	0.078603	-0.033241

Logs

In addition to raw data, the bundler can also bundle an additional user-specified directory in the GRAW. All files in this user-specified log directory will be copied to the GRAW on a per-flight basis.

Optional Parameters

Each GRAW contains a “mission_data.yaml” metadata file. This file contains additional information about each bundle including the calibration file name, data checksums, and sensor types. Optional information such as UAV pilot credentials, location, and conditions can be added to the mission_data.yaml at the time of bundle submission.

Output

When creating a GRAW bundle, the user must specify a name and output directory location. Later in processing, product names can be prepended with the GRAW name, and the processing directory (GPRO) can be stored in the same directory alongside the GRAW.

The GRAW output structure is organized with system calibration, processing extent, mission metadata, and a copy of the bundling pipeline in the GRAW root directory. Sensor data from each unique sensor id is organized by flight inside each sensor id folder. Log data is organized by flight into a common “logs” folder.

Upon submission of a processing job including hyperspectral reflectance target selection, an additional “targets.yaml” file is saved to the GRAW. This file contains information regarding which pixels have been selected for each target and where the target file locations are. In the GRYFN processing GUI, pre-selected target pixels may be re-loaded upon selecting a GRAW for processing which contains this “targets.yaml” file.

PreviousUser ManualNextNew Job