L1A Processing#

L1A processing takes data from CCSDS packets to L1A Dataset objects suitable for writing as NetCDF.

Each L1A data product contains a single packet type (APID), decoded and reformatted for easier usage in the processing system. The structure of L1A products is controlled by L1A processing packet configurations, which instruct the L1A pipeline how to restructure the packet data. For example, packets that contain multiple samples of a single data point are restructured to pivot the samples into their own data array with a time dimension for the sample time. Fields that appear only once per packet are left associated with the “PACKET” index dimension (no coordinate).

Steps:

Read packet data using Space Packet Parser
Fetch the L1A PacketConfiguration object to configure L1A processing
Create an xr.Dataset according to the PacketConfiguration, expanding multi-sample fields into sample-indexed arrays and aggregating binary blob fields as configured
Look up the L1A product definition path via get_l1a_product_definition_path(apid) and write the Dataset to NetCDF using write_libera_data_product()

Configuration Overview#

L1A processing draws on three layers of configuration:

Global runtime config (config.json): controls file paths and behaviour flags such as SKIP_PACKET_HEADER_BYTES. All values can be overridden by environment variables of the same name.
L1A processing configs YAML (l1a_processing_configs.yml): defines per-APID packet structure (sample groups, aggregation groups, array groups, time field mappings). Its path is set by the LIBERA_L1A_PROCESSING_CONFIGS_PATH config key.
XTCE packet definitions (e.g. icie_xtce_tlm.xml): define the binary field layout at the bit-field level, consumed by Space Packet Parser. Each PacketConfiguration references the appropriate XTCE file via its packet_definition_config_key.

Ground Testing Data#

By default, SKIP_PACKET_HEADER_BYTES is 0 in config.json. This is correct for flight and production data delivered through the SDC downlink pipeline.

Raw ground testing captures from hardware-in-the-loop systems (e.g. Hydra/FSW) prepend an extra 8-byte record header to each packet before the standard CCSDS primary header. To process a raw capture directly, set SKIP_PACKET_HEADER_BYTES to 8. This is a global setting that affects all packets in a processing run.

Ground captures delivered to the SDC are demuxed beforehand (see Demuxed ground CCSDS files) with the record header already stripped, so they need the default 0 and none of this applies to them.

Set it via environment variable before running:

export SKIP_PACKET_HEADER_BYTES=8

In test code, override it with monkeypatch:

monkeypatch.setenv("SKIP_PACKET_HEADER_BYTES", "8")

The value is read once per call to parse_packets_to_l1a_dataset() (or overridden via its skip_header_bytes= argument) and forwarded to Space Packet Parser as skip_header_bytes. extract_data_time_range follows the same rule: explicit skip_header_bytes= if given, otherwise the config value. scan_ground_ccsds_file does not consult config at all — it defaults to GROUND_CCSDS_SKIP_HEADER_BYTES (0), which is what a demuxed file needs.

Demuxed ground CCSDS files#

Ground captures are demuxed outside the pipeline into one file per APID, with the 8-byte record header stripped. Basenames (no extension) match:

LIBERA_SDC_<apid>_ccsds_<yyyy>_<doy>_<hh>_<mm>_<ss>

Example: LIBERA_SDC_1057_ccsds_2026_191_14_00_00. Use LiberaGroundCcsdsFilename / AbstractValidFilename.from_file_path to validate one and to read its APID and archive prefix:

from libera_utils.io.filenaming import LiberaGroundCcsdsFilename

fn = LiberaGroundCcsdsFilename("LIBERA_SDC_1057_ccsds_2026_191_14_00_00")
fn.apid              # 1057
fn.bin_start         # datetime(2026, 7, 10, 14, 0, tzinfo=UTC)
fn.archive_prefix    # "GroundCCSDS/1057/2026/07/10"

The APID field accepts any value in the CCSDS 11-bit range (0-2047), including APIDs with no LiberaApid member, so an unmodelled APID can still be archived. The DPI is DataProductIdentifier.l0_ground_ccsds (GROUND-CCSDS) for every APID, distinct from EDOS PDS products. These names are accepted by the manual ingest CLI (s3-utils put / manual_ingest_data_products) so captures can be staged into the SDC Ingest Dropbox without CNM/ASDC delivery.

The time fields are the start of the time bin the file was cut from, not the span of the packets inside it — packet times can fall outside the bin, and in practice do. The GroundCCSDS/<apid>/<yyyy>/<mm>/<dd>/ prefix is just an expansion of those fields and carries no guarantee about the packet or data times within. Searchable times come from File Metadata at ingest, not from the archive path. During ground testing the two can differ by large gaps, since the simulated spacecraft clock runs at a mission-era epoch while the filename records wall-clock binning. Camera data times also legitimately precede packet times by hours, because WFOV images are downlinked well after they are taken.

Scanning a demuxed ground file#

libera_utils.l1a.ground_ccsds.scan_ground_ccsds_file returns the packet and science data time spans one file contributes to File Metadata. The APID comes from the basename unless passed explicitly:

from libera_utils.l1a.ground_ccsds import scan_ground_ccsds_file

span = scan_ground_ccsds_file("LIBERA_SDC_1036_ccsds_2026_191_14_00_00")
span.apid                                        # LiberaApid.icie_rad_sample
span.first_packet_time, span.last_packet_time
span.first_data_time, span.last_data_time        # None unless data-time indexed
span.degraded_reason                             # why data times are absent, else None

It returns None, logging the reason, when no packet-time span can be produced at all: the APID has no LiberaApid member, has no L1A packet configuration, the file holds nothing parseable for it, or every packet time is implausible. Several APIDs present in ground captures (icie_sw_stat, icie_seq_hk, icie_fp_hk, icie_log_msg, icie_axis_hk, icie_ana_hk) currently have no packet configuration, so they archive but yield no searchable times.

A data-time failure is narrower: the packet-time span is still returned, with first_data_time/last_data_time left None and degraded_reason set. This is the expected outcome for a WFOV file whose SOP packet landed in a different bin.

Implausible timestamps#

Hardware-in-the-loop ground testing can emit a leading packet before the onboard clock has received its first time-sync command. Its day/second counters read near-zero, which decodes to a timestamp just after CCSDS_EPOCH (1958-01-01) — enough to stretch a min()/max() span across ~68 years, and downstream to send the File Metadata applicable-date day-walk across the same range. A corrupted high-order bit in a day counter does the same at the other end.

scan_ground_ccsds_file and extract_data_time_range both filter through libera_utils.l1a.packets.drop_implausible_telemetry_times, which drops NaT and anything outside MIN_VALID_TELEMETRY_TIME–MAX_VALID_TELEMETRY_TIME (config keys, defaults 2020-01-01 and 2045-01-01). Both bounds are fixed dates so that a span written to File Metadata does not depend on when the extraction ran; the ceiling is far enough out to admit DITL captures running at a mission-era epoch.

Filtering happens before any min()/max() is taken, and exclusions are logged at WARNING. If nothing survives for an APID, scan_ground_ccsds_file returns None and extract_data_time_range raises DataTimeUndeterminedError, rather than returning a bogus span.

Data-time extraction (ingest applicable dates)#

Camera and radiometer science times are not the same as CCSDS secondary header packet times. For File Metadata applicable-date indexing, use libera_utils.l1a.data_time_extractors.extract_data_time_range:

Data-time indexed APIDs (DATA_TIME_INDEXED_APIDS): icie_wfov_sci, icie_rad_sample, icie_rad_full, icie_cal_sample, icie_cal_full, icie_axis_sample, jpss_sc_pos.
Camera: SOP packet FSW image timestamps (reuses wfov_image_metadata helpers).
Radiometer / cal sample APIDs: sample epoch + period from the L1A processing config — without expanding all sample data fields into an L1A product.
icie_axis_sample: on the per-sample (time_field_patterns) path; every sample carries its own timestamp rather than being derived from an epoch plus a fixed period.
jpss_sc_pos (APID 11): the extent of both of its sample groups (see below).
Raises DataTimeUndeterminedError when the span cannot be determined, and returns None for a WFOV file with no in-window SOP packet.
extract_data_time_range_from_dataset takes an already-parsed packet dataset, so a caller that has parsed the APID once (as scan_ground_ccsds_file does) need not re-read the file.
Demuxed ground CCSDS files need no header skip; the default SKIP_PACKET_HEADER_BYTES of 0 is correct.

All other APIDs remain packet-time indexed (Construction Record first/last packet times).

What a data time span means#

A span is the full extent of data present in the file: the earliest data time to the latest, across every timeseries the file carries, even when those times come from different clocks. It is deliberately not narrowed to the range where all of a file’s timeseries are simultaneously available, because the span exists for ingest indexing. Completeness is the consumer’s judgement:

WFOV: every in-window SOP contributes, including one whose image is truncated at the end of the file. The span therefore does not match the L1A product’s CAMERA_TIME range for a chunked file, where CAMERA_TIME covers only images completing SOP-to-EOP.
jpss_sc_pos (APID 11): both sample groups contribute, so the span is not restricted to the range covered by both.

JPSS SC position (APID 11): two sample-time clocks#

A JPSS SC position packet carries three timeseries: its own packet time, an ephemeris sample time (ADAET1*, the ADGPS sample group) and an attitude sample time (ADAET2*, the ADCFA group). The two sample times are applied independently by the spacecraft and do not coincide — on the jpss1 test PDS they run 100 ms apart, so each group’s span starts and ends at a different instant.

The recorded extent runs from the earliest time either group reports to the latest, so on that file it starts on an ADCFA sample and ends on an ADGPS one. Disjoint groups are not an error: the span simply covers both and the gap between them. If one group is filtered out entirely by the plausibility window, the surviving group’s times are the whole span; only an APID with no usable times in any group raises DataTimeUndeterminedError.

A consumer that needs ephemeris and attitude together — geolocation does — must intersect the two sample-time ranges itself from the samples in the file. The span in File Metadata will not have done that for it, and a file whose span covers a given instant does not guarantee both timeseries cover it.

L1A Packet Processing Configurations#

Per-APID processing configurations are defined in l1a_processing_configs.yml (path resolved from LIBERA_L1A_PROCESSING_CONFIGS_PATH). Each entry is keyed by the APID name from the LiberaApid enum. Configurations are lazily loaded and cached on the first call to get_packet_config().

from libera_utils.constants import LiberaApid
from libera_utils.l1a.l1a_packet_configs import get_packet_config

config = get_packet_config(LiberaApid.icie_axis_sample)
print(config.packet_time_coordinate)                   # → "PACKET_ICIE_TIME"
print(config.sample_groups[0].sample_time_dimension)   # → "AXIS_SAMPLE_ICIE_TIME"

PacketConfiguration#

The top-level object describing how to process one packet type.

The computed property packet_time_coordinate returns PACKET_{time_source}_TIME (e.g. PACKET_ICIE_TIME for packet_time_source: "ICIE"). This is the non-dimension coordinate attached to the PACKET dimension in the output Dataset.

Minimal example — a housekeeping packet with no samples or aggregations:

icie_nom_hk:
  packet_apid: "icie_nom_hk"
  packet_time_fields:
    day_field: "ICIE__TM_DAY_NOM_HK"
    ms_field: "ICIE__TM_MS_NOM_HK"
    us_field: "ICIE__TM_US_NOM_HK"
  packet_definition_config_key: "LIBERA_PACKET_DEFINITION"
  packet_time_source: "ICIE"

A JPSS packet using a different XTCE definition and time source:

jpss_sc_pos:
  packet_apid: "jpss_sc_pos"
  packet_time_fields:
    day_field: "DAYS"
    ms_field: "MSEC"
    us_field: "USEC"
  packet_definition_config_key: "JPSS_GEOLOCATION_PACKET_DEFINITION"
  packet_time_source: "JPSS"
  sample_groups:
    - ... # see SampleGroup below

TimeFieldMapping#

CCSDS packet timestamps are split across multiple integer fields. TimeFieldMapping names which XTCE packet fields hold each component. All fields are optional; at least one must be present.

Field	Description
`day_field`	Days since mission epoch
`s_field`	Seconds within the day
`ms_field`	Milliseconds (additive)
`us_field`	Microseconds (additive)

When TimeFieldMapping is used in time_field_patterns inside a SampleGroup with sample_count > 1, each field name must include %i as a placeholder for the sample index (e.g. "ICIE__AXIS_SAMPLE_TM_SEC%i").

SampleTimeSource#

Identifies which subsystem clock provides the timestamps for a packet or sample group.

Value	System
`ICIE`	Instrument Control and Interface Electronics (Libera main processor)
`FPE`	Focal Plane Electronics (Libera detector subsystem)
`JPSS`	JPSS spacecraft system clock

The time_source value is embedded in dimension names. A SampleGroup with name: "RAD_SAMPLE" and time_source: "FPE" produces the output dimension RAD_SAMPLE_FPE_TIME.

SampleGroup#

A SampleGroup describes one set of related samples within a packet that share timing characteristics. Each group produces a new dimension in the output Dataset (the sample time dimension), with all configured data fields mapped to it.

Field	Type	Description
`name`	str	Group identifier; used to build the sample time dimension name
`sample_count`	int	Number of samples per packet
`data_field_patterns`	list[str]	XTCE field name patterns; use `%i` for the sample index
`time_source`	`SampleTimeSource`	Clock source for sample timestamps
`time_field_patterns`	`TimeFieldMapping`	Per-sample timestamp fields — Timing Mode A
`epoch_time_fields`	`TimeFieldMapping`	Single epoch timestamp per packet — Timing Mode B
`sample_period`	int (µs)	Fixed period between samples in microseconds — Timing Mode B

Exactly one timing mode must be used; providing both or neither is a configuration error.

The computed property sample_time_dimension returns {name}_{time_source}_TIME (e.g. AXIS_SAMPLE_ICIE_TIME).

After sample expansion, data field names in the output Dataset have %i removed (trailing underscores are also stripped). For example, ICIE__AXIS_AZ_FILT%i becomes ICIE__AXIS_AZ_FILT.

A {name}_packet_index variable (integer, dimensioned by the sample time dimension) is always created alongside the expanded data. It maps each sample back to its originating packet index in the PACKET dimension, enabling efficient joins between per-packet housekeeping and per-sample science data.

Timing Mode A — Explicit Per-Sample Timestamps#

Use this mode when each sample within a packet carries its own timestamp fields. All field names in time_field_patterns must contain %i. The processor iterates i from 0 to sample_count - 1, resolves each field name, and produces a flat time array of length n_packets × sample_count.

# AXIS_SAMPLE: 50 Az/El encoder samples, each with its own ICIE timestamp
icie_axis_sample:
  packet_apid: "icie_axis_sample"
  packet_time_fields:
    day_field: "ICIE__TM_DAY_AXIS_SAMPLE"
    ms_field: "ICIE__TM_MS_AXIS_SAMPLE"
    us_field: "ICIE__TM_US_AXIS_SAMPLE"
  sample_groups:
    - name: "AXIS_SAMPLE"
      time_field_patterns:
        s_field: "ICIE__AXIS_SAMPLE_TM_SEC%i"
        us_field: "ICIE__AXIS_SAMPLE_TM_SUB%i"
      data_field_patterns:
        - "ICIE__AXIS_AZ_FILT%i"
        - "ICIE__AXIS_EL_FILT%i"
      sample_count: 50
      time_source: "ICIE"
  packet_definition_config_key: "LIBERA_PACKET_DEFINITION"
  packet_time_source: "ICIE"

Timing Mode B — Epoch + Periodic Sampling#

Use this mode when samples are evenly spaced from a single epoch timestamp recorded per packet. epoch_time_fields names the fields holding the epoch (no %i placeholders). sample_period is specified as an integer in microseconds. Sample times are computed as epoch + i × sample_period for i in 0..sample_count-1.

Note that packet_time_source and time_source can differ. The packet-level timestamp (from ICIE) and the sample-level timing (from FPE) are tracked independently. This cross-source pattern is typical for science packets where the instrument main processor records a coarse packet time but the detector subsystem provides precise per-sample timing.

# RAD_SAMPLE: 50 radiometer samples at 5 ms intervals, FPE-timed
icie_rad_sample:
  packet_apid: "icie_rad_sample"
  packet_time_fields:
    day_field: "ICIE__TM_DAY_RAD_SAMPLE"
    ms_field: "ICIE__TM_MS_RAD_SAMPLE"
    us_field: "ICIE__TM_US_RAD_SAMPLE"
  sample_groups:
    - name: "RAD_SAMPLE"
      epoch_time_fields:
        s_field: "ICIE__RAD_SAMP_START_HI"
        us_field: "ICIE__RAD_SAMP_START_LO"
      sample_period: 5000 # microseconds (5 ms between samples)
      data_field_patterns:
        - "ICIE__RAD_SAMPLE%i_0"
        - "ICIE__RAD_SAMPLE%i_1"
        - "ICIE__RAD_SAMPLE%i_2"
        - "ICIE__RAD_SAMPLE%i_3"
      sample_count: 50
      time_source: "FPE"
  packet_definition_config_key: "LIBERA_PACKET_DEFINITION"
  packet_time_source: "ICIE"

AggregationGroup#

Some packets carry large binary payloads that XTCE decodes into many numbered scalar fields instead of one binary field (e.g. hundreds of individual single-byte fields for a single payload). AggregationGroup reassembles these into a single bytes-typed variable per packet, reducing variable count and simplifying downstream access.

Field	Type	Description
`name`	str	Output variable name in the Dataset
`field_pattern`	str	XTCE field name pattern with `%i` for the field index
`field_count`	int	Number of fields to aggregate (indices `0` to `field_count-1`)
`dtype`	numpy dtype str	Target dtype, e.g. `\|S972` for a 972-byte fixed-length bytes value

The total byte size of all aggregated fields must equal dtype.itemsize. A ValueError is raised at processing time if there is a mismatch.

No packet currently uses aggregation_groups. WFOV SCI (icie_wfov_sci) used to be the motivating case, reassembling ICIE__WFOV_DATA from 972 individual byte fields. Its XTCE definition now decodes ICIE__WFOV_DATA directly as a single BinaryParameterType field, so Space Packet Parser hands back the whole payload as one field already and icie_wfov_sci declares no aggregation_groups at all. The mechanism is still fully supported for any future packet whose XTCE definition splits a binary payload into many numbered scalar fields the same way WFOV’s used to; the example below is illustrative of that shape, not a real current config.

# Illustrative: reassembling a hypothetical 972-byte payload split into 972 numbered byte fields
# by its XTCE definition (no current packet configuration actually needs this)
some_packet:
  packet_apid: "some_packet"
  packet_time_fields:
    day_field: "SOME_PACKET_TM_DAY"
    ms_field: "SOME_PACKET_TM_MS"
    us_field: "SOME_PACKET_TM_US"
  aggregation_groups:
    - name: "SOME_PACKET_DATA"
      field_pattern: "SOME_PACKET_DATA_%i"
      field_count: 972
      dtype: "|S972"
  packet_definition_config_key: "LIBERA_PACKET_DEFINITION"
  packet_time_source: "ICIE"

ArrayGroup#

Some packets contain numbered enum/status fields that represent fixed slot arrays rather than independent packet variables. ArrayGroup stacks these fields into a single 2D variable with dimensions ["PACKET", "ARRAY_{N}"], where N is field_count.

Field	Type	Description
`name`	str	Output variable name in the Dataset
`field_pattern`	str	XTCE field name pattern with `%i` for the field index
`field_count`	int	Number of fields to stack (indices `0` to `field_count-1`)
`dimension`	str	Output array dimension, must be `ARRAY_{field_count}`
`dtype`	numpy dtype str	Target dtype for each array element (for example `\|S8` or `uint16`)

Unlike AggregationGroup, ArrayGroup preserves element boundaries and supports direct index-based access in downstream code.

# NOM_HK: waypoint and sequence enums stacked as indexed arrays
icie_nom_hk:
  packet_apid: "icie_nom_hk"
  packet_time_fields:
    day_field: "ICIE__TM_DAY_NOM_HK"
    ms_field: "ICIE__TM_MS_NOM_HK"
    us_field: "ICIE__TM_US_NOM_HK"
  array_groups:
    - name: "ICIE__SW_FP_WP_ST_WP"
      field_pattern: "ICIE__SW_FP_WP_ST_WP%i"
      field_count: 128
      dimension: "ARRAY_128"
      dtype: "|S8"
    - name: "ICIE__SW_SEQ_EXEC_POS_OP"
      field_pattern: "ICIE__SW_SEQ_EXEC_POS_OP%i"
      field_count: 8
      dimension: "ARRAY_8"
      dtype: "uint16"
  packet_definition_config_key: "LIBERA_PACKET_DEFINITION"
  packet_time_source: "ICIE"

L1A Product Structure#

This varies by packet but there is some consistent behavior:

Every L1A product has a "PACKET" index dimension that is simply an index of packets

Note: Space Packet Parser (SPP) internally creates datasets with a lowercase "packet" dimension. The pipeline immediately renames this to "PACKET" to conform to the SDC naming standard (SDC_PACKET_DIMENSION = "PACKET" in packets.py). All downstream code and product definitions must use "PACKET".
Every L1A product has a packet time coordinate with dimension "PACKET"
Fields appearing once per packet are associated with the "PACKET" index dimension
Every sample set (possibly multiple) has a sample time coordinate that is a dimension coordinate (coordinate name == dimension name)
Every sample variable has a dimension for its sample time
Samples taken at the same time (possibly across multiple fields) are associated with the same sample time dimension
Every sample group has a {name}_packet_index variable (integer, same dimension as the sample data) that maps each sample back to its originating packet index in the PACKET dimension. This enables efficient joins between per-packet metadata and per-sample science data. The sample axis is sorted by sample time, not by packet, so {name}_packet_index is usually but not necessarily non-decreasing: where two adjacent packets’ sample clocks skew by less than a sample interval their sample blocks interleave and the index steps backwards at those positions. Consumers must use it as an element-wise mapping and must not assume that each packet’s samples form one contiguous block.

WFOV camera science (APID 1040) image metadata#

Packet Data Structure - Slicing and Reconstructing#

A single wide field of view (WFOV) image is too large for one CCSDS packet, so the camera splits it into a sequence of packets, each carrying one slice of the image plus fields describing where that slice belongs:

Field	Description
`ICIE__MEM_DUMP_FLAGS_WFOV`	This packet’s position in the image sequence: `SOP` (Start of Photo), `MOP` (Middle of Photo), `EOP` (End of Photo), or `SINGLE` (whole image fit in one packet — not supported, see below)
`ICIE__MEM_DUMP_OFFSET_WFOV`	Byte offset of this slice within the reassembled image
`ICIE__MEM_DUMP_LENGTH_WFOV`	Number of valid image bytes carried in this packet
`ICIE__WFOV_DATA`	The raw image-slice payload for this packet

libera_utils reconstructs each full image by walking packets in stream order: an SOP packet with ICIE__MEM_DUMP_OFFSET_WFOV == 0 opens a new image, each subsequent packet is appended as long as its offset matches the running byte count, and an EOP packet closes the image out. Any break in that sequence — an offset that doesn’t line up, an EOP with no preceding SOP, or a SOP left dangling with no matching EOP — discards the in-progress image instead of stitching it incorrectly, and is counted in PacketCountNotUsedInImages below.

Edge-of-window truncation is expected and handled separately. Each call to this pipeline processes one packet-stream window (e.g. one processing run’s worth of downlinked packets), and it’s normal — not an error — for the first packet in that window to not be a qualifying SOP (the image it belongs to started before the window) and for the last packet to not be an EOP (its image continues into the next window). Both edges are silently dropped from the output the same way a genuine mid-stream break is, but are not counted in PacketCountNotUsedInImages, since that attribute is reserved for real anomalies. Instead, FirstImageIncomplete / LastImageIncomplete (booleans, see below) flag whether this window’s leading/trailing edge was truncated at all.

WFOV science packets carry two independent time sources:

PACKET_ICIE_TIME: CCSDS telemetry time from ICIE__TM_DAY/MS/US_WFOV_SCI on every packet. This coordinate is used for packet ordering and deduplication.
CAMERA_TIME: FSW image acquisition time decoded from each complete stitched image (SOP→EOP with valid offsets). One row is added per successfully stitched image in packet stream order.

During L1A parsing for APID 1040, libera_utils:

Stitches image slice packets from a qualifying SOP (ICIE__MEM_DUMP_FLAGS_WFOV == "SOP" and ICIE__MEM_DUMP_OFFSET_WFOV == 0) through EOP into a full NAND image blob.
Stores the complete compressed JPEG-LS image on CAMERA_TIME as WFOV_COMPRESSED_IMAGE (uint8/BLOB_BYTE with WFOV_COMPRESSED_IMAGE_LENGTH; readers must use image[:length] to drop zero padding).
Decodes the 176-byte WFOV header (36-byte FSW block + 140-byte FPGA block) as a single atomic unit — either there are enough bytes for the whole header or there aren’t; there’s no independent per-sub-block size check. Fields land on CAMERA_TIME across four categories: WFOV_FSW_HEADER_*, WFOV_IMAGE_HEADER_*, WFOV_IMAGE_FOOTER_*, and WFOV_FPGA_STATUS_*. Separately, the trailing 8-byte NAND footer is checked against a known-good magic byte pattern (not decoded into fields) to catch corrupted images; mismatches count toward FooterMismatchCount.
Drops ICIE__WFOV_DATA from the output entirely and sets PACKET_IMAGE_ID (0..N-1, or -1 for packets not part of a complete image).

ICIE__WFOV_DATA is removed from the dataset unconditionally, regardless of whether a packet’s data ended up folded into a complete image or not: for packets in a complete image, that content is already duplicated (compressed) in WFOV_COMPRESSED_IMAGE; for packets that never completed an image — truncated at a window edge or dropped for a genuine anomaly — the raw payload cannot be assembled into an image from this granule alone. PACKET_IMAGE_ID is the only per-packet trace-back to a stitched image. Incomplete or failed images do not receive a CAMERA_TIME row.

Granules that yield no images#

If no complete image can be stitched from the packet stream, parse_packets_to_l1a_dataset raises ValueError instead of returning a dataset, and it does so before dropping ICIE__WFOV_DATA — so the caller still holds every raw packet for diagnosis. The message carries the packet count and all four counters.

This is a deliberate hard failure rather than an empty product. A granule with no images has no CAMERA_TIME rows to derive a filename from and would fail product conformance on every CAMERA_TIME variable, so there is nothing valid to write; failing at the point of detection is more useful than an opaque error from the filenaming code much later.

Unsupported `SINGLE` packets#

A packet flagged SINGLE (a whole image in one packet) is not supported. It is not expected during normal operations, so each one is logged as a warning and dropped: the packet counts toward PacketCountNotUsedInImages and keeps PACKET_IMAGE_ID == -1, but produces no image. A SINGLE arriving mid-collection also abandons the image in progress. There is no dedicated file-level counter — use the warning in the logs and PACKET_IMAGE_ID == -1 to investigate.

File-level quality attributes:

PacketCountNotUsedInImages (integer ≥ 0): total packets swept up in a rejected/incomplete SOP→EOP attempt due to a genuine anomaly (dangling SOP aborted by a new SOP, offset gap, orphan EOP, bad SOP offset) — excludes the expected truncation at this window’s own leading/trailing edge, which is reported separately below
ErrorFlaggedImageCount: complete images whose FPGA status block has any error bit set (see WFOV_FPGA_STATUS_*)
FooterMismatchCount: complete images whose trailing 8-byte NAND footer didn’t match the expected magic bytes
HeaderParseErrorCount: structurally complete images discarded because the 176-byte WFOV header could not be decoded (too few stitched bytes to contain a full header). Without a header there is no acquisition time, and an image with no time is unusable, so it gets no CAMERA_TIME row at all rather than a NaT one. Its packets also count toward PacketCountNotUsedInImages and keep PACKET_IMAGE_ID == -1
FirstImageIncomplete (0/1, semantically boolean — NetCDF attributes have no bool type): 1 if this window’s first packet wasn’t a qualifying SOP (i.e. the image it belongs to started before this window)
LastImageIncomplete (0/1, semantically boolean): 1 if this window ended mid-collection (a dangling SOP that never reached its EOP because the window ran out, not because of an anomaly)

Decoded metadata on the CAMERA_TIME dimension:

Supporting variables: CAMERA_PACKET_INDEX, WFOV_HEADER_PARSE_VALID, WFOV_COMPRESSED_IMAGE, WFOV_COMPRESSED_IMAGE_LENGTH
FSW header fields: WFOV_FSW_HEADER_* (19 uppercase field names matching the libera_cam FSW header layout)
Image header fields: WFOV_IMAGE_HEADER_* (21 fields from the FPGA block’s image header)
Image footer fields: WFOV_IMAGE_FOOTER_* (5 fields from the FPGA block’s internal footer)
FPGA status fields: WFOV_FPGA_STATUS_* (7 single-bit error flags from the FPGA status word)

When writing the L1A NetCDF product, pass time_variable="CAMERA_TIME" to write_libera_data_product() so the filename reflects the first and last complete image FSW times in packet order. Use PACKET_ICIE_TIME for packet ordering and all other non-filename uses.

Dual exposure and VIDEO timing#

CAMERA_TIME is the first integration time. FSW stamps one acquisition time per complete image. In DUAL mode (WFOV_FSW_HEADER_IMG_MODE == 0), the second sequential exposure lags the first by about 111–350 ms. That offset is not stored as a separate L1A time coordinate (also noted on the WFOV_FSW_HEADER_IMG_MODE variable attributes).
Which pixels used which exposure is encoded in the 13th bit of each decompressed pixel. L1A keeps the JPEG-LS payload compressed in WFOV_COMPRESSED_IMAGE, so per-pixel exposure masks and per-pixel times are an L1B responsibility after decompression.
VIDEO mode (WFOV_FSW_HEADER_IMG_MODE == 1) can produce two NAND images from one camera trigger with identical FSW timestamps. Distinguish members with CAMERA_PACKET_INDEX (and packet stream order); do not assume CAMERA_TIME alone is a unique image key. WFOV_IMAGE_HEADER_READOUT is independent of WFOV_FSW_HEADER_IMG_MODE and should not be used to infer VIDEO pairing.

Downstream (libera_cam): Each CAMERA_TIME row already has a complete compressed JPEG-LS image in WFOV_COMPRESSED_IMAGE (trim with WFOV_COMPRESSED_IMAGE_LENGTH) plus decoded FSW/FPGA metadata on the same dimension. L1B decompresses that payload directly; it does not re-stitch packets or re-parse headers.

For example, for N packets, the AXIS_SAMPLE packet containing Azimuth and Elevation mechanism data comes down with 50 Az and El samples per packet (a sample group). It’s L1A product has:

coordinates:
  # Packet timestamp
  PACKET_ICIE_TIME:
    dtype: datetime64[ns]
    dimensions: ["PACKET"]
    attributes:
      long_name: Packet timestamp from ICIE main processor
    encoding:
      units: nanoseconds since 1958-01-01
      calendar: standard
      dtype: int64
  # Sample timestamp
  AXIS_SAMPLE_ICIE_TIME:
    dtype: datetime64[ns]
    dimensions: ["AXIS_SAMPLE_ICIE_TIME"]
    attributes:
      long_name: Azimuth and elevation encoder sample timestamp
    encoding:
      units: nanoseconds since 1958-01-01
      calendar: standard
      dtype: int64

variables:
  # There are more variables not listed here

  # Per packet checksum
  ICIE__AXIS_SAMPLE_CHECKSUM:
    dtype: uint32
    dimensions: ["PACKET"]
    attributes:
      long_name: ICIE axis sample packet checksum
  # Azimuth samples
  ICIE__AXIS_AZ_FILT:
    dtype: float32
    dimensions: ["AXIS_SAMPLE_ICIE_TIME"]
    attributes:
      long_name: ICIE azimuth axis filtered encoder reading
      units: radians
  # Elevation samples
  ICIE__AXIS_EL_FILT:
    dtype: float32
    dimensions: ["AXIS_SAMPLE_ICIE_TIME"]
    attributes:
      long_name: ICIE elevation axis filtered encoder reading
      units: radians
  # Packet index for each sample
  AXIS_SAMPLE_packet_index:
    dtype: int64
    dimensions: ["AXIS_SAMPLE_ICIE_TIME"]
    attributes:
      long_name: Packet index for axis sample data
      comment: Maps each axis sample to its originating packet index

ObsID-trimmed NOM-HK products#

After a daily NOM-HK-DECODED product is produced, calibration pipelines need a per-ObsID subset of that file. Helpers in libera_utils.l1a.nom_hk_trim detect contiguous runs of known calibration Observation IDs (ObsIDs) (as cataloged in libera_utils.obsids.OBSID_REGISTRY) and write one NetCDF per run:

from libera_utils.l1a.nom_hk_trim import write_trimmed_nom_hk_products

# After parse_packets_to_l1a_dataset(...) for NOM-HK:
trimmed_paths = write_trimmed_nom_hk_products(
    nom_hk_ds,
    output_dir,
    time_variable="PACKET_ICIE_TIME",
    add_archive_path_prefix=True,  # L1A preprocessor / ingest dropbox
)

TRIMMED products reuse the NOM-HK-DECODED variable schema; only ProductID (and thus the filename product token) changes. Science/scan modes (ObsIDs 0-2, 128, 132, and 136–140) are cataloged but do not emit TRIMMED files.

The ProductID names the run’s calibration dependency family (NOM-HK-SWC-FAMILY-TRIMMED, NOM-HK-SOLAR-FAMILY-TRIMMED, …) rather than its individual ObsID, and one processing step is deployed per family. One day therefore normally produces several files sharing a family ProductID — one per ObsID in that family — distinguished by their filename time ranges. Each file covers exactly one ObsID run, and the ObsID stays readable from the ICIE__SW_OBSID_RAD / ICIE__SW_OBSID_WFOV variable inside the file.

Normal operations expect each calibration ObsID at most once per day. If the same (source, obsid) appears in multiple disjoint runs, each run is written as a separate file and a warning is logged. Two different ObsIDs of one family are not that case and do not warn.

See the ObsID Registry page for the (source, obsid) keying, how downstream repos dispatch cal-combine steps from it, and how to register a new calibration ObsID.