Observation ID (ObsID) Registry

Libera’s ICIE flight software tags certain telemetry packets with a small integer ObsID identifying which calibration event or scan mode is running (gain/noise cal, a specific LED or blackbody temperature, a solar diffuser face, a lunar scan, cross-track imaging, and so on). libera_utils.obsids is the sole source of truth mapping those ObsIDs to the DataProductIdentifiers they produce. This page covers how the registry is structured and how it is used, both inside this package and by downstream algorithm repos.

ObsID sources

ObsIDs are reported in two separate NOM-HK telemetry fields, one per instrument:

Source	NOM-HK variable	Typical events
RAD	ICIE__SW_OBSID_RAD	Radiometer cal (gain, noise, SWC, LWC, solar, lunar, VIIRS)
WFOV	ICIE__SW_OBSID_WFOV	Camera cal (CT/RAPS video, darks, VIIRS lunar)

The numeric values are not globally unique — ObsID 256 means “SWC-365NM” on the radiometer but “Darks of Darks” on the camera. Because of this, everything in libera_utils.obsids is keyed by (NomHkObsidSource, obsid), never by obsid alone. NomHkObsidSource is the enum that disambiguates which telemetry field a given ObsID was read from.

ObsIdSpec and OBSID_REGISTRY

Each known (source, obsid) pair maps to one ObsIdSpec:

@dataclass(frozen=True)
class ObsIdSpec:
    obsid: int
    source: NomHkObsidSource
    kind: ObsIdKind
    description: str
    trimmed_product: DataProductIdentifier | None
    cal_product: DataProductIdentifier | None

• kind is one of ObsIdKind.RAD_CAL, ObsIdKind.CAM_CAL, or ObsIdKind.SCIENCE. Science/scan modes (e.g. Cross Track, RAP Scan, the Geo scans) are cataloged for documentation but never produce TRIMMED or CAL products — both product fields are None.

• trimmed_product is the L1A NOM-HK-*-FAMILY-TRIMMED DataProductIdentifier produced by the trim step (below) for this ObsID, or None for non-trim-eligible entries. It names a calibration dependency family, not a single ObsID — see below.

• cal_product is the CAL-level DataProductIdentifier a downstream cal-combine algorithm is expected to produce for this ObsID, or None.

OBSID_REGISTRY is the dict[tuple[NomHkObsidSource, int], ObsIdSpec] built from all of these. Look entries up with get_obsid_spec, or iterate the trim-eligible subset with iter_trim_eligible (optionally filtered by source):

from libera_utils.obsids import NomHkObsidSource, get_obsid_spec, iter_trim_eligible

spec = get_obsid_spec(NomHkObsidSource.RAD, 256)
spec.cal_product        # DataProductIdentifier.cal_swc_365nm
spec.trimmed_product    # DataProductIdentifier.l1a_icie_nom_hk_swc_family_trimmed

# All RAD entries that produce a TRIMMED product (excludes science/scan modes):
rad_trim_eligible = list(iter_trim_eligible(NomHkObsidSource.RAD))

get_obsid_spec raises KeyError for a pair that isn’t registered — there is no silent fallback, so an unrecognized ObsID in telemetry surfaces immediately rather than being misfiled.

Calibration dependency families

An ObsID does not get its own TRIMMED product. Several ObsIDs share one: the TRIMMED column names a calibration dependency family, one product per group of ObsIDs that a downstream algorithm processes identically and that therefore share a set of input products. libera_rad runs the same shortwave-LED cal on ObsIDs 256 through 261 regardless of wavelength, so all six share NOM-HK-SWC-FAMILY-TRIMMED. There is one ProcessingStepIdentifier per family, not per ObsID.

Family ProductID	Source	ObsIDs
NOM-HK-GAIN-FAMILY-TRIMMED	RAD	512, 515
NOM-HK-SWC-FAMILY-TRIMMED	RAD	256–261
NOM-HK-LWC-FAMILY-TRIMMED	RAD	320–324
NOM-HK-SOLAR-FAMILY-TRIMMED	RAD	384–395
NOM-HK-LUNAR-FAMILY-TRIMMED	RAD	448, 449
NOM-HK-RAD-VIIRS-LUNAR-FAMILY-TRIMMED	RAD	513, 514
NOM-HK-CT-VIDEO-FAMILY-TRIMMED	WFOV	129, 130, 131
NOM-HK-RAPS-VIDEO-FAMILY-TRIMMED	WFOV	133, 134, 135
NOM-HK-DARKS-FAMILY-TRIMMED	WFOV	256, 257, 258
NOM-HK-WFOV-VIIRS-LUNAR-FAMILY-TRIMMED	WFOV	513, 514

TRIM_FAMILIES is the inverse view of that column, and get_family_specs is how a downstream step enumerates the ObsIDs — and CAL products — it is responsible for:

from libera_utils.constants import DataProductIdentifier
from libera_utils.obsids import get_family_specs

for spec in get_family_specs(DataProductIdentifier.l1a_icie_nom_hk_swc_family_trimmed):
    spec.obsid, spec.cal_product   # (256, cal_swc_365nm), (257, cal_swc_405nm), ...

Two invariants hold, and both are enforced at import time:

• Each ObsID keeps its own CAL product. A family groups inputs, not outputs: a family step maps every ObsID it sees to exactly one distinct CAL product. A duplicate cal_product cell raises ValueError.

• A family never spans both NOM-HK ObsID fields. Trimming scans one field (spec.source.value) at a time, so every trimmed file is attributable to one source. ObsIDs 513/514 (VIIRS lunar) run on both instruments and are registered as two families, NOM-HK-RAD-VIIRS-LUNAR-FAMILY-TRIMMED and NOM-HK-WFOV-VIIRS-LUNAR-FAMILY-TRIMMED, each with its own CAL products.

The ObsID itself is never lost: a trimmed file still carries the ICIE__SW_OBSID_RAD / ICIE__SW_OBSID_WFOV variable it was cut on, so a consumer recovers the exact ObsID from the data rather than from the ProductID.

Consumer 1: the L1A trim step

libera_utils.l1a.nom_hk_trim is the first real consumer. After a daily NOM-HK-DECODED product is produced, calibration pipelines need a per-ObsID subset of it. find_obsid_runs scans the Dataset for contiguous runs of each trim-eligible ObsID (using spec.source.value to pick the right NOM-HK field), and write_trimmed_nom_hk_products writes one NetCDF per run, stamped with that run’s family ProductID:

from libera_utils.l1a.nom_hk_trim import write_trimmed_nom_hk_products

# After parse_packets_to_l1a_dataset(...) for NOM-HK:
trimmed_paths = write_trimmed_nom_hk_products(
    nom_hk_ds,
    output_dir,
    time_variable="PACKET_ICIE_TIME",
    add_archive_path_prefix=True,  # L1A preprocessor / ingest dropbox
)

TRIMMED products reuse the NOM-HK-DECODED variable schema; only ProductID (and thus the filename product token) changes. Because the ProductID names a family, one day normally yields several files sharing a ProductID — six NOM-HK-SWC-FAMILY-TRIMMED granules for six LED ObsIDs, say — told apart by their filename time ranges. Each file still covers exactly one ObsID run.

Normal operations expect each calibration ObsID at most once per day — if the same (source, obsid) appears in multiple disjoint runs, each run is written as a separate file and a warning is logged, since that’s unexpected outside of ground testing. Different ObsIDs of one family are not that case and do not warn.

Consumer 2: downstream cal-combine dispatch

The registry’s second job is letting downstream algorithm repos (e.g. libera_rad) build their own ObsID dispatch without hand-maintaining a duplicate ObsID → product mapping. A downstream repo derives its dispatch table directly from the registry:

from libera_utils.constants import DataProductIdentifier
from libera_utils.obsids import get_family_specs

# Build once at import time: the ObsIDs this step is responsible for, and their CAL outputs
cal_event_by_obsid = {
    spec.obsid: spec
    for spec in get_family_specs(DataProductIdentifier.l1a_icie_nom_hk_swc_family_trimmed)
}

The radiometer cal-combine ProcessingStepIdentifier members (cal-gain-family, cal-swc-family, cal-lwc-family, cal-solar-family) all set shared_ecr_name=CAL_RAD_SHARED_ECR_NAME, so ProcessingStepIdentifier.ecr_name resolves every one of them to the same cal-rad-docker-repo image, and each step’s products are the CAL products of its family’s ObsIDs. A Batch job reads the ObsID out of the trimmed input it was handed and runs the matching algorithm: one container image, and one deployed step per family.

The other L1A inputs a cal step needs

NOM-HK is the only product the L1A preprocessor trims. A calibration algorithm needs more than NOM-HK — the shortwave LED cal, for instance, also needs PEV-SW, PEC-SW, RAD-SAMPLE, CAL-SAMPLE and AXIS-SAMPLE — and those arrive as the full daily L1A granules. The cal container reads the run’s time range off the TRIMMED NOM-HK filename it was handed and subsets them itself.

Which products those are is stored in libera_utils/data/trim_family_inputs.csv. This file maps each family to its L1A inputs, and the two catalog files are cross-checked at import, so a family cannot exist in one without the other:

from libera_utils.constants import DataProductIdentifier
from libera_utils.obsids import get_family_inputs

get_family_inputs(DataProductIdentifier.l1a_icie_nom_hk_swc_family_trimmed)
# (PEV-SW-STAT-DECODED, PEC-SW-STAT-DECODED,
#  RAD-SAMPLE-DECODED, CAL-SAMPLE-DECODED, AXIS-SAMPLE-DECODED)

That tuple, plus the family’s own TRIMMED product, is what a cal-*-family node’s input-products should list in libera_cdk’s processing_system_dag.json. A cal step is expected to take its family’s NOM-HK already trimmed as the TRIMMED product, so required_inputs normally omits the full-day NOM-HK-DECODED granule — listing it would stage a second, redundant NOM-HK input. Families whose dependency set is still undecided return an empty tuple; declaring it is part of closing TODO[LIBSDC-811].

Coverage

• Every radiometer (RAD_CAL) and camera (CAM_CAL) entry has both trimmed_product and cal_product set, so all of them are trim-eligible.

• Science/scan mode entries (Boot-Up, Safe Mode, Stowed, Cross Track, RAP Scan, Along Track, Earth Target, the Geo scans) are catalog-only and never trim-eligible.

• Radiometer cal-combine steps exist for the gain, SWC, LWC and solar families. The lunar (448/449), VIIRS-lunar (513/514) and camera families have no ProcessingStepIdentifier step (TODO[LIBSDC-811]); only the TRIMMED side is wired up for them.

• Camera cal ObsIDs are registered on WFOV only, and the loader enforces that. Science modes are dual-registered, since both instruments assert them, but CT video (129-131), RAPS video (133-135) and the darks (256-258) are asserted only on ICIE__SW_OBSID_WFOV, so get_obsid_spec(NomHkObsidSource.RAD, 129) raises KeyError.

Extending the registry

When a new calibration ObsID needs to be added:

1. Add its CAL DataProductIdentifier member in libera_utils/constants.py. If it belongs to an existing calibration dependency family, reuse that family’s NOM-HK-*-FAMILY-TRIMMED product; only add a new TRIMMED member when the ObsID genuinely introduces a new input dependency, in which case add a ProcessingStepIdentifier for it too.

2. Add a row to libera_utils/data/obsid_registry.csv naming those two members, with a plain-language description (it shows up in error messages and this documentation, so keep it accurate rather than terse). Edit the file with a text editor or the csv module — never a spreadsheet app that may rewrite quoting, since descriptions contain commas. No Python change is needed for an ObsID joining an existing family.

3. If the ObsID joins the family of an existing cal step, add its CAL product to that step’s products list so the step still declares everything it can emit. If instead you added a new TRIMMED family in step 1, add a matching row to libera_utils/data/trim_family_inputs.csv naming the L1A products that family’s algorithm consumes — import fails if the two files disagree. Leave required_inputs empty if the dependency set is not settled yet.

4. Do this here, in libera_utils, first — not as a local dict in a downstream repo. Downstream dispatch tables should be derived from get_family_specs/iter_trim_eligible/get_obsid_spec, matching the pattern in Consumer 2 above.