Version Changes#

5.11.1#

  • FEAT: Add docker-compose entries for spice kernel making.

5.11.0#

  • FEAT: Add libera_utils.l1a.data_time_extractors for per-file science data-time spans without full L1A assembly, covering every DATA_TIME_INDEXED_APIDS member (WFOV SOP FSW image times; RAD/CAL/AXIS sample epoch+period or per-sample times). extract_data_time_range returns None for a WFOV file holding no SOP packet, which is expected when a large image’s mem-dump is chunked across files or downlink passes.

  • FEAT: jpss_sc_pos (APID 11) is data-time indexed. Its span runs from the earliest to the latest time reported by either of the packet’s independently timestamped sample groups (ADGPS ephemeris, ADCFA attitude), so it records the extent of data present rather than the range both groups cover. A consumer needing ephemeris and attitude together must intersect the two sample-time ranges itself.

  • FEAT: Add demuxed ground-test CCSDS support: LiberaGroundCcsdsFilename and DataProductIdentifier.l0_ground_ccsds for the LIBERA_SDC_<apid>_ccsds_<yyyy>_<doy>_<hh>_<mm>_<ss> naming (L0 archive prefix GroundCCSDS/<apid>/<yyyy>/<mm>/<dd>/). Each file holds one APID, read from the basename, with no record header before the CCSDS primary header. The name’s time fields are the bin start the file was cut from, not the span of the packets inside it.

  • FEAT: Add libera_utils.l1a.ground_ccsds.scan_ground_ccsds_file, returning one file’s packet and science data time spans for File Metadata ingest. It returns None, logging the reason, when no packet-time span is possible at all, and returns the packet span with degraded_reason set when only the science data times are unavailable.

  • FEAT: Manual ingest (s3-utils put / manual_ingest_data_products) accepts demuxed ground CCSDS files. verify_ingestion checks archival and the base File Metadata row for these; the searchable row is not verified.

  • FIX: verify_ingestion treats its File Metadata record count as a floor rather than an exact match. A file’s data can span more than one day, giving it a searchable record per applicable date, so an L0 PDS with a multi-day span (as a jpss_sc_pos file can have) no longer polls to timeout on a count of 3 against an expected 2.

  • FEAT: Extracted time spans drop NaT and timestamps outside MIN_VALID_TELEMETRY_TIME-MAX_VALID_TELEMETRY_TIME (defaults 2020-01-01 and 2045-01-01, both config-overridable). A ground capture can carry a packet whose onboard clock has not yet been time-synced, decoding to just after CCSDS_EPOCH (1958-01-01), which would otherwise stretch a span across ~68 years.

  • FEAT: extract_data_time_range and parse_packets_to_l1a_dataset accept an optional skip_header_bytes= override, falling back to SKIP_PACKET_HEADER_BYTES when omitted, so one process can handle both flight and ground files without mutating shared config.

  • FEAT: libera_utils.l1a.packets.expand_sample_times is now public (was _expand_sample_times), and parse_packets_to_l1a_dataset takes ground_data and verbose as keyword-only arguments.

5.10.11#

  • MAINT: The production Docker image no longer sets LEAPSECOND_FILE_ENV. It pointed at a directory containing no naif*.tls, so curryer’s lookup found nothing there; KernelManager.load_naif_kernels sets the variable at runtime to the directory of the LSK it furnishes.

5.10.10#

  • FEAT: KernelManager.ensure_kernel_coverage verifies that the furnished kernels cover every requested target across a time window, raising before a gap surfaces as an obscure SPICE failure deep in a later computation or as silently wrong numbers. Opt-in: call it after furnishing and before the first computation that reads the kernels.

  • MAINT: Require lasp-curryer >= 0.5.2, for curryer.kernels.coverage and the packaged leapsecond kernel that makes LEAPSECOND_FILE_ENV an override rather than a requirement.

5.10.9#

  • BUGFIX: WFOV_FSW_HEADER_AZIMUTH_ANGLE in the ICIE WFOV SCI L1A product is documented as degrees, not radians as it was incorrectly listed before.

  • FIX: Whitespace clean up in trim family csv

  • BUGIX: Test L1A product definition units for Axis-Sample updated to match the actual products

5.10.8#

  • BUGFIX: libera-utils ecr-upload now accepts the local image in the usual Docker form, image-name:image-tag (e.g. ecr-upload l2-comp-flux my-image:1.2.3). Previously this positional argument had to be a bare image name and the tag had to be passed separately, so a name:tag value silently produced a Local image not found: name:tag:latest error. Colons in a registry host:port (localhost:5000/my-image) are not treated as tags.

  • BREAKING: ecr-upload now requires a local image tag instead of defaulting to latest. Algorithm images are built under an explicit docker build -t <name>:<version>, so a local latest usually does not exist and the old default failed later with an opaque Local image not found: <name>:latest — after the L2 Team Role assumption round-trip. Omitting the tag now fails immediately with an explanatory error. Pass my-image:latest explicitly if that really is the image you want. The --ecr-tags default of latest for the remote ECR tag is unchanged.

  • MAINT: The ecr-upload --image-tag option is deprecated in favor of image-name:image-tag. It still works and logs a warning pointing at the new syntax; supplying both is an error unless they agree. push_image_to_ecr accepts image_tag=None and takes the tag from image_name instead, which must then carry one.

5.10.7#

  • BREAKING: The four per-channel radiometer frames (LIBERA_SW_RAD_COORD, LIBERA_SSW_RAD_COORD, LIBERA_LW_RAD_COORD, LIBERA_TOT_RAD_COORD) are replaced by a single co-aligned LIBERA_RAD_COORD frame and one LIBERA_RAD instrument/FOV in the frame and instrument kernels for both JPSS-4 and NOAA-20. The four channels share a boresight, so the per-channel definitions carried identical geometry. Downstream callers must transform to LIBERA_RAD_COORD instead of a per-channel frame.

  • BREAKING: The four per-channel fixed-offset SPK configs are replaced by one libera_rad.fixed_offset.spk.json per mission, so LIBERA_KERNEL_STATIC_SPK_CONFIGS in data/config.json drops from 8 static SPK configs to 5.

5.10.6#

  • BREAKING: L1A trimmed NOM-HK products are now one per calibration dependency family rather than one per ObsID: 10 NOM-HK-<FAMILY>-FAMILY-TRIMMED members replace the 40 NOM-HK-<EVENT>-TRIMMED members added in 5.10.3. Each ObsID keeps its own CAL product, and the ObsID stays readable from the ICIE__SW_OBSID_* variable inside each trimmed file. A family never spans both ObsID fields, so RAD and WFOV VIIRS lunar (513/514) are registered as separate families with distinct ProductIDs.

  • BREAKING: Registry validation now rejects a CAL product claimed by more than one ObsID, and a TRIMMED family registered on both the RAD and WFOV ObsID fields. The 5.10.3 check that a TRIMMED product is claimed by only one ObsID is removed, since sharing one is what defines a family.

  • FEAT: Add four radiometer cal-combine ProcessingStepIdentifier family steps (cal-gain-family, cal-swc-family, cal-lwc-family, cal-solar-family), each declaring every CAL product its family’s ObsIDs can yield. Lunar and camera family steps are deferred (TODO[LIBSDC-811]).

  • FEAT: ProcessingStepIdentifier.ecr_name supports an optional shared ECR name so all radiometer cal steps resolve to cal-rad-docker-repo. Each job dispatches on the ObsID carried by the TRIMMED input it is handed, so no per-job step identity has to be configured separately.

  • FEAT: Add libera_utils.obsids.TRIM_FAMILIES and get_family_specs() so a cal step enumerates the ObsIDs — and CAL products — of its trimmed input family without hand-maintaining the mapping.

  • FEAT: Add data/trim_family_inputs.csv (FAMILY_INPUTS / get_family_inputs()) recording the L1A products each family’s cal step consumes besides its own TRIMMED product; the two together are the input-products set a libera_cdk cal-*-family node should declare. The two catalogs are cross-checked at import.

  • FEAT: Add libera_utils.l1a.nom_hk_trim to detect contiguous ObsID runs in decoded NOM-HK and write one trimmed file per run, stamped with that run’s family ProductID. Several files per day normally share a family ProductID and are told apart by their filename time ranges.

  • FEAT: Add libera_utils.l1a.packet_slicing as the supported way to subset a decoded L1A product. select_packets and slice_l1a_dataset_to_time_window treat PACKET as the driver — whole packets survive or none of them does — and renumber {sample_group}_packet_index against the surviving packet axis, so sample-to-packet traceability holds across a subset. Slicing either axis directly does not.

  • FIX: parse_packets_to_l1a_dataset now sorts the packet axis into its final order before samples are expanded, instead of sorting at the end. The trailing sort permuted PACKET without permuting the sample axes, silently invalidating the {sample_group}_packet_index variables it had already written. Output ordering is unchanged; the indices are now correct.

  • DOCS: Add the ObsID Registry user-docs page, covering the (source, obsid) keying, how downstream repos dispatch cal-combine steps from the registry, and the decision that NOM-HK is the only product the preprocessor trims — cal containers subset the full daily L1A inputs themselves.

5.10.5#

  • FEAT: WFOV SCI (APID 1040) L1A processing stitches complete SOP→EOP images onto CAMERA_TIME, stores compressed JPEG-LS payloads in WFOV_COMPRESSED_IMAGE (uint8/BLOB_BYTE + WFOV_COMPRESSED_IMAGE_LENGTH), and traces packets back to their image with PACKET_IMAGE_ID.

  • FEAT: Decode the 176-byte WFOV header (FSW + FPGA) onto CAMERA_TIME as WFOV_FSW_HEADER_*, WFOV_IMAGE_HEADER_*, WFOV_IMAGE_FOOTER_*, and WFOV_FPGA_STATUS_*; trailing 8-byte NAND footer is validated against a known magic pattern (FooterMismatchCount).

  • FEAT: Add file-level quality attrs PacketCountNotUsedInImages, ErrorFlaggedImageCount, FooterMismatchCount, HeaderParseErrorCount, plus FirstImageIncomplete / LastImageIncomplete flagging expected truncation at a packet window’s leading/trailing edge (PacketCountNotUsedInImages counts only genuine mid-stream anomalies).

  • BREAKING: CAMERA_TIME now contains one row per complete stitched image only (not every qualifying SOP). Removed redundant WFOV_IMAGE_COMPLETE.

  • BREAKING: ICIE__WFOV_DATA is dropped from the WFOV L1A product entirely — every packet’s raw payload is by then either duplicated in WFOV_COMPRESSED_IMAGE or unusable from this granule alone. PACKET_IMAGE_ID is the only remaining per-packet trace-back.

  • FEAT: WFOV SCI processing now raises ValueError when no complete image can be stitched. The raise happens before ICIE__WFOV_DATA is dropped, so the raw packets survive for diagnosis, and the message carries all stitch counters.

  • FIX: A structurally complete image whose 176-byte header cannot be decoded is now discarded and counted in HeaderParseErrorCount, rather than emitted with a NaT CAMERA_TIME that crashed filename generation. CAMERA_TIME is therefore always free of NaT.

  • FIX: SINGLE-flagged WFOV packets (whole image in one packet) are unsupported and were silently misattributed to window-edge truncation. They now log a warning and count toward PacketCountNotUsedInImages; a SINGLE mid-collection abandons the image in progress.

5.10.4#

  • FEAT: Register ADM-specific standard dimensions in libera_dimensions.yml (viewing-geometry, surface-type, and cloud/wind property bins for ERBE-like and TRMM-like ADM products)

5.10.3#

  • FEAT: Replace the eight alpha calibration DataProductIdentifier members (cal_solar_face{1,2,3}_combined, cal_lw_temp{1,2,3}_combined, cal_gain_combined, cal_sw_combined / SOLAR-FACE1-COMBINED, LW-TEMP1-COMBINED, GAIN-COMBINED, SW-COMBINED, …) with one CAL product per calibration ObsID (cal_gain/GAIN, cal_swc_365nm/SWC-365NM, cal_lwc_310k/LWC-310K, cal_solar_ssw_pri/SOLAR-SSW-PRI, the lunar/VIIRS-lunar and camera cal products, etc.). Downstream consumers must move to the new members; no aliases are kept.

  • FEAT: Add ObsID-trimmed NOM-HK DataProductIdentifiers (NOM-HK-<EVENT>-TRIMMED), one per calibration ObsID, produced by L1A Step 1 preprocessing and consumed by the per-ObsID cal-combine steps.

  • FEAT: Add libera_utils.obsids and the libera_utils/data/obsid_registry.csv catalog: the ICIE software ObsID registry mapping (NomHkObsidSource, obsid) to its TRIMMED and CAL ProductIDs, with get_obsid_spec and iter_trim_eligible for downstream dispatch. The CSV is validated at import (unknown member names, wrong data level, malformed rows, kind/source mismatches, duplicate ObsIDs, and TRIMMED products claimed by more than one ObsID all raise ValueError).

5.10.2#

  • BUGFIX: enforce_data_array_conformance no longer emits a UserWarning (or log line) for every encoding key present on a DataArray but absent from the product definition. Extra keys (typically xarray/NetCDF runtime metadata such as source, original_shape, preferred_chunks, compression stubs) are dropped silently; conflicting values for keys that are defined still warn. This removes thousands of spurious warnings when packaging products built from opened L1A NetCDF inputs.

5.10.1#

  • FEAT: ecr-upload now registers the algorithm version(s) it uploads. After pushing, it emits a NewAlgorithmImage event (one per non-latest ECR tag) to the SDC event bus so the Registrar creates the corresponding versioned AWS Batch job definition. Adds --verify (blocks until both the ECR image and the Batch job definition are confirmed present, up to --timeout seconds).

  • FEAT: Add the register-algorithm-image CLI for emitting a NewAlgorithmImage event for an image already in ECR (--image-digest/--verify/--timeout). The new aws/algorithm_registration.py module provides put_new_algorithm_image_event and verify_algorithm_registration (checks the ECR image exists and an ACTIVE Batch job definition references it).

  • FEAT: Quieter ecr-upload push logging — per-layer progress ticks are suppressed, a per-tag summary and the resulting image digest are logged at INFO, and -v/--verbose enables full DEBUG push detail. push_image_to_ecr now returns a {tag: digest} mapping.

  • MAINT: AWS region for the ECR/EventBridge/Batch operations is now taken from the session’s AWS configuration (profile / AWS_REGION) instead of a hard-coded us-west-2; a clear error is raised if no region is configured, and --verify warns when the image URI’s region differs from the session region.

  • DOCS: Document register-algorithm-image and the new ecr-upload flags in the usage docs.

5.10.0#

  • FEAT: Kernel generation applies the measured Libera opto-mechanical frame misalignments from OAV3 ground testing. The Az/El mechanism CKs now encode rotation about the measured axes of rotation, and the radiometer frames carry the measured line-of-sight boresight, so the generated kernels reflect the true instrument geometry instead of nominal placeholders. The three source unit vectors (LIBERA_AZ_AOR_IN_STAND, LIBERA_EL_AOR_IN_STAND, LIBERA_EL0_Z_IN_STAND) are stored as keywords in the frame kernel; the Az/El CK configs switch from Euler angles to quaternions, and kernel_maker.add_mechanism_ck_quaternions builds the per-sample rotations about the measured axes (applied after the encoder correction).

5.9.3#

  • FEAT: Az/El mechanism CKs now apply deterministic sinusoidal encoder-angle corrections during kernel generation, so the stored quaternions reflect the true mechanism rotation rather than the raw encoder readout. Corrections are applied to the filtered encoder angles in create_kernel_from_l1a (L1A telemetry is left unchanged), with matching forward/inverse helpers (correct_azimuth/correct_elevation, uncorrect_azimuth/uncorrect_elevation, and DataFrame-level apply_encoder_corrections/reverse_encoder_corrections)

  • BUGFIX: Remove np.NaN as curryer no longer pins Numpy < 2

5.9.2#

  • FEAT: Expand the RBSP+VIIRS imager DataProductIdentifier set (l2_unf_rad_imager, l2_nb_bb_imager_camtime, l2_toa_flux_imager, and matching AUX scene-ID/FMATCH/ADM members)

  • BUGFIX: rename camera/imager DPI and ProcessingStepIdentifier members for consistency with product strings: l2_unf_cam→l2_unf_rad_cam/l2-unf-rad-cam, l2_cf_rad_time/l2_cf_rad→l2_cf_cam/l2-cf-cam, l2_cf_cam_time/l2_cf_cam→l2_cf_cam_camtime/l2-cf-cam-camtime, l2_nb_bb_cam_time/NB-BB-CAM-TIME→l2_nb_bb_cam_camtime/NB-BB-CAM-CAMTIME/l2-nb-bb-cam-camtime, l2_unf_imager→l2_unf_rad_imager/l2-unf-rad-imager, l2_nb_bb_imager→l2_nb_bb_imager_camtime/l2-nb-bb-imager-camtime

5.9.1#

  • FEAT: Scene Identification now reports the property bin min/max bounds alongside the scene ID. For each footprint and classification variable, identify_scenes adds scene_bin_{type}_{variable}_min/_max variables giving the bounds of the matched scene’s bin (unbounded sides and unmatched footprints reported as NaN). Controlled by the new report_bin_bounds flag (default True) on FootprintData.identify_scenes and SceneDefinition.identify_and_update; existing scene_id_{type} columns are unchanged.

  • FEAT: Add SceneDefinition.get_bin_bounds_for_scene_id and Scene.get_bin_bounds to look up the property bin bounds for a given scene ID.

5.9.0#

  • BREAKING: Rename DataLevel.ANC to DataLevel.AUX and split ancillary archive routing into separate AUX and CAL archive buckets.

  • BREAKING: Replace provisional L2/ANC DataProductIdentifier and ProcessingStepIdentifier values with the camera cloud-fraction and RBSP+VIIRS imager product sets for July delivery (renamed product strings, new AUX/L2 members, and updated L2 team IAM role mappings).

  • DOCS: Update user docs for the new product/level identifiers.

5.8.6#

  • FEAT: Manual processing now runs through the SDC event bus instead of triggering Step Functions directly. step-function-trigger emits a ManualProcessing event for a single step, and a new manual-processing CLI submits arbitrary custom DAGs (or the default DAG) across one or more dates, with an optional --verify that polls the Coordination Table.

  • FEAT: ecr-upload assumes the algorithm’s per-team L2 Team Role for L2/ADM images so non-admin L2 developers can push to their ECR repos; non-L2 steps use the default/--profile session unchanged.

5.8.5#

  • FEAT: s3-utils put now performs manual SDC data ingest instead of a direct archive upload. It accepts multiple file paths, stages each file to the SDC Ingest Dropbox bucket, and emits a single NewFilesAvailable event to the SDC event bus; the SDC Data Ingester then handles archiving and metadata/data-availability records. Accepts L0 and data product filenames (manifests rejected).

  • FEAT: Add get_l2_team_role_session (in aws/utils.py) which builds a boto session from a profile/AWS_PROFILE and assumes an L2 team role (default L2Developer/LiberaUtils), raising ValueError if the base profile is not permitted to assume it. s3_put_cli_handler now uses it as its session source.

  • FEAT: Add --verify/--timeout to s3-utils put. With --verify, the CLI blocks after emitting the ingest event and polls (read-only) until each file is confirmed in its archive bucket and recorded in the File Metadata table (and, for non-L0 products, the Data Availability table), logging a per-file summary and raising on timeout (default 5 minutes). Adds find_dynamodb_table_in_account_by_partial_name to aws/utils.py.

  • MAINT: Remove the unused libera_utils/db/dynamodb_utils.py module (stale write-side helpers and an unused get_dynamodb_table); the ingest-verification DynamoDB reads are inlined in verify_ingestion.

5.8.4#

  • FEAT: KernelManager.load_naif_kernels also loads an extended Earth PCK (earth_*_predict.bpc via NAIF_EARTH_EXTENDED_PCK_REGEX) so ITRF93 orientation is available beyond the short high-precision earth_000101_*.bpc window (required for JPSS geolocation and future-epoch processing).

5.8.3#

  • FEAT: Add NOM-HK array groups to stack waypoint status and sequence execution fields into indexed arrays (ARRAY_128 and ARRAY_8)

  • BUGFIX: Normalize Unicode packet string fields during aggregation to ensure deterministic byte sizing

  • NOTE: NOM-HK L1A changes the shape and dtype of five variables (replacing per-index fields with indexed arrays). Treat this as breaking for external consumers of those variables; a patch bump was chosen during early development. Changed variables: ICIESW_FP_WP_ST_WP, ICIESW_SEQ_EXEC_BUF_OP, ICIESW_SEQ_EXEC_POS_OP, ICIESW_SEQ_ST_OP, ICIE__SW_SEQ_STOP_CD_OP

5.8.2#

  • FEAT: Faster ICIE WFOV SCI (APID 1040) packet-to-L1A by refreshing icie_xtce_tlm.xml and combining the 972-field WFOV aggregation_groups block from icie_wfov_sci in l1a_processing_configs.yml.

5.8.1#

  • FEAT: Added alpha calibration combination constants for data products

5.8.0#

  • BREAKING: KernelManager.load_libera_dynamic_kernels accepts only a Sequence[str | pathlib.Path | S3Path] (manifest-ordered path list). Directory expansion, scalar Path, and scalar local str are not accepted (use [path] for a single file). Materialization is inlined with a single normalization pass per entry; _materialize_dynamic_kernel_paths / _is_dynamic_sources_sequence are removed from the dispatch path.

  • BREAKING: Renamed load_libera_dynamic_kernels argument dynamic_kernel_directory to dynamic_kernel_sources (broader semantics; no compatibility alias).

  • FEAT: KernelFileCache can materialize kernels from local filesystem paths (Path or non-HTTP str), with documented resolution rules for relative paths and stable cache freshness after copy.

  • FEAT: KernelManager caches Libera static kernels under the versioned user cache via KernelFileCache after building them under the existing short temporary directory. When every required artifact is already cached (and within cache_timeout_days), static kernel creation is skipped and kernels are furnished from the cache, consistent with NAIF generic kernels.

  • Improve KernelFileCache docstrings and spice_utils type annotations (including ensure_spice overloads and tighter return types on kernel helpers).

5.7.1#

  • BUGFIX: Fix packet parsing to l1a of datasets with deduplicated timestamps for final data assembly

  • FEAT: Added ground data option for deduplication to allow ground data processing to continue

  • BUGFIX: With ground_data=True, non-identical duplicate coordinates no longer raise when verbose=False; warnings only if verbose=True.

  • BUGFIX: Duplicate-coordinate diagnostics no longer cite the source sequence counter when that coordinate is absent from the dataset slice (e.g. inside sample groups).

5.7.0#

  • FEAT: Added metadata writer to make UMM-G metadata files from NetCDF-4 data files

5.6.0#

  • FEAT: Add dimension validation to data product definition

  • Support safer enforcement of product definition specs on data products

  • Improve logging and warning visibility in data product writer

5.5.6#

  • FEAT: Adding support for APIDs 1000, 1002, 1035, 1043, and 1044 to support calibration and status telemetry packets

5.5.5#

  • CONFIG: Update the ICIE TLM XTCE to latest version

5.5.4#

  • BUGFIX: Implement KernelManager for Make_Kernel, furnishing necessary kernels

  • REFACTOR: Port Make_Kernel and SPICE-based time conversions to spice_utils to avoid circular imports

5.5.2#

  • BUGFIX: Improve resiliency of JSON logging system for cloudwatch logs

5.5.1#

  • BUGFIX: Fixed the netcdf writing needed for L1B camera product and clarifying netcdf engine usage

5.5.0#

  • REFACTOR: Adjusted CLI interface for AWS tools to improve usability and consistency

  • FEAT: Allows the usage of the profile argument in the cli tools for specifying AWS credentials profiles

  • REFACTOR: Changing the s3 utilities for list and put to use the new constants tools more effectively

5.4.8#

  • FEAT: Add lookups between L1a, APID, and L1a product definitions

5.4.7#

  • FEAT: Update Space Packet Parser dependency to take advantage of packet filtering functionality

5.4.6#

  • BUGFIX: update xarray version due to breaking change in netcdf writing

5.4.5#

  • FEAT: Add enforcement of valid versioning in Filename classes

5.4.4#

  • FEAT: Allow passing dynamic product level attributes to data product writer

5.4.3#

  • FEAT: Allow adding the archive prefix to the path when calling write_libera_data_product

5.4.2#

  • Change xarray netcdf engine to h5netcdf because netcdf4 does not support writing to filelike objects (e.g. opened files).

5.4.1#

  • Added a KernelManager class to handle SPICE kernel caching and retrieval

5.4.0#

  • REFACTOR: Refactor kernel_maker to use L1a Dataset object. Packet-based kernel creation is still possible via create_kernels_from_packets, which runs L1a processing internally

  • CONFIG: Update Curryer kernel configs to match Libera AXIS_SAMPLE packets fields

  • Move the L1A processing configurations to YAML instead of Python objects

  • BUGFIX: Fix bug in packets.py::_drop_duplicates to allow dropping based on coordinate or dimension

5.3.0#

  • FEAT: scene_id module alpha release capable of identifying scenes from footprint data in Ceres SSF files.

5.2.2#

  • Updated the Curryer version pin to use at least 0.2.0, which is when the problematic subdependencies were removed

5.2.1#

  • Added scene_id module for processing footprint data to produce datasets with assigned scene IDs according to libera_utils standard definitions.

5.2.0#

  • FEAT: Implement processing to L1A Dataset from packets

  • CONFIG: Add full ICIE XTCE to Libera Utils package

5.1.0#

  • Added UMMGranule Pydantic class to validate datasets against the UMM-G schema

  • Added methods to convert Libera datasets to UMMGranule instances

5.0.0#

  • FEAT: Add write_libera_data_product to netcdf.py for one-shot product writes from numpy arrays

  • BREAKING: Remove L1A ProcessingStepIdentifier (node) enum constants

  • BREAKING: Refactor DataProductDefinition (previously DataProductConfig) and product definition validation

  • MAINT: Remove hdf.py and associated tests

4.0.0#

  • BREAKING: Removed L0 file chunking support: The chunk_number parameter and related methods have been removed from L0 data product identifiers

  • BREAKING: Refactored identifier classes: Moved DataProductIdentifier and ProcessingStepIdentifier from libera_utils.aws.constants to libera_utils.constants

  • BREAKING: Removed deprecated classes: ProductName, CkObject, and SpkObject enums have been removed and consolidated into DataProductIdentifier

  • BREAKING: SPICE kernel filename consolidation: Merged separate SPICE kernel filename classes into LiberaProductFilename

  • Enhanced DataProductIdentifier: Now includes embedded metadata with processing level information and simplified lookup methods

  • Enhanced ProcessingStepIdentifier: Now includes embedded product relationships and improved level derivation

  • New DataLevel enum: Added structured processing level enumeration with built-in archive bucket name mapping

  • Filename validation: Updated L0 filename patterns to remove chunk number validation

  • Improved type safety: Better type hints and validation throughout identifier classes

  • Updated documentation: Refreshed user documentation to reflect new file naming conventions

  • Added the Libera APID values we care about for science processing to the LiberaApid enum class

3.2.4#

  • Changed the SPICE kernel creation API

  • Added the SPICE kernel creation Docker image

  • Added the Geolocation Tier-1 integration test case and data files

3.2.3#

  • Improved ECR authentication issues with the libera-utils ecr-upload CLI command for working with SSO authentication

  • Updated S3 list tool to print by default

  • Improved tooling of the aws constants module for both DataProductIdentifier and ProcessingStepIdentifier

  • Added step function and policy names to ProcessingStepIdentifier

  • Updated constants for L2 data products

  • Updated step function trigger tool with url output and naming improvements

  • Exposed useful L2 tools at highest level of the libera_utils package

3.2.2#

  • Added dimension handling for data product configuration

  • Added internal DataArray data storage matching to known dimensions

  • Added jenkins file for publishing to PyPI

  • Added contact listserv for project

  • Added json and yaml linting to the pre-commit hooks

  • Added prettier configuration for formatting markdown, json, and yaml files

  • Added internal Dataset object to the Configurable NetCDF-4 metadata object

  • Added writing of output file for the Configurable NetCDF-4 data product object

3.2.1#

  • Added automatic formatting to the repository using pre-commit hooks

  • Added a list-serv for ci/cd output as part of the Jenkins files

  • BUGFIX: Fixed filenaming changes from 3.2.0 that affected the CDK deployment and usage

3.2.0#

  • Added pydantic models for a Configurable NetCDF-4 metadata object

  • Includes static metadata that is populated and known by these tools

  • Includes dynamic metadata that must be provided by user of these tools

  • Changed the naming scheme of the SPICE data files to include SPK or CK

3.1.1#

  • BUGFIX: Testing improvement for mocking docker image building

3.1.0#

  • Added Curryer (lasp-curryer) library dependency for SPICE kernel creation and geolocation

  • Added static SPICE kernels and configuration files for geolocating NOAA-20 / CERES

  • BREAKING: Removed geolocation submodule, replaced by interfaces within Curryer

3.0.0#

  • BREAKING: Removed support for python version 3.9 and 3.10

  • BREAKING: Updated the aws constants ProcessingStepIdentifier dump method to be to_str_with_chunk_number

  • Improved the standardization of CLI commands to call cli_handler functions that wrap the main functionality

2.5.2#

  • Added the s3-utils cli interface with subcommands put, list, and cp for ease of use s3 interactions

  • The cli subcommand libera-utils s3-utils put will upload a file to the correct S3 archive bucket given an algorithm

  • The cli subcommand libera-utils s3-utils list will list all files in a given S3 archive bucket for a given algorithm

  • The cli subcommand libera-utils s3-utils cp will copy a file from one S3 archive bucket to another location

  • Added the Libera Archive bucket naming pattern to the aws constants module

  • Improved type hinting for the smart_open module

  • Improved readability of cli testing

  • Removing testing against python 3.9 and 3.10

2.5.1#

  • BUGFIX: missing pyyaml dependency that prevented usage of logutil module

2.5.0#

  • Reimplement Manifest class as a pydantic model and integrate with dependent code

  • BREAKING: Remove the deprecated add_file_to_manifest method from Manifest class

  • BREAKING: Rename outpath to out_path on the Manifest class write method

2.4.5#

  • Add --ecr-image-tags option to libera-utils ecr-upload CLI for tagging remote algorithm images in an ECR

  • Add validation and serialization methods to the DataProductIdentifier and ProcessingStepProductIdentifier enums

2.4.4#

  • Allow overriding the standard docker config.json file with a minimal file for ECR uploading to prevent ECR upload permission failures from cached docker login credentials from CDK deployments

2.4.3#

  • Changes to ecr_upload to support programmatic building and pushing of Docker images

  • Remove DynamoDB docs (moved to libera_cdk)

2.4.2#

  • BREAKING: Remove the AnyFilename polymorphic class. Please use AbstractValidFilename.from_file_path()

2.4.1#

  • Updating requirements of methods to use keyword arguments rather than positional arguments

  • Adding ProcessingStepIdentifier and DataProductIdentifier to the filenaming classes

  • Updating ecr names to work with the completion checker testing in libera_cdk

2.4.0#

  • Add properties to filenaming classes to retrieve data_product_id and processing_step_id

  • Add ProcessingStepIdentifier and DataProductIdentifier standardization to be used by downstream repos

2.3.1#

  • Fix os.path.join bug in filenaming module that broke mocked S3 paths and also fix typehinting

2.3.0#

  • Create CLI tools for AWS ECR image upload and Step Function triggering

  • Update manifest filenames to use ULID instead of timestamp for unique identifiers

  • Change logutil configure_task_logging to optionally log JSON to console

  • Allow configure_task_logging to optionally propagate DEBUG messages from specific loggers

  • Update documentation for how the database is used in the Libera project in DynamoDB

  • Create tools for DynamoDB in AWS for .pds files (CONS and PDS)

  • Replace the use of PostgreSQL with DynamoDB for the Libera project

2.2.0#

  • Add AnyFilename polymorphic class

  • Change filename of all products to a LiberaDataProductFilename that inherits from AnyFilename

  • Update filenaming convention to be all capital letters

  • Improve API for manifest module

  • Add prefixing to Filename classes for predictable archive paths

  • Add prefixing for manifest files for predictable and navigable paths in s3 buckets

  • Update git to include lfs and move test data to lfs

  • Improve database manager including caching improvements

  • Improve smart_copy_file and bug fixes to smart_open testing

  • Refactoring and improving pds ingest for database entries and integration testing in CDK

  • Added handling of construction records and pds files appropriately when ingesting

    • This includes reading a construction record and removing the pds file entry for the construction record itself

  • Improved testing of pds ingest and pds file orm models to more accurately reflect use cases

  • Added output manifest creation from input manifest to match timestamps in filenames of input and output manifests

  • Refactored pds ingest to use AnyPath objects for handling file locations

  • Added error handling to pds ingest

2.1.1#

  • Update dependency specification to speed up dependency resolution wrt botocore/urllib3

  • Improve database initialization to work with libera_cdk changes

  • Fix bug in Dockerfile that incorrectly set the default entrypoint

  • Add preliminary instrument kernel

2.1.0#

  • Improve API to Manifest and Manifest.add_files

  • Add manifest filename enforcement to Manifest class

  • Update filenaming conventions for product filenames and SPICE kernels

  • Allow adding an s3 bucket/prefix as a basepath for filenames

2.0.1#

  • Remove the extras dependency spec because of the way SQLAlchemy imports models

2.0.0#

  • Add filenaming classes

  • Add manifest file class

  • Add construction record parser

  • Update DB schema to store construction records

  • Update kernel generation CLI to use manifest file pattern

  • Shift database and spice related libraries to extras (not installed by default)

  • Add smart_copy_file function that can copy files to and from S3 and filesystem locations transparently

  • Remove HDF-EOS5 filehandling code

  • Add quality flag classes

  • Change license to BSD3

1.0.0#

  • Stub out project structure

  • Add build and release processes to readme

  • Switch to Poetry for project dependency configuration and build management

  • Add geolocation module

  • Add tools in spiceutil module for caching SPICE kernels from NAIF

  • Add missing unit testing coverage

  • Add spice.md documentation on how the package uses and manages SPICE kernels

  • Add database tooling, dev database, and ORM setup

  • Add smart_open for opening local or S3 objects

  • Add logging utility functions for setting up application logging