Changelog#

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased#

5.25.0 - 2026-11-11#

Added#

  • libera_utils.footprint_matching.product assembles and writes the FMATCH products for all four operational modes (assemble_fmatch_dataset, write_fmatch_product, aggregate_external_variables, build_radiometer_footprints, compute_derived_viewing_geometry).

    • Geolocation and viewing angles pass through from L1B, keyed by their L1B RAD-4CH names (L1B_PASSTHROUGH_VARIABLES), which are also the product names. PSF_Bbox_*, Sunglint_Angle, PSF_Coverage_Fraction, the aggregated external variables and the CERES 75%/95% coverage bits in Quality_Flag are computed. Sunglint_Angle uses the L1B relative azimuth convention, so it is zero at the specular direction.

    • Off-limb records are kept and written as fill with zero coverage. Footprints are weighted with AngularPSFWeigher by default; pass RadialWeigher for footprints without a cone-angle rate.

    • Missing inputs raise rather than write fill: a variable no input produced raises ValueError unless allow_placeholders=True, and an unusable ephemeris raises MissingGeolocationError. Camera-timescale modes cannot be aggregated yet.

    • Integer category and pixel-index variables use -1 (FMATCH_INTEGER_FILL_VALUE) as fill; bitmasks such as Quality_Flag use 0.

  • aggregate_external_variables runs on a process pool when the FMATCH_WORKERS configuration key or environment variable is above one, with the same output as a single-process run.

  • AncillaryReader.preload() loads a reader’s whole-run data before the first query.

Changed#

  • lasp-curryer requires >= 0.5.3, <1.0, for curryer.compute.spatial.relative_azimuth.

5.24.0 - 2026-11-11#

Added#

  • Add libera_utils.aws.ecr_upload.ImageTagAlreadyExistsError, a ValueError that push_image_to_ecr and ecr-upload raise when a requested tag other than latest already exists in the ECR repository, whatever image it holds: found by a check before anything is pushed, or refused by ECR during the push when another push took the tag after that check. The message names every such tag and the repository, links the repository’s page in the AWS console, and says to push under a new version tag, or to delete the image under the tag by hand and push again. A refused push is not retried, and ecr-upload registers nothing.

  • Product definitions for the FMATCH-CAM(-CAMTIME) and FMATCH-IMAGER(-CAMTIME) products. FMATCH-IMAGER also carries the RBSP CLDPIX/FLASHFlux cloud fields and the extended SSF cloud/aerosol/albedo variables (multi-axis SSF flattened to 1-D per footprint, carried by IMAGER alone through their VariableSpec.modes), and the five ERA5 pressure-level fields and their standard deviations as 2-D variables on (RADIOMETER_TIME, PRESSURE_LEVEL) (e.g. ERA5_Pressure_Temperature) with a PRESSURE_LEVEL coordinate in hPa. PRESSURE_LEVEL (size 37) joins the standard dimensions. A reader declares such stacked product variables with PRODUCT_STACKS (ProductStack), and AncillaryReader.product_stacks() adds each stack’s standard-deviation companion stack; a stack that names a spec the reader does not produce, stacks a spec twice, takes a product spec’s name or mixes members with and without companions raises TypeError when the reader class is defined. Cloud_Fraction_Camera in FMATCH-CAM(-CAMTIME) is defined in percent [0, 100] to match the CF-CAM input. FMATCH-CAM(-CAMTIME) carry no ERA5 10 m winds or VIIRS BRDF kernel weights, since no consumer of the CAM products reads them. Every integer category, pixel-index and camera pixel-bound variable declares _FillValue: -1, since 0 is a real value in each; the Quality_Flag bitmask and the PSEUDOFOOTPRINT index declare none. Dimensionless variables carry no units attribute (CF §3.1), and VIIRS_CLDPROP_D3_Cloud_Optical_Thickness carries the CLDPROP_D3 source’s combined-phase long_name and its valid_min/valid_max [0, 150].

  • libera_utils.footprint_matching.aggregation.stack_product_values collects a reader’s stacked specs into one array per stacked product variable and raises ValueError when a member’s scalar is missing.

Changed#

  • BREAKING: Before pushing, push_image_to_ecr and ecr-upload check ECR for every requested tag other than latest and push nothing if any of them already exists, even in a repository that would accept the push, which used to replace the image under the tag. A session that pushes a version tag now needs ecr:DescribeImages on the repository, and a missing repository fails the check before any push. Tags other than latest are then pushed before latest, so a tag taken by another push since the check leaves latest where it was.

  • BREAKING: SCENE-ID-CAM and SCENE-ID-CAM-CAMTIME variables take their product names (Scene_ID_ERBE, Scene_ID_Unfiltering, Scene_Bin_ERBE_Cloud_Fraction_Min, Scene_Bin_ERBE_Solar_Zenith_Surface_Min, Surface_Type, IGBP_Surface_Type, Cloud_Fraction, Solar_Zenith_Surface, Camera_Pixel_X_Min, …). FootprintData holds each variable a product writes under its product name (the FootprintVariables values), so create_and_write_data_product writes the declared variables as they are.

  • BREAKING: The VIIRS BRDF reader’s variables drop the repeated brdf_ prefix (brdf_shortwave_fiso → Shortwave_Fiso, and likewise for each band and kernel weight), and the VIIRS AOD reader’s aod_550 is Optical_Depth_550, so the FMATCH product variables are VIIRS_BRDF_<Band>_<Param> and VIIRS_AOD_Optical_Depth_550.

  • BREAKING: FMATCH reader READER_KEYs are their product-variable prefixes (CLDPIX, ERA5_Pressure, ERA5_Surface, FLASHFlux, IGBP, NISE, VIIRS_AOD, VIIRS_BRDF, VIIRS_CLDPROP_D3), so ReaderRegistry.get, FMATCH_MODE_READERS and the source_file_paths keys of build_sources use them. Reader spec names are Camel_Case (Sea_Ice_Concentration, Surface_Type_Primary, Temperature_500hPa), and STANDARD_DEVIATION_SUFFIX is _Standard_Deviation.

  • BREAKING: The shipped scene definitions are ERBE.csv, TRMM.csv and Unfiltering.csv. Their columns end in _Min/_Max and name the classification variables Surface_Type, Cloud_Fraction, Solar_Zenith_Surface, Viewing_Zenith_Surface and Relative_Azimuth_Surface. SceneDefinition.type is the CSV filename stem as spelled, and its outputs are Scene_ID_{type} and Scene_Bin_{type}_{variable}_Min/_Max. A column ending in lower-case _min/_max raises ValueError.

  • SCENE-ID-CAM and SCENE-ID-CAM-CAMTIME take the L1B RAD-4CH and L1B CAM long_names for the variables they pass through from L1B (Solar_Zenith_Surface, Viewing_Zenith_Surface, Relative_Azimuth_Surface, and in SCENE-ID-CAM-CAMTIME Latitude, Longitude, Altitude), and their Cloud_Fraction long_name says the CF-CAM values are in percent rather than scaled to percent. Units and values are unchanged.

Removed#

  • BREAKING: OperationalMode.IMAGER_FLASH (FMATCH-IMAGER-FLASH). No FMATCH-IMAGER-FLASH product is planned, so the CERES FLASHFlux reader feeds FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME only.

  • BREAKING: The ERA5 surface reader’s 10 m winds and the VIIRS BRDF reader no longer serve the CAM modes: ERA5_Surface and VIIRS_BRDF feed FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME only, so they leave the CAM entries of FMATCH_MODE_READERS, and build_sources for a CAM mode rejects their source_file_paths keys.

5.23.0 - 2026-11-05#

Added#

  • Add the Packet Ordering page to the user guide, which describes how the L1A PACKET axis is put in SRC_SEQ_CTR order, what raises and warns, and the losses the counter cannot size. It replaces the packet axis order section of the L1A Processing page.

  • Add libera_utils.l1a.packet_ordering.check_packet_order(ds, packet_time_coordinate), which emits a UserWarning where packet time steps backward along the PACKET axis, with the count, the largest step and the first one’s counters and times. It raises ValueError only when SRC_SEQ_CTR or the packet time variable is missing.

  • Add libera_utils.l1a.packet_ordering.sort_by_sequence_counter(ds, packet_time_coordinate), which orders the PACKET axis by SRC_SEQ_CTR and drops repeated packets as parse_packets_to_l1a_dataset does; it reorders by counter only between packets within REORDER_WINDOW (10 s) of each other, places packets by packet time across a counter step that repeats or goes backward further apart, and logs each such cut at INFO with the counters and packet times either side of it; when it reorders, it consumes its input, removing the PACKET variables from the dataset passed in, so use the returned dataset. Also add libera_utils.l1a.packet_ordering.out_of_order_positions(ds, packet_time_coordinate), which returns the PACKET positions where SRC_SEQ_CTR repeats or steps backward between packets within 10 s of each other.

Changed#

  • BREAKING: parse_packets_to_l1a_dataset orders the PACKET axis by SRC_SEQ_CTR instead of by packet time, whatever the order of the input files; packet time only resolves which counter wrap a packet falls in. Published {group}_packet_index, CAMERA_PACKET_INDEX and PACKET_IMAGE_ID values change, and packet time is not monotonic along PACKET; sort packet times before numpy.searchsorted or numpy.interp. Input files are parsed in file name order, which input_files lists.

  • BREAKING: parse_packets_to_l1a_dataset drops duplicate packets by SRC_SEQ_CTR instead of by packet time, so a distinct packet that shares a packet time with another is kept. Packets that share a packet time but not a counter are no longer treated as duplicates. Packets that share a counter but differ raise ValueError unless ground_data=True, which keeps the first. The duplicate warning reads “Dropped N packet(s) received twice: same SRC_SEQ_CTR and contents as a kept packet”, or, when ground_data=True keeps the first of packets that differ, “Dropped N packet(s) that repeat a kept packet’s SRC_SEQ_CTR” with the counts of each.

  • BREAKING: nom_hk_trim.find_obsid_runs raises ValueError for a NOM-HK dataset without PACKET_ICIE_TIME or SRC_SEQ_CTR, or whose SRC_SEQ_CTR repeats or steps backward between packets within 10 s of each other, which is the case for a NOM-HK product written in packet-time order by an earlier release; decode its L0 files again.

  • BREAKING: parse_packets_to_l1a_dataset raises ValueError where packets adjacent in an input file with consecutive SRC_SEQ_CTR would be placed a counter wrap apart, for ground and flight data alike. A single packet whose packet time would place it a wrap away from both such neighbors is instead dropped, with a UserWarning naming its counter and packet time. This includes a join between two input files whose names sort against the time order of their data, when the file that sorts first, holding the later data, ends one counter below where the next file starts.

Removed#

  • BREAKING: Remove packets.SRC_SEQ_CTR_DIMENSION.

Fixed#

  • LiberaDataProductDefinition.generate_data_product_filename takes the start and end from the minimum and maximum of the time variable, not its first and last values.

5.22.0 - 2026-11-04#

Changed#

  • An output-products entry in a manual-processing custom DAG may be a mapping with id and optional, as well as a bare data product id string. optional, on an input or an output, needs an SDC deployment that includes libera_cdk PR #96 (LIBSDC-892); the Job Creator of an earlier deployment rejects an output mapping and ignores optional on an input.

5.21.0 - 2026-11-02#

Added#

  • Add libera_utils.l1a.day_window. day_bounds(day) returns the day and its buffered window, DAY_BUFFER either side, as timezone-aware UTC datetimes. trim_l1a_to_day_window(ds, day=) keeps the packets in the closed window [D − DAY_BUFFER, D + 1 + DAY_BUFFER] (10 minutes) through slice_l1a_dataset_to_time_window, selecting on every sample group and keeping whole packets. check_data_times(ds) logs a warning for each data-time axis that repeats a time or steps backwards; it does not raise.

  • packet_slicing handles WFOV SCI CAMERA_TIME: an image selected on camera time keeps every packet from SOP through EOP, CAMERA_PACKET_INDEX is renumbered and keeps its int32 dtype, PACKET_IMAGE_ID is renumbered against the surviving CAMERA_TIME axis, with -1 for a kept packet whose image row was dropped, and packets in no image (PACKET_IMAGE_ID == -1) are dropped. Slicing WFOV SCI without PACKET_IMAGE_ID raises ValueError.

  • Add packet_slicing.data_time_axes(ds): a dataset’s sample axes, or its packet time coordinate when it has none. Slicing, the day trim and check_data_times select on these axes.

  • Add libera_utils.l1a.day_coverage.evaluate_day_coverage for the L1A day-assembly completeness check over the buffered day window [D − DAY_BUFFER, D + 1 + DAY_BUFFER]. Each APID’s DayCoveragePolicy is the day_coverage block of its entry in l1a_processing_configs.yml: FRACTION passes a day when its L0 spans reach both window edges (covers_window) and cover at least window_coverage_frac of the window (window_frac); PRESENCE passes on any presence in the buffered window, tested by overlaps_day_window. Span endpoints are timezone-aware; a naive one raises TypeError. Spans separated by up to DEFAULT_SEAM_TOLERANCE (1 s) merge, and a span within it of a window edge reaches the edge, because a file’s span runs from its first to its last sample timestamp and under-reports its occupancy by one sample period.

  • Add libera_utils.l1a.day_coverage.measure_time_axis_coverage, which measures day and buffer occupancy from a granule’s own time axis, in one pass: each sample occupies the spacing to the next, except across a gap, a spacing above DEFAULT_GAP_FACTOR (5) times the median cadence. A single sample occupies no time. Unlike evaluate_day_coverage, it sees gaps inside a file.

  • Add libera_utils.l1a.l1a_packet_configs.configured_apids(), the APIDs with a packet configuration.

  • Add the l1a-processing CLI (libera_utils.aws.l1a_join.request_l1a_processing), which emits an L1AProcessingRequest event asking the L1A Preprocessor to assemble an APID’s day for one applicable date, running the completeness check unless --force is given. It accepts only APIDs with a packet configuration. It builds its session from --profile, or the default profile, and assumes no role. _to_date in manual_processing moves to libera_utils.time.to_date.

  • Add a max_workers argument (default 1) to parse_packets_to_dataset and parse_packets_to_l1a_dataset, which parses files concurrently in fork worker processes, keeping the packet axis in the order given. A worker reads an S3 path with the client the path carries, or with the default S3 client for a URI string, over its own connections. A worker’s exception is re-raised with the worker’s traceback attached as a note. A worker that dies without reporting raises RuntimeError with its exit code, and max_workers below 1, or above 1 on a platform without fork, raises ValueError.

  • Add libera_utils.io.smart_open.is_resource_exhaustion(error), which is true for MemoryError and for an OSError whose errno is ENOSPC, EDQUOT, EMFILE or ENFILE. Only the error itself is examined, not its __cause__.

Changed#

  • BREAKING: PacketConfiguration requires a day_coverage field (DayCoveragePolicy), and every entry in l1a_processing_configs.yml carries one.

  • BREAKING: slice_l1a_dataset_to_time_window no longer takes packet_time_var; a dataset with no sample axis is sliced on the packet time coordinate its APID’s packet configuration names.

  • parse_packets_to_dataset accepts a loaded XtcePacketDefinition as well as a path.

  • parse_packets_to_dataset parses files one at a time onto a preallocated packet axis instead of passing them to create_dataset together, which cost ~800 MiB per two-hour NOM-HK file and ran a full day out of memory. It skips a file with no packets of the APID, and raises ValueError when files parse to different variable sets.

  • manual-processing and step-function-trigger take the UTC calendar date of a timezone-aware date or datetime.

  • Document daily L1A assembly (evaluate_day_coverage → multi-file parse → trim → check_data_times → measure_time_axis_coverage, then a report file for each L1A granule, not yet written).

Removed#

  • BREAKING: Remove packet_slicing.DEFAULT_PACKET_TIME_VAR.

  • BREAKING: Remove the WFOV_HEADER_PARSE_VALID variable from WFOV SCI L1A products, and wfov_image_metadata.WFOV_HEADER_PARSE_VALID_VAR. It was always true: an image whose header cannot be decoded is discarded and counted in HeaderParseErrorCount.

Fixed#

  • manual_ingest_data_products (s3-utils put) splits its files across NewFilesAvailable events of at most MAX_FILES_PER_INGEST_EVENT (5) files, keeping a construction record and the PDS files that share its SCID, first APID, created time and numeric ID in one event; PDS files without their construction record are split at the same limit. A single event holding a day of ground CCSDS captures could not be archived inside one 10 minute Data Ingester invocation.

  • BREAKING: extract_data_time_range and scan_ground_ccsds_file let an error for which is_resource_exhaustion is true, such as a full local disk while an S3 file downloads to the cache, propagate unchanged instead of raising DataTimeUndeterminedError or returning None.

5.20.0 - 2026-10-30#

Added#

  • libera_utils.footprint_matching.weighting: RadialWeigher and AngularPSFWeigher weight each native sample where it lies, and each WeightField carries the footprint’s coverage: the share of the PSF’s energy inside the native cell of at least one finite sample (each sample covers the square of half-width RESOLUTION_KM / sqrt(2), foreshortened along-scan by cos(VZA) in the angular frame, measured on a reference grid in the weigher’s own frame), so a swath gap or a source beyond its time window reads as missing coverage. AngularPSFWeigher raises ValueError when cone_angle_rate is missing or not finite, or the viewing zenith is not finite.

  • libera_utils.footprint_matching.aggregation.aggregate_sample_variables, one scalar per product variable from a footprint’s weighted samples, raising ValueError for a product variable it cannot map to a read variable or whose aggregation is not a registered strategy, and for values or weights that do not match the reader’s variables and samples. check_coverage(coverage) applies the CERES 75 % / 95 % rule to WeightField.coverage.

  • The weighted_positive_std aggregation: the _standard_deviation companion of a weighted_log_mean variable is the spread of the same positive samples the geometric mean averages.

  • Optical depths are aggregated as CERES does: imager-pixel cloud optical depth (CLDPIX cloud_optical_depth) with the PSF-weighted log mean (CERES SSF-85), and every aerosol optical depth (VIIRS Deep Blue aod_550, SSF aerosol_optical_depth, match_aot and the imager land and ocean AODs) and every optical depth its source has already averaged (SSF cloud_optical_depth_lower/_upper, VIIRS L3 cloud_optical_thickness) with the PSF-weighted arithmetic mean (CERES SSF-137).

  • notebooks/05_weighting_aggregation.ipynb, a demonstration of the weighers and the aggregation engine.

  • footprint_matching.geometry.psf_ground_radius_km and slant_range_km: the PSF’s analytic ground radius and the satellite-to-ground distance at a viewing zenith, used by the PSF weighers.

Changed#

  • notebooks/04_sources_query.ipynb also demonstrates coverage.

5.19.0 - 2026-10-30#

Added#

  • libera_utils.footprint_matching.sources.build_sources, which instantiates the readers an OperationalMode’s product uses over their staged granules and returns them keyed by reader registry key. It raises KeyError when the staged mapping is missing a source the mode uses or names one it does not, TypeError when an entry is a single path rather than a list, ValueError when an entry holds no granule, and FileNotFoundError when a staged granule is not an existing file; construction opens no granule, so an unreadable one raises from the first samples_in that loads it. Each reader holds its source once (a static reader its whole indexed body of data, a stepped reader the one hourly step nearest the current footprint, a pass reader the granules in its time window).

  • notebooks/04_sources_query.ipynb, a demonstration of the box query and time matching.

5.18.0 - 2026-10-29#

Added#

  • The swath readers CERES FLASHFlux (flashflux, FLASHFluxReader, reading the RBSP FLASHFlux product in the CERES SSF file layout for FMATCH-IMAGER-FLASH, FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME, with its extended cloud/aerosol/albedo fields carried by FMATCH-IMAGER alone) and CERES CLDPIX, and the VIIRS daily L3 cloud-property reader viirs_cldprop_d3 (VIIRSCldpropD3Reader, CLDPROP_D3_VIIRS_NOAA20, DAILY), completing the nine-reader registry. Every FLASHFlux and CLDPIX field, geolocation and observation time included, is read with its own _FillValue and valid_range, and every VIIRS cloud-property field with its own _FillValue; a field without one raises ValueError. CLDPIX cloud phase and mask codes that the file’s flag_meanings label bad_data are dropped before the mode, and a categorical field without flag_values/flag_meanings raises ValueError. A FLASHFlux granule missing a group or variable the reader reads, or whose variable has other dimensions than the reader expects, raises ValueError naming it and the file’s product_version, as does a CLDPIX granule missing a variable the reader reads. A time variable without a units attribute raises ValueError. The FLASHFlux scene and ADM type codes are int16. The FLASHFlux and CLDPIX readers are PassReaders (TIME_MATCH = NEAREST_PASS): they convert CLDPIX colatitude to latitude and normalize 0..360 longitudes to -180..180, raising ValueError when a CLDPIX Latitude declares a valid_range other than [0, 180] or a FLASHFlux footprint latitude is not in degrees_north, read each footprint’s (Time_and_Position/time) or pixel’s (Observation_Time, fractional hours each dated to the day nearest time_coverage_start) observation time, serve every footprint or pixel as its own time-stamped sample from the pass nearest the footprint time, the samples within PASS_WINDOW of the observation time of the sample nearest the footprint time, so a footprint midway between two passes gets only the nearer (none beyond MAX_TIME_OFFSET), choosing that sample from those inside the footprint’s own box and from the padding only when none falls inside, hold only the hourly granules within that offset of the current footprint, and measure the granule’s own sample spacing (PointSamples.spacing_km, the larger of the along-scan and along-track spacings) for the coverage measure in place of the nominal RESOLUTION_KM. PassReader joins StaticReader and SteppedGridReader as a reader base, and ReaderRegistry.get_readers_for_mode returns the readers that contribute to an OperationalMode’s product (FMATCH_MODE_READERS, derived from the built-in readers’ VariableSpec.modes).

  • notebooks/03_readers_swath.ipynb, a demonstration of the swath readers.

  • Fill and valid-range helpers in libera_utils.io.netcdf, used by the CERES FLASHFlux, CLDPIX and VIIRS cloud-property readers: fill_in_dtype returns a _FillValue in a given dtype, or None when no value of that dtype can equal it; field_fill_value and field_valid_range return a netCDF4 variable’s own _FillValue and valid_range, raising ValueError when it has none; apply_fill_and_valid_range returns a raw, optionally masked array as float64 with masked entries, fill values (compared in the raw dtype) and values outside an inclusive valid_range (either order) set to NaN.

Changed#

  • BREAKING: An AncillaryReader is constructed from a sequence of staged granule paths: a bare str or Path raises TypeError, an empty sequence ValueError, and a path that is not an existing file FileNotFoundError naming the reader and every such path. A single path was previously accepted, and a missing file failed only when it was read.

  • BREAKING: VariableSpec takes a required modes, a frozenset[OperationalMode] naming every product that carries the variable, in place of required_mode. An empty modes raises ValueError, and one that is not a frozenset of OperationalMode members raises TypeError. A standard-deviation companion from with_standard_deviation_companions carries its parent’s modes.

Removed#

  • BREAKING: OperationalMode.rank, VariableSpec.required_mode, VariableSpec.only_modes and spec_active_in_mode. A variable belongs to a product exactly when mode in spec.modes.

5.17.0 - 2026-10-29#

Added#

  • write_libera_data_product takes worker_compressed_variables, a tuple of Dask-backed data variables whose chunks are shuffled and deflated on the dask.distributed workers instead of one at a time under the HDF5 write lock. The writing process stores the compressed chunks with h5py write_direct_chunk, and the file reads back identical to a normal write. It requires the h5netcdf engine and an active distributed.Client, and each named variable’s Dask chunks must be whole multiples of its on-disk chunk shape, its filters deflate with or without shuffle, and its dtype integer or floating point; otherwise ValueError before any data is computed. A write that fails after creating the file deletes it. The default, an empty tuple, writes with to_netcdf as before.

  • KernelManager.populate_cache() fills the on-disk NAIF and static kernel caches and leaves the SPICE kernel pool empty, for a process to call before starting workers that each load their own kernels.

  • dask[distributed] is a declared dependency; worker-compressed writes import it and h5py.

Changed#

  • kernel_maker.create_kernels_from_manifest reads the input L1A product once and creates each requested SPICE kernel at the same time, each in its own forked process, so the Az/El and JPSS kernel steps take about as long as their slowest kernel rather than the sum. It requires Linux and raises ValueError on any other platform before reading the L1A. Kernel filenames and the output manifest are unchanged. The ValueError raised after every kernel was attempted when any kernel fails now also gives each failed kernel’s reason: its process’s traceback, or its exit code when the process sent nothing; a kernel process killed by the operating system counts as that kernel failing. A kernel process flushes CloudWatch log handlers before it exits. Peak memory is about the sum of the concurrent kernels’ (an estimated 18 GB for a full-day Az/El run, from about 9 GB measured per kernel). A kernel identifier repeated in kernel_product_ids raises ValueError.

  • BREAKING: kernel_maker.create_kernels_from_manifest raises a shared-setup failure before any kernel is attempted, with its own exception type, instead of the aggregated ValueError listing every kernel as failed: an unreadable L1A file raises ValueError, a kernel work directory path too long for curryer raises RuntimeError (set LIBERA_TEMP_DIR to a shorter base path), and a failure populating the kernel cache raises RuntimeError, FileNotFoundError or a requests.RequestException.

  • write_libera_data_product writes an S3 product to a temporary local file, uploads it and removes the file, instead of keeping it in the cloudpathlib cache for a read that the manifest checksum needed. A full-day product no longer stays on local disk after upload, and the write no longer reads cloudpathlib’s private cache path or sends a request to sync the cache file’s modification time.

5.16.0 - 2026-10-29#

Changed#

  • BREAKING: Manifest files no longer record checksums, input or output. Manifest.files is a list of absolute path strings, written as "files": ["s3://...", ...]. A manifest in the previous {"filename": ..., "checksum": ...} record format raises ValidationError when read. A reader on libera_utils 5.15.0 or earlier treats each listed path as a file to checksum, so it downloads every listed file; move readers and writers of a manifest to this version together.

  • Manifest.add_files() checks that each file exists and is a file, raising IsADirectoryError for a local directory, an S3 bucket root or an S3 prefix, but no longer reads it, so an S3 file is not downloaded and a file larger than local disk can be listed.

  • BREAKING: The Manifest constructor no longer checks that a listed path exists; only add_files() does.

  • BREAKING: A path listed twice in a manifest is an error, where it was skipped with a warning: Manifest.add_files() raises ValueError for a path already listed or given twice, adding nothing, and the constructor and Manifest.from_file() raise ValidationError.

  • create_kernels_from_manifest no longer validates input checksums.

Removed#

  • BREAKING: ManifestFileRecord, Manifest.validate_checksums(), Manifest.check_file_structure() and libera_utils.io.manifest.calculate_checksum.

Fixed#

  • A file whose content matched another listed file’s was dropped, with a warning, by Manifest.add_files(), the constructor and Manifest.from_file(). Every file is now listed under its own name.

5.15.0 - 2026-10-28#

Added#

  • The libera_utils.footprint_matching.readers plugin subsystem: an AncillaryReader base with auto-registration (StaticReader for a daily composite, map or time-stamped swath held in memory; SteppedGridReader for a time series of global grids served one step at a time), a ReaderRegistry, and HDF4 and HDF-EOS5 I/O helpers. A reader serves the native samples of its source inside a footprint’s bounding box as a SampleSet (every pixel, footprint or grid point at its own position, no averaging, binning or interpolation), matched to the footprint time by its TimeMatch policy: DAILY (IGBP, NISE, VIIRS BRDF and AOD) or NEAREST_STEP (ERA5). A reader whose READER_KEY is already registered to another class is rejected when its module is imported. A query raises ValueError when a point cloud holds a coordinate outside latitude [-90, 90] or longitude [-180, 180], when a NEAREST_PASS reader serves samples without times, or when the footprint time a time-selecting reader needs is NaT. This release adds the gridded readers:

    • ERA5 single-level (era5_surface, 7 variables) and 37-level pressure (era5_pressure) fields, serving the hourly step nearest each footprint’s time (none beyond 1 h), rejecting a file with no time axis, and declaring RESOLUTION_KM = 28.0, the 0.25 degree grid spacing.

    • VIIRS Deep Blue AOD (AERDB_D3_VIIRS_NOAA20) with a categorical aerosol_type, applying each field’s own _FillValue and valid_range (AOD 0 to 10).

    • VIIRS BRDF (VJ143C1), served at the 0.05 degree cell centers (the file’s lat/lon are cell edges, checked against the grid’s HDF-EOS5 StructMetadata.0 corners), applying each field’s own _FillValue, valid_range, scale_factor and add_offset.

    • IGBP land cover (MCD12Q1), held as its uint8 sinusoidal tiles and geolocated per query, declaring RESOLUTION_KM = 0.463, the 463.3 m sinusoidal cell.

    • NSIDC NISE sea ice (AMSR-2 provenance), with the unused Extent codes (102, 105-251) and the coastline and suspect-ice quality codes (252, 253) missing (NaN) in every layer. Each hemisphere’s HDF-EOS StructMetadata.0 grid is checked against the 721 x 721 25 km EASE-Grid it is geolocated on (dimensions, corners and GCTP_LAMAZ projection), and a granule that differs raises ValueError.

  • notebooks/02_readers_gridded.ipynb, a demonstration of the gridded readers.

  • pyhdf (IGBP and NISE HDF4 reads) and h5py (VIIRS BRDF HDF5 reads) are runtime dependencies. pyhdf needs the HDF4 C library (libhdf4-dev) at build time when no prebuilt wheel is available, as on aarch64; the Docker image installs it.

  • DataLevel.ANC for ancillary inputs produced outside the SDC and ingested from their upstream providers. It archives to libera-ancillary-data (LiberaDataBucketName.ANCILLARY_ARCHIVE_BUCKET); AUX keeps the non-leveled products the SDC produces.

  • Twelve DataProductIdentifier members at DataLevel.ANC, one per external input product, with ProductIDs of the form <SOURCE>-<PRODUCT>, PRODUCT led by the observing platform where the upstream product is made per platform: anc_ceres_noaa20_cldpix (CERES-NOAA20-CLDPIX), anc_ceres_noaa20_flashflux (CERES-NOAA20-FLASHFLUX), anc_ceres_noaa20_ssf (CERES-NOAA20-SSF), anc_nsidc_land_ocean_mask (NSIDC-LAND-OCEAN-MASK), anc_nsidc_ice_snow (NSIDC-ICE-SNOW), anc_modis_igbp (MODIS-IGBP), anc_era5_pressure_level (ERA5-PRESSURE-LEVEL), anc_era5_single_level (ERA5-SINGLE-LEVEL), anc_viirs_noaa20_aod (VIIRS-NOAA20-AOD), anc_viirs_noaa20_brdf (VIIRS-NOAA20-BRDF), anc_viirs_noaa20_cloud_properties (VIIRS-NOAA20-CLOUD-PROPERTIES) and anc_viirs_noaa20_m4_sdr (VIIRS-NOAA20-M4-SDR). No processing step produces them. The DataProductIdentifier docstring states the ancillary ProductID pattern.

Changed#

  • The h5netcdf dependency is declared with its h5py extra (h5netcdf[h5py]), since h5netcdf no longer installs h5py by default.

  • The dev dependency group depends on pyjpegls instead of pillow-jpls for JPEG-LS decoding in the WFOV integration test. pillow-jpls 1.3.2, its latest release, publishes wheels only through cp312, so Python 3.13 and 3.14 installs build it from source, which fails on gcc 14 hosts; pyjpegls binds the same CharLS codec.

5.14.0 - 2026-10-27#

Added#

  • create_product_dataset (array input) and write_libera_data_product (Dataset input) set the ACDD date_created attribute to the creation time, ISO 8601 UTC at microsecond precision. A value already present is overwritten with a warning.

  • libera_utils.io.product_definition.DATE_CREATED_ATTRIBUTE and ALGORITHM_VERSION_ATTRIBUTE, the names of those product attributes.

Changed#

  • BREAKING: The revision in Libera data product filenames is the UTC creation time at microsecond precision, R%Y%m%dT%H%M%S%f, e.g. LIBERA_L1B_RAD-4CH_V1-2-3_20270102T112233_20270102T122233_R20270103T080910123456.nc, so files produced within the same second no longer clobber each other in the dropbox bucket. Names with the earlier second-resolution R%y%j%H%M%S revision no longer parse: LiberaDataProductFilename raises ValueError for them. The UMM-G GranuleUR, which is the filename stem, and the .cmr.json metadata name follow the new form.

  • BREAKING: libera_utils.io.filenaming.REVISION_TS_FORMAT is now R%Y%m%dT%H%M%S%f.

  • BREAKING: date_created is a required standard product attribute, and generate_data_product_filename takes the filename revision from it. A strict conformance check on a Dataset built neither by create_product_dataset nor by write_libera_data_product reports it missing.

  • BREAKING: A product definition YAML that gives date_created a value raises ValueError.

  • BREAKING: UMM-G generation (UMMGranule.from_dataset, write_data_product_metadata_ummg) raises ValueError when a Dataset has no date_created, and emits it as the DataGranule ProductionDateTime and the Create provider date. It previously read a ProductionDateTime attribute and, without one, warned and used the current time.

  • UMM-G TemporalExtent no longer falls back to a ProductionDateTime attribute; without the Range* attributes it is omitted.

  • from_filename_parts takes revision: datetime | None; the default None uses the current UTC time at the call.

  • L1A packet parsing and the NOM-HK trimmer no longer set date_created; the trimmer drops the parent product’s value so the writer sets a fresh one.

  • kernel_maker no longer builds a revision for SPICE kernel filenames; the default applies.

Removed#

  • BREAKING: .h5 is no longer an accepted data product filename extension.

Fixed#

  • The revision default in from_filename_parts was datetime.now(tz=UTC) evaluated once at import, so every product written by one process shared a single revision. Revisions are now generated per call.

  • LiberaDataProductFilename raises ValueError at construction, or when its path is assigned, when a date or time in the name does not exist, such as month 13. The error previously surfaced at the first filename_parts access.

  • A stray debug print in UMMGDatasetTransformer.extract_temporal_extent is removed.

5.13.0 - 2026-10-26#

Added#

  • The API docs now include libera_utils.l1a. The package had no __init__.py, so Sphinx skipped it.

  • Link the UMM-G granule page from the user docs index.

  • Manifest.write() takes a keyword-only overwrite=False, and out_path may be a directory, an S3 prefix, or a full manifest file path.

  • Manifest.output_manifest_from_input_manifest() takes an optional configuration= dict, and raises ManifestError for an input manifest with no ULID.

Changed#

  • The docs build fails on any warning: doc/Makefile runs sphinx-build -W --keep-going, and .readthedocs.yaml sets fail_on_warning.

  • ReadTheDocs builds the docs on Python 3.13 (Ubuntu 24.04), and the docgen dependency group requires Sphinx 9.1 or newer on Python 3.12 or newer.

  • BREAKING: Manifest.ulid_code is a read-only property derived from filename. Assigning to it raises; assign filename instead. ulid_code= is still accepted by the constructor.

  • BREAKING: Manifest.write() sets filename to the path it wrote.

  • BREAKING: Manifest.write() no longer takes a filename argument; pass the full manifest file path as out_path instead. The old positional form write(out_path, filename) raises TypeError.

  • BREAKING: Manifest.write() raises FileExistsError for an existing target unless overwrite=True is passed, and ManifestError for a target whose INPUT/OUTPUT label disagrees with manifest_type or that would change the manifest’s ULID, and ManifestError for an out_path that has an extension but is not a valid manifest filename.

  • BREAKING: Manifest.validate_checksums() raises ManifestError instead of ValueError. A file that cannot be found fails validation.

  • BREAKING: Manifest.from_file() raises ManifestError for a path that is not a valid manifest filename, or when the filename or ulid_code stored in the file disagrees with the ULID in the path.

  • BREAKING: Manifest.filename is validated on assignment, and a filename or manifest_type whose INPUT/OUTPUT label disagrees with the other raises ValidationError.

  • BREAKING: A file record with a null checksum, such as one read from a manifest file, raises ValidationError instead of being recomputed.

  • BREAKING: MANIFEST_FILE_REGEX must match the whole basename, so names such as LIBERA_INPUT_MANIFEST_<ULID>.json.bak are no longer valid manifest filenames.

Removed#

  • numpydoc from the docgen dependency group. sphinx.ext.napoleon parses the numpy style docstrings.

  • BREAKING: libera_utils.io.manifest.get_ulid_code. Use Manifest.ulid_code, or ManifestFilename(path).filename_parts.ulid_code.

Fixed#

  • Fix the docstring, markdown and Sphinx config defects behind the docs build’s 9841 warnings, including cross-references to private or renamed targets (create_l1a_dataset is now parse_packets_to_l1a_dataset).

  • Fix the docs build warnings and errors in the footprint_matching and scene_identification docstrings and the write_libera_data_product and LiberaVariableDefinition Raises sections, which the build now treats as failures.

  • A second Manifest.write() to a different directory no longer writes to the first location.

  • A manifest with no filename serializes filename as null, not the string "None".

5.12.0 - 2026-10-21#

Added#

  • The libera_utils.footprint_matching package, the foundation of the FMATCH (footprint matching) algorithm:

    • footprint_matching.types holds the shared data classes and enumerations: OperationalMode (the five FMATCH modes, ordered by latency), FmatchCoverageFlag, BoundingBox, RadiometerFootprint and VariableSpec, with spec_active_in_mode and with_standard_deviation_companions. SampleSet, GridSamples, PointSamples and TimeMatch describe a reader’s native samples inside a footprint’s box (every pixel, footprint or grid point at its own position, no averaging, binning or interpolation), the two native forms a reader loads, and the policy by which a reader selects samples in time (DAILY, NEAREST_STEP, NEAREST_PASS). A RadiometerFootprint carries its observation time, which the readers match ancillary samples to, and its bbox is None exactly when it is an off-limb view.

    • footprint_matching.psf evaluates the point spread function (psf_weight) and its 95%-energy angular extent (psf_95_energy_extent), plus static_fov_extent and conservative_along_scan_extent. The Libera PSF has not been delivered, so the module uses the heritage CERES analytic PSF (CERES ATBD 4.4) as a stand-in; its CERES constants are isolated in one block so that swapping in the Libera PSF changes only that file.

    • footprint_matching.geometry turns L1B footprint geolocation and viewing geometry into the latitude/longitude bounding box of the ground the radiometer senses (compute_footprint_bounding_box, and compute_footprint_bounding_boxes for a batch). It ray-traces the PSF extent onto the WGS84 ellipsoid using curryer’s spatial tools, rounds the box outward so it always encloses the footprint, and handles dateline crossings and footprints that enclose a pole. A boresight that misses the Earth raises OffLimbError, a footprint whose box corners run off the limb raises PartialFootprintError, and unusable geolocation or ephemeris raises MissingGeolocationError. The module also provides compute_viewing_frames / ViewingFrame, surface_cell_ecef_km and project_to_angular, which project ground grid cells into the radiometer’s angular frame for PSF weighting.

  • notebooks/01_geometry_psf.ipynb, a demonstration of the PSF model and footprint geometry.

  • pyproj is a runtime dependency, used for WGS84 geodesic distances.

5.11.5 - 2026-10-14#

Added#

  • SCENE-ID-CAM (radiometer timescale, written on RADIOMETER_TIME) and SCENE-ID-CAM-CAMTIME (camera timescale) algorithm runners, each with a product definition and a FootprintData.from_fmatch_cam / from_fmatch_cam_camtime reader. Both run from one libera-utils scene-id <manifest> command and one libera-utils-scene-id Docker target; the LIBERA_SCENE_ID_VARIANT environment variable (cam or cam-camtime) selects the variant, and the runner raises ValueError when it is unset or unknown. The variants share their manifest/dropbox plumbing through scene_id_algorithm.run_algorithm, parameterized by a per-variant SceneIdRunnerConfig.

  • SCENE-ID-CAM-CAMTIME stores one image subsection per record on the 2-D (CAMERA_TIME, PSEUDOFOOTPRINT) grid and passes the FMATCH footprint identifiers (camera time, pixel-index ranges, PSF bounding box, boresight geolocation) straight through. CAMERA_TIME is a 1-D dimension coordinate (one entry per image) and PSEUDOFOOTPRINT a 0-based image-subsection index (an image is segmented into many, possibly overlapping, subsections that share its time).

  • The unfiltering scene definition, and the standard_scene_definitions / default_scene_definitions helpers.

  • Viewing geometry in scene identification: solar_zenith_angle, viewing_zenith_angle and relative_azimuth_angle are classification variables with scene_bin_{type}_{angle}_min/max bounds. The bins currently span the full physical range as a placeholder, so scene IDs are unchanged.

  • FootprintData.to_time_product() and its to_radiometer_time_product() wrapper, which promote the observation-time variable to a coordinate and finalize the dataset on its per-footprint axis for writing.

Changed#

  • BREAKING: Scene identification moved to the libera_utils.scene_identification package; import scene_id and scene_definitions from there.

  • Default scene classifications are now TRMM, ERBE and unfiltering, and identify_scenes() with no arguments uses them.

5.11.4 - 2026-10-14#

Changed#

  • write_libera_data_product always hands the NetCDF engine a filesystem path, never an open file object. An S3 destination is written to the path’s cloudpathlib cache file and uploaded from there, replacing any existing object at the same key. The cache keeps the product afterwards, as it did when the writer went through CloudPath.open, so reading it back in the same process (the output manifest’s checksum, for one) does not download it again. Both h5netcdf and netcdf4 now write to S3; netcdf4 previously wrote into the cloudpathlib cache and uploaded nothing.

  • encoding.chunksizes in a product definition is stored as a tuple, so a YAML list is accepted by the h5netcdf engine as well as netcdf4. A chunksizes that is not a list of positive integers with one entry per dimension now raises ValidationError when the definition loads rather than when the product is written.

Fixed#

  • calculate_checksum, which Manifest.add_files calls for every output file, reads in 16 MiB blocks instead of reading the whole file into memory, so a product larger than the host’s memory can be added to a manifest.

  • Writing a product under the distributed Dask scheduler with the h5netcdf engine no longer fails with TypeError: cannot pickle '_io.BufferedRandom' object.

5.11.3 - 2026-10-09#

Added#

  • All Filename classes in libera_utils.io.filenaming are hashable. hash() is the hash of the path as given, so filenames can be dictionary keys and set members. Reassigning path changes the hash, so do not mutate a filename that is already in a set or a dict.

Changed#

  • BREAKING: AbstractValidFilename.__eq__ compares paths only and returns NotImplemented for non-filename operands, so filename == "some string" is False instead of raising AttributeError. Two filenames with the same basename in different directories or buckets are not equal. Previously the parsed filename parts were compared as well, which was redundant because they are derived from the path.

Fixed#

  • Comparing two Manifest models when one has filename=None no longer raises AttributeError from the filename comparison.

5.11.2 - 2026-10-09#

Changed#

  • Fix the intermittent PytestUnraisableExceptionWarning CI failures. Leaked handles (see Fixed), test fixtures that detached root logging handlers without closing them, and an h5py.File handed a file object it does not own produced ResourceWarnings during garbage collection. pytest drains those from one global queue at each test’s setup, call and teardown, so the failure landed on whichever unrelated test was running at the time. Tests that assert no warnings now use the shared strict_warnings mark, which keeps warnings-as-errors but declines to own unraisable exceptions leaked elsewhere in the suite.

  • calculate_checksum no longer decompresses *.gz input before hashing. A manifest checksum has to cover the bytes as stored so it can be compared against the file a data provider delivered; the previous value hashed content that exists nowhere on disk and would change if the file were recompressed at a different level. Any checksum an earlier version recorded for a *.gz file must be regenerated.

Fixed#

  • smart_open now closes the underlying file object when the GzipFile it returns is closed. GzipFile only closes a file it opened itself, so reading any *.gz through smart_open leaked an open file handle until the garbage collector finalized it. This affected library callers (calculate_checksum, metadata_writer, SPICE kernel downloads), not just tests.

  • smart_open passes the requested mode to GzipFile when opening a *.gz file. Writing one no longer emits the FutureWarning GzipFile raises when it infers a write mode from the file object, and 'wb+' now opens for writing instead of read-only.

  • smart_open rejects a text mode for a *.gz path before opening the file, so the OSError no longer leaves an open file handle behind.

  • Writing a *.gz file through smart_open with an S3Path or AnyPath that already had a local cached copy (a second write, or a write after a read through the same object) no longer leaves cloudpathlib’s cache marked dirty, which made the next read raise OverwriteDirtyFileError.

5.11.1 - 2026-09-25#

Added#

  • Add docker-compose entries for spice kernel making.

5.11.0 - 2026-09-09#

Added#

  • Add libera_utils.l1a.data_time_extractors for per-file science data-time spans without full L1A assembly, covering every DATA_TIME_INDEXED_APIDS member (WFOV SOP FSW image times; RAD/CAL/AXIS sample epoch+period or per-sample times). extract_data_time_range returns None for a WFOV file holding no SOP packet, which is expected when a large image’s mem-dump is chunked across files or downlink passes.

  • jpss_sc_pos (APID 11) is data-time indexed. Its span runs from the earliest to the latest time reported by either of the packet’s independently timestamped sample groups (ADGPS ephemeris, ADCFA attitude), so it records the extent of data present rather than the range both groups cover. A consumer needing ephemeris and attitude together must intersect the two sample-time ranges itself.

  • Add demuxed ground-test CCSDS support: LiberaGroundCcsdsFilename and DataProductIdentifier.l0_ground_ccsds for the LIBERA_SDC_<apid>_ccsds_<yyyy>_<doy>_<hh>_<mm>_<ss> naming (L0 archive prefix GroundCCSDS/<apid>/<yyyy>/<mm>/<dd>/). Each file holds one APID, read from the basename, with no record header before the CCSDS primary header. The name’s time fields are the bin start the file was cut from, not the span of the packets inside it.

  • Add libera_utils.l1a.ground_ccsds.scan_ground_ccsds_file, returning one file’s packet and science data time spans for File Metadata ingest. It returns None, logging the reason, when no packet-time span is possible at all, and returns the packet span with degraded_reason set when only the science data times are unavailable.

  • Manual ingest (s3-utils put / manual_ingest_data_products) accepts demuxed ground CCSDS files. verify_ingestion checks archival and the base File Metadata row for these; the searchable row is not verified.

  • extract_data_time_range and parse_packets_to_l1a_dataset accept an optional skip_header_bytes= override, falling back to SKIP_PACKET_HEADER_BYTES when omitted, so one process can handle both flight and ground files without mutating shared config.

Changed#

  • Extracted time spans drop NaT and timestamps outside MIN_VALID_TELEMETRY_TIME-MAX_VALID_TELEMETRY_TIME (defaults 2020-01-01 and 2045-01-01, both config-overridable). A ground capture can carry a packet whose onboard clock has not yet been time-synced, decoding to just after CCSDS_EPOCH (1958-01-01), which would otherwise stretch a span across ~68 years.

  • libera_utils.l1a.packets.expand_sample_times is now public (was _expand_sample_times), and parse_packets_to_l1a_dataset takes ground_data and verbose as keyword-only arguments.

Fixed#

  • verify_ingestion treats its File Metadata record count as a floor rather than an exact match. A file’s data can span more than one day, giving it a searchable record per applicable date, so an L0 PDS with a multi-day span (as a jpss_sc_pos file can have) no longer polls to timeout on a count of 3 against an expected 2.

5.10.11 - 2026-09-08#

Changed#

  • The production Docker image no longer sets LEAPSECOND_FILE_ENV. It pointed at a directory containing no naif*.tls, so curryer’s lookup found nothing there; KernelManager.load_naif_kernels sets the variable at runtime to the directory of the LSK it furnishes.

5.10.10 - 2026-09-08#

Added#

  • KernelManager.ensure_kernel_coverage verifies that the furnished kernels cover every requested target across a time window, raising before a gap surfaces as an obscure SPICE failure deep in a later computation or as silently wrong numbers. Opt-in: call it after furnishing and before the first computation that reads the kernels.

Changed#

  • Require lasp-curryer >= 0.5.2, for curryer.kernels.coverage and the packaged leapsecond kernel that makes LEAPSECOND_FILE_ENV an override rather than a requirement.

5.10.9 - 2026-09-08#

Fixed#

  • WFOV_FSW_HEADER_AZIMUTH_ANGLE in the ICIE WFOV SCI L1A product is documented as degrees, not radians as it was incorrectly listed before.

  • Whitespace clean up in trim family csv

  • Test L1A product definition units for Axis-Sample updated to match the actual products

5.10.8 - 2026-09-03#

Changed#

  • BREAKING: ecr-upload now requires a local image tag instead of defaulting to latest. Algorithm images are built under an explicit docker build -t <name>:<version>, so a local latest usually does not exist and the old default failed later with an opaque Local image not found: <name>:latest — after the L2 Team Role assumption round-trip. Omitting the tag now fails immediately with an explanatory error. Pass my-image:latest explicitly if that really is the image you want. The --ecr-tags default of latest for the remote ECR tag is unchanged.

  • The ecr-upload --image-tag option is deprecated in favor of image-name:image-tag. It still works and logs a warning pointing at the new syntax; supplying both is an error unless they agree. push_image_to_ecr accepts image_tag=None and takes the tag from image_name instead, which must then carry one.

Fixed#

  • libera-utils ecr-upload now accepts the local image in the usual Docker form, image-name:image-tag (e.g. ecr-upload l2-comp-flux my-image:1.2.3). Previously this positional argument had to be a bare image name and the tag had to be passed separately, so a name:tag value silently produced a Local image not found: name:tag:latest error. Colons in a registry host:port (localhost:5000/my-image) are not treated as tags.

5.10.7 - 2026-09-03#

Changed#

  • BREAKING: The four per-channel radiometer frames (LIBERA_SW_RAD_COORD, LIBERA_SSW_RAD_COORD, LIBERA_LW_RAD_COORD, LIBERA_TOT_RAD_COORD) are replaced by a single co-aligned LIBERA_RAD_COORD frame and one LIBERA_RAD instrument/FOV in the frame and instrument kernels for both JPSS-4 and NOAA-20. The four channels share a boresight, so the per-channel definitions carried identical geometry. Downstream callers must transform to LIBERA_RAD_COORD instead of a per-channel frame.

  • BREAKING: The four per-channel fixed-offset SPK configs are replaced by one libera_rad.fixed_offset.spk.json per mission, so LIBERA_KERNEL_STATIC_SPK_CONFIGS in data/config.json drops from 8 static SPK configs to 5.

5.10.6 - 2026-09-02#

Added#

  • Add four radiometer cal-combine ProcessingStepIdentifier family steps (cal-gain-family, cal-swc-family, cal-lwc-family, cal-solar-family), each declaring every CAL product its family’s ObsIDs can yield. Lunar and camera family steps are deferred (TODO[LIBSDC-811]).

  • ProcessingStepIdentifier.ecr_name supports an optional shared ECR name so all radiometer cal steps resolve to cal-rad-docker-repo. Each job dispatches on the ObsID carried by the TRIMMED input it is handed, so no per-job step identity has to be configured separately.

  • Add libera_utils.obsids.TRIM_FAMILIES and get_family_specs() so a cal step enumerates the ObsIDs — and CAL products — of its trimmed input family without hand-maintaining the mapping.

  • Add data/trim_family_inputs.csv (FAMILY_INPUTS / get_family_inputs()) recording the L1A products each family’s cal step consumes besides its own TRIMMED product; the two together are the input-products set a libera_cdk cal-*-family node should declare. The two catalogs are cross-checked at import.

  • Add libera_utils.l1a.nom_hk_trim to detect contiguous ObsID runs in decoded NOM-HK and write one trimmed file per run, stamped with that run’s family ProductID. Several files per day normally share a family ProductID and are told apart by their filename time ranges.

  • Add libera_utils.l1a.packet_slicing as the supported way to subset a decoded L1A product. select_packets and slice_l1a_dataset_to_time_window treat PACKET as the driver — whole packets survive or none of them does — and renumber {sample_group}_packet_index against the surviving packet axis, so sample-to-packet traceability holds across a subset. Slicing either axis directly does not.

Changed#

  • BREAKING: L1A trimmed NOM-HK products are now one per calibration dependency family rather than one per ObsID: 10 NOM-HK-<FAMILY>-FAMILY-TRIMMED members replace the 40 NOM-HK-<EVENT>-TRIMMED members added in 5.10.3. Each ObsID keeps its own CAL product, and the ObsID stays readable from the ICIE__SW_OBSID_* variable inside each trimmed file. A family never spans both ObsID fields, so RAD and WFOV VIIRS lunar (513/514) are registered as separate families with distinct ProductIDs.

  • BREAKING: Registry validation now rejects a CAL product claimed by more than one ObsID, and a TRIMMED family registered on both the RAD and WFOV ObsID fields. The 5.10.3 check that a TRIMMED product is claimed by only one ObsID is removed, since sharing one is what defines a family.

  • Add the ObsID Registry user-docs page, covering the (source, obsid) keying, how downstream repos dispatch cal-combine steps from the registry, and the decision that NOM-HK is the only product the preprocessor trims — cal containers subset the full daily L1A inputs themselves.

Fixed#

  • parse_packets_to_l1a_dataset now sorts the packet axis into its final order before samples are expanded, instead of sorting at the end. The trailing sort permuted PACKET without permuting the sample axes, silently invalidating the {sample_group}_packet_index variables it had already written. Output ordering is unchanged; the indices are now correct.

5.10.5 - 2026-08-31#

Added#

  • WFOV SCI (APID 1040) L1A processing stitches complete SOP→EOP images onto CAMERA_TIME, stores compressed JPEG-LS payloads in WFOV_COMPRESSED_IMAGE (uint8/BLOB_BYTE + WFOV_COMPRESSED_IMAGE_LENGTH), and traces packets back to their image with PACKET_IMAGE_ID.

  • Decode the 176-byte WFOV header (FSW + FPGA) onto CAMERA_TIME as WFOV_FSW_HEADER_*, WFOV_IMAGE_HEADER_*, WFOV_IMAGE_FOOTER_*, and WFOV_FPGA_STATUS_*; trailing 8-byte NAND footer is validated against a known magic pattern (FooterMismatchCount).

  • Add file-level quality attrs PacketCountNotUsedInImages, ErrorFlaggedImageCount, FooterMismatchCount, HeaderParseErrorCount, plus FirstImageIncomplete / LastImageIncomplete flagging expected truncation at a packet window’s leading/trailing edge (PacketCountNotUsedInImages counts only genuine mid-stream anomalies).

  • WFOV SCI processing now raises ValueError when no complete image can be stitched. The raise happens before ICIE__WFOV_DATA is dropped, so the raw packets survive for diagnosis, and the message carries all stitch counters.

Changed#

  • BREAKING: CAMERA_TIME now contains one row per complete stitched image only (not every qualifying SOP). Removed redundant WFOV_IMAGE_COMPLETE.

  • BREAKING: ICIE__WFOV_DATA is dropped from the WFOV L1A product entirely — every packet’s raw payload is by then either duplicated in WFOV_COMPRESSED_IMAGE or unusable from this granule alone. PACKET_IMAGE_ID is the only remaining per-packet trace-back.

Fixed#

  • A structurally complete image whose 176-byte header cannot be decoded is now discarded and counted in HeaderParseErrorCount, rather than emitted with a NaT CAMERA_TIME that crashed filename generation. CAMERA_TIME is therefore always free of NaT.

  • SINGLE-flagged WFOV packets (whole image in one packet) are unsupported and were silently misattributed to window-edge truncation. They now log a warning and count toward PacketCountNotUsedInImages; a SINGLE mid-collection abandons the image in progress.

5.10.4 - 2026-08-25#

Added#

  • Register ADM-specific standard dimensions in libera_dimensions.yml (viewing-geometry, surface-type, and cloud/wind property bins for ERBE-like and TRMM-like ADM products)

5.10.3 - 2026-08-24#

Added#

  • Replace the eight alpha calibration DataProductIdentifier members (cal_solar_face{1,2,3}_combined, cal_lw_temp{1,2,3}_combined, cal_gain_combined, cal_sw_combined / SOLAR-FACE1-COMBINED, LW-TEMP1-COMBINED, GAIN-COMBINED, SW-COMBINED, …) with one CAL product per calibration ObsID (cal_gain/GAIN, cal_swc_365nm/SWC-365NM, cal_lwc_310k/LWC-310K, cal_solar_ssw_pri/SOLAR-SSW-PRI, the lunar/VIIRS-lunar and camera cal products, etc.). Downstream consumers must move to the new members; no aliases are kept.

  • Add ObsID-trimmed NOM-HK DataProductIdentifiers (NOM-HK-<EVENT>-TRIMMED), one per calibration ObsID, produced by L1A Step 1 preprocessing and consumed by the per-ObsID cal-combine steps.

  • Add libera_utils.obsids and the libera_utils/data/obsid_registry.csv catalog: the ICIE software ObsID registry mapping (NomHkObsidSource, obsid) to its TRIMMED and CAL ProductIDs, with get_obsid_spec and iter_trim_eligible for downstream dispatch. The CSV is validated at import (unknown member names, wrong data level, malformed rows, kind/source mismatches, duplicate ObsIDs, and TRIMMED products claimed by more than one ObsID all raise ValueError).

5.10.2 - 2026-07-31#

Fixed#

  • enforce_data_array_conformance no longer emits a UserWarning (or log line) for every encoding key present on a DataArray but absent from the product definition. Extra keys (typically xarray/NetCDF runtime metadata such as source, original_shape, preferred_chunks, compression stubs) are dropped silently; conflicting values for keys that are defined still warn. This removes thousands of spurious warnings when packaging products built from opened L1A NetCDF inputs.

5.10.1 - 2026-07-31#

Added#

  • ecr-upload now registers the algorithm version(s) it uploads. After pushing, it emits a NewAlgorithmImage event (one per non-latest ECR tag) to the SDC event bus so the Registrar creates the corresponding versioned AWS Batch job definition. Adds --verify (blocks until both the ECR image and the Batch job definition are confirmed present, up to --timeout seconds).

  • Add the register-algorithm-image CLI for emitting a NewAlgorithmImage event for an image already in ECR (--image-digest/--verify/--timeout). The new aws/algorithm_registration.py module provides put_new_algorithm_image_event and verify_algorithm_registration (checks the ECR image exists and an ACTIVE Batch job definition references it).

  • Quieter ecr-upload push logging — per-layer progress ticks are suppressed, a per-tag summary and the resulting image digest are logged at INFO, and -v/--verbose enables full DEBUG push detail. push_image_to_ecr now returns a {tag: digest} mapping.

Changed#

  • AWS region for the ECR/EventBridge/Batch operations is now taken from the session’s AWS configuration (profile / AWS_REGION) instead of a hard-coded us-west-2; a clear error is raised if no region is configured, and --verify warns when the image URI’s region differs from the session region.

  • Document register-algorithm-image and the new ecr-upload flags in the usage docs.

5.10.0 - 2026-07-24#

Added#

  • Kernel generation applies the measured Libera opto-mechanical frame misalignments from OAV3 ground testing. The Az/El mechanism CKs now encode rotation about the measured axes of rotation, and the radiometer frames carry the measured line-of-sight boresight, so the generated kernels reflect the true instrument geometry instead of nominal placeholders. The three source unit vectors (LIBERA_AZ_AOR_IN_STAND, LIBERA_EL_AOR_IN_STAND, LIBERA_EL0_Z_IN_STAND) are stored as keywords in the frame kernel; the Az/El CK configs switch from Euler angles to quaternions, and kernel_maker.add_mechanism_ck_quaternions builds the per-sample rotations about the measured axes (applied after the encoder correction).

5.9.3 - 2026-07-24#

Added#

  • Az/El mechanism CKs now apply deterministic sinusoidal encoder-angle corrections during kernel generation, so the stored quaternions reflect the true mechanism rotation rather than the raw encoder readout. Corrections are applied to the filtered encoder angles in create_kernel_from_l1a (L1A telemetry is left unchanged), with matching forward/inverse helpers (correct_azimuth/correct_elevation, uncorrect_azimuth/uncorrect_elevation, and DataFrame-level apply_encoder_corrections/reverse_encoder_corrections)

Fixed#

  • Remove np.NaN as curryer no longer pins Numpy < 2

5.9.2 - 2026-07-17#

Added#

  • Expand the RBSP+VIIRS imager DataProductIdentifier set (l2_unf_rad_imager, l2_nb_bb_imager_camtime, l2_toa_flux_imager, and matching AUX scene-ID/FMATCH/ADM members)

Fixed#

  • rename camera/imager DPI and ProcessingStepIdentifier members for consistency with product strings: l2_unf_cam→l2_unf_rad_cam/l2-unf-rad-cam, l2_cf_rad_time/l2_cf_rad→l2_cf_cam/l2-cf-cam, l2_cf_cam_time/l2_cf_cam→l2_cf_cam_camtime/l2-cf-cam-camtime, l2_nb_bb_cam_time/NB-BB-CAM-TIME→l2_nb_bb_cam_camtime/NB-BB-CAM-CAMTIME/l2-nb-bb-cam-camtime, l2_unf_imager→l2_unf_rad_imager/l2-unf-rad-imager, l2_nb_bb_imager→l2_nb_bb_imager_camtime/l2-nb-bb-imager-camtime

5.9.1 - 2026-07-16#

Added#

  • Scene Identification now reports the property bin min/max bounds alongside the scene ID. For each footprint and classification variable, identify_scenes adds scene_bin_{type}_{variable}_min/_max variables giving the bounds of the matched scene’s bin (unbounded sides and unmatched footprints reported as NaN). Controlled by the new report_bin_bounds flag (default True) on FootprintData.identify_scenes and SceneDefinition.identify_and_update; existing scene_id_{type} columns are unchanged.

  • Add SceneDefinition.get_bin_bounds_for_scene_id and Scene.get_bin_bounds to look up the property bin bounds for a given scene ID.

5.9.0 - 2026-07-14#

Changed#

  • BREAKING: Rename DataLevel.ANC to DataLevel.AUX and split ancillary archive routing into separate AUX and CAL archive buckets.

  • BREAKING: Replace provisional L2/ANC DataProductIdentifier and ProcessingStepIdentifier values with the camera cloud-fraction and RBSP+VIIRS imager product sets for July delivery (renamed product strings, new AUX/L2 members, and updated L2 team IAM role mappings).

  • Update user docs for the new product/level identifiers.

5.8.6 - 2026-07-06#

Added#

  • Manual processing now runs through the SDC event bus instead of triggering Step Functions directly. step-function-trigger emits a ManualProcessing event for a single step, and a new manual-processing CLI submits arbitrary custom DAGs (or the default DAG) across one or more dates, with an optional --verify that polls the Coordination Table.

  • ecr-upload assumes the algorithm’s per-team L2 Team Role for L2/ADM images so non-admin L2 developers can push to their ECR repos; non-L2 steps use the default/--profile session unchanged.

5.8.5 - 2026-06-29#

Added#

  • s3-utils put now performs manual SDC data ingest instead of a direct archive upload. It accepts multiple file paths, stages each file to the SDC Ingest Dropbox bucket, and emits a single NewFilesAvailable event to the SDC event bus; the SDC Data Ingester then handles archiving and metadata/data-availability records. Accepts L0 and data product filenames (manifests rejected).

  • Add get_l2_team_role_session (in aws/utils.py) which builds a boto session from a profile/AWS_PROFILE and assumes an L2 team role (default L2Developer/LiberaUtils), raising ValueError if the base profile is not permitted to assume it. s3_put_cli_handler now uses it as its session source.

  • Add --verify/--timeout to s3-utils put. With --verify, the CLI blocks after emitting the ingest event and polls (read-only) until each file is confirmed in its archive bucket and recorded in the File Metadata table (and, for non-L0 products, the Data Availability table), logging a per-file summary and raising on timeout (default 5 minutes). Adds find_dynamodb_table_in_account_by_partial_name to aws/utils.py.

Changed#

  • Remove the unused libera_utils/db/dynamodb_utils.py module (stale write-side helpers and an unused get_dynamodb_table); the ingest-verification DynamoDB reads are inlined in verify_ingestion.

5.8.4 - 2026-06-08#

Added#

  • KernelManager.load_naif_kernels also loads an extended Earth PCK (earth_*_predict.bpc via NAIF_EARTH_EXTENDED_PCK_REGEX) so ITRF93 orientation is available beyond the short high-precision earth_000101_*.bpc window (required for JPSS geolocation and future-epoch processing).

5.8.3 - 2026-06-04#

Note: NOM-HK L1A changes the shape and dtype of five variables (replacing per-index fields with indexed arrays). Treat this as breaking for external consumers of those variables; a patch bump was chosen during early development. Changed variables: ICIE__SW_FP_WP_ST_WP, ICIE__SW_SEQ_EXEC_BUF_OP, ICIE__SW_SEQ_EXEC_POS_OP, ICIE__SW_SEQ_ST_OP, ICIE__SW_SEQ_STOP_CD_OP

Added#

  • Add NOM-HK array groups to stack waypoint status and sequence execution fields into indexed arrays (ARRAY_128 and ARRAY_8)

Fixed#

  • Normalize Unicode packet string fields during aggregation to ensure deterministic byte sizing

5.8.2 - 2026-06-03#

Added#

  • Faster ICIE WFOV SCI (APID 1040) packet-to-L1A by refreshing icie_xtce_tlm.xml and combining the 972-field WFOV aggregation_groups block from icie_wfov_sci in l1a_processing_configs.yml.

5.8.1 - 2026-06-03#

Added#

  • Added alpha calibration combination constants for data products

5.8.0 - 2026-05-15#

Added#

  • KernelFileCache can materialize kernels from local filesystem paths (Path or non-HTTP str), with documented resolution rules for relative paths and stable cache freshness after copy.

  • KernelManager caches Libera static kernels under the versioned user cache via KernelFileCache after building them under the existing short temporary directory. When every required artifact is already cached (and within cache_timeout_days), static kernel creation is skipped and kernels are furnished from the cache, consistent with NAIF generic kernels.

Changed#

  • BREAKING: KernelManager.load_libera_dynamic_kernels accepts only a Sequence[str | pathlib.Path | S3Path] (manifest-ordered path list). Directory expansion, scalar Path, and scalar local str are not accepted (use [path] for a single file). Materialization is inlined with a single normalization pass per entry; _materialize_dynamic_kernel_paths / _is_dynamic_sources_sequence are removed from the dispatch path.

  • BREAKING: Renamed load_libera_dynamic_kernels argument dynamic_kernel_directory to dynamic_kernel_sources (broader semantics; no compatibility alias).

  • Improve KernelFileCache docstrings and spice_utils type annotations (including ensure_spice overloads and tighter return types on kernel helpers).

5.7.1 - 2026-05-13#

Added#

  • Added ground data option for deduplication to allow ground data processing to continue

Fixed#

  • Fix packet parsing to l1a of datasets with deduplicated timestamps for final data assembly

  • With ground_data=True, non-identical duplicate coordinates no longer raise when verbose=False; warnings only if verbose=True.

  • Duplicate-coordinate diagnostics no longer cite the source sequence counter when that coordinate is absent from the dataset slice (e.g. inside sample groups).

5.7.0 - 2026-05-01#

Added#

  • Added metadata writer to make UMM-G metadata files from NetCDF-4 data files

5.6.0 - 2026-03-19#

Added#

  • Add dimension validation to data product definition

  • Support safer enforcement of product definition specs on data products

Changed#

  • Improve logging and warning visibility in data product writer

5.5.6 - 2026-03-19#

Added#

  • Adding support for APIDs 1000, 1002, 1035, 1043, and 1044 to support calibration and status telemetry packets

5.5.5 - 2026-03-02#

Changed#

  • Update the ICIE TLM XTCE to latest version

5.5.4 - 2026-03-02#

Changed#

  • Port Make_Kernel and SPICE-based time conversions to spice_utils to avoid circular imports

Fixed#

  • Implement KernelManager for Make_Kernel, furnishing necessary kernels

5.5.2 - 2026-01-27#

Fixed#

  • Improve resiliency of JSON logging system for cloudwatch logs

5.5.1 - 2026-01-27#

Fixed#

  • Fixed the netcdf writing needed for L1B camera product and clarifying netcdf engine usage

5.5.0 - 2026-01-27#

Added#

  • Allows the usage of the profile argument in the cli tools for specifying AWS credentials profiles

Changed#

  • Adjusted CLI interface for AWS tools to improve usability and consistency

  • Changing the s3 utilities for list and put to use the new constants tools more effectively

5.4.8 - 2026-01-22#

Added#

  • Add lookups between L1a, APID, and L1a product definitions

5.4.7 - 2026-01-22#

Added#

  • Update Space Packet Parser dependency to take advantage of packet filtering functionality

5.4.6 - 2026-01-22#

Fixed#

  • update xarray version due to breaking change in netcdf writing

5.4.5 - 2026-01-16#

Added#

  • Add enforcement of valid versioning in Filename classes

5.4.4 - 2026-01-13#

Added#

  • Allow passing dynamic product level attributes to data product writer

5.4.3 - 2026-01-10#

Added#

  • Allow adding the archive prefix to the path when calling write_libera_data_product

5.4.2 - 2026-01-09#

Changed#

  • Change xarray netcdf engine to h5netcdf because netcdf4 does not support writing to filelike objects (e.g. opened files).

5.4.1 - 2026-01-09#

Added#

  • Added a KernelManager class to handle SPICE kernel caching and retrieval

5.4.0 - 2025-12-08#

Changed#

  • Refactor kernel_maker to use L1a Dataset object. Packet-based kernel creation is still possible via create_kernels_from_packets, which runs L1a processing internally

  • Update Curryer kernel configs to match Libera AXIS_SAMPLE packets fields

  • Move the L1A processing configurations to YAML instead of Python objects

Fixed#

  • Fix bug in packets.py::_drop_duplicates to allow dropping based on coordinate or dimension

5.3.0 - 2025-12-08#

Added#

  • scene_id module alpha release capable of identifying scenes from footprint data in Ceres SSF files.

5.2.2 - 2025-12-03#

Changed#

  • Updated the Curryer version pin to use at least 0.2.0, which is when the problematic subdependencies were removed

5.2.1 - 2025-11-14#

Added#

  • Added scene_id module for processing footprint data to produce datasets with assigned scene IDs according to libera_utils standard definitions.

5.2.0 - 2025-11-12#

Added#

  • Implement processing to L1A Dataset from packets

Changed#

  • Add full ICIE XTCE to Libera Utils package

5.1.0 - 2025-11-07#

Added#

  • Added UMMGranule Pydantic class to validate datasets against the UMM-G schema

  • Added methods to convert Libera datasets to UMMGranule instances

5.0.0 - 2025-11-07#

Added#

  • Add write_libera_data_product to netcdf.py for one-shot product writes from numpy arrays

Changed#

  • BREAKING: Refactor DataProductDefinition (previously DataProductConfig) and product definition validation

  • Remove hdf.py and associated tests

Removed#

  • BREAKING: Remove L1A ProcessingStepIdentifier (node) enum constants

4.0.0 - 2025-09-18#

Added#

  • New DataLevel enum: Added structured processing level enumeration with built-in archive bucket name mapping

  • Added the Libera APID values we care about for science processing to the LiberaApid enum class

Changed#

  • BREAKING: Refactored identifier classes: Moved DataProductIdentifier and ProcessingStepIdentifier from libera_utils.aws.constants to libera_utils.constants

  • BREAKING: SPICE kernel filename consolidation: Merged separate SPICE kernel filename classes into LiberaProductFilename

  • Enhanced DataProductIdentifier: Now includes embedded metadata with processing level information and simplified lookup methods

  • Enhanced ProcessingStepIdentifier: Now includes embedded product relationships and improved level derivation

  • Filename validation: Updated L0 filename patterns to remove chunk number validation

  • Improved type safety: Better type hints and validation throughout identifier classes

  • Updated documentation: Refreshed user documentation to reflect new file naming conventions

Removed#

  • BREAKING: Removed L0 file chunking support: The chunk_number parameter and related methods have been removed from L0 data product identifiers

  • BREAKING: Removed deprecated classes: ProductName, CkObject, and SpkObject enums have been removed and consolidated into DataProductIdentifier

3.2.4 - 2025-09-11#

Added#

  • Added the SPICE kernel creation Docker image

  • Added the Geolocation Tier-1 integration test case and data files

Changed#

  • Changed the SPICE kernel creation API

3.2.3 - 2025-09-11#

Added#

  • Added step function and policy names to ProcessingStepIdentifier

Changed#

  • Improved ECR authentication issues with the libera-utils ecr-upload CLI command for working with SSO authentication

  • Updated S3 list tool to print by default

  • Improved tooling of the aws constants module for both DataProductIdentifier and ProcessingStepIdentifier

  • Updated constants for L2 data products

  • Updated step function trigger tool with url output and naming improvements

  • Exposed useful L2 tools at highest level of the libera_utils package

3.2.2 - 2025-06-27#

Added#

  • Added dimension handling for data product configuration

  • Added internal DataArray data storage matching to known dimensions

  • Added jenkins file for publishing to PyPI

  • Added contact listserv for project

  • Added json and yaml linting to the pre-commit hooks

  • Added prettier configuration for formatting markdown, json, and yaml files

  • Added internal Dataset object to the Configurable NetCDF-4 metadata object

  • Added writing of output file for the Configurable NetCDF-4 data product object

3.2.1 - 2025-06-13#

Added#

  • Added automatic formatting to the repository using pre-commit hooks

  • Added a list-serv for ci/cd output as part of the Jenkins files

Fixed#

  • Fixed filenaming changes from 3.2.0 that affected the CDK deployment and usage

3.2.0 - 2025-05-19#

Added#

  • Added pydantic models for a Configurable NetCDF-4 metadata object

Changed#

  • Includes static metadata that is populated and known by these tools

  • Includes dynamic metadata that must be provided by user of these tools

  • Changed the naming scheme of the SPICE data files to include SPK or CK

3.1.1 - 2025-05-08#

Fixed#

  • Testing improvement for mocking docker image building

3.1.0 - 2025-05-08#

Added#

  • Added Curryer (lasp-curryer) library dependency for SPICE kernel creation and geolocation

  • Added static SPICE kernels and configuration files for geolocating NOAA-20 / CERES

Removed#

  • BREAKING: Removed geolocation submodule, replaced by interfaces within Curryer

3.0.0 - 2025-04-25#

Changed#

  • BREAKING: Updated the aws constants ProcessingStepIdentifier dump method to be to_str_with_chunk_number

  • Improved the standardization of CLI commands to call cli_handler functions that wrap the main functionality

Removed#

  • BREAKING: Removed support for python version 3.9 and 3.10

2.5.2 - 2025-03-03#

Added#

  • Added the s3-utils cli interface with subcommands put, list, and cp for ease of use s3 interactions

  • Added the Libera Archive bucket naming pattern to the aws constants module

Changed#

  • The cli subcommand libera-utils s3-utils put will upload a file to the correct S3 archive bucket given an algorithm

  • The cli subcommand libera-utils s3-utils list will list all files in a given S3 archive bucket for a given algorithm

  • The cli subcommand libera-utils s3-utils cp will copy a file from one S3 archive bucket to another location

  • Improved type hinting for the smart_open module

  • Improved readability of cli testing

Removed#

  • Removing testing against python 3.9 and 3.10

2.5.1 - 2025-01-22#

Fixed#

  • missing pyyaml dependency that prevented usage of logutil module

2.5.0 - 2024-12-14#

Changed#

  • Reimplement Manifest class as a pydantic model and integrate with dependent code

  • BREAKING: Rename outpath to out_path on the Manifest class write method

Removed#

  • BREAKING: Remove the deprecated add_file_to_manifest method from Manifest class

2.4.5 - 2024-11-27#

Added#

  • Add --ecr-image-tags option to libera-utils ecr-upload CLI for tagging remote algorithm images in an ECR

  • Add validation and serialization methods to the DataProductIdentifier and ProcessingStepProductIdentifier enums

2.4.4 - 2024-10-21#

Changed#

  • Allow overriding the standard docker config.json file with a minimal file for ECR uploading to prevent ECR upload permission failures from cached docker login credentials from CDK deployments

2.4.3 - 2024-10-16#

Changed#

  • Changes to ecr_upload to support programmatic building and pushing of Docker images

Removed#

  • Remove DynamoDB docs (moved to libera_cdk)

2.4.2 - 2024-10-14#

Removed#

  • BREAKING: Remove the AnyFilename polymorphic class. Please use AbstractValidFilename.from_file_path()

2.4.1 - 2024-10-04#

Changed#

  • Updating requirements of methods to use keyword arguments rather than positional arguments

  • Adding ProcessingStepIdentifier and DataProductIdentifier to the filenaming classes

  • Updating ecr names to work with the completion checker testing in libera_cdk

2.4.0 - 2024-09-11#

Added#

  • Add properties to filenaming classes to retrieve data_product_id and processing_step_id

  • Add ProcessingStepIdentifier and DataProductIdentifier standardization to be used by downstream repos

2.3.1 - 2024-08-28#

Fixed#

  • Fix os.path.join bug in filenaming module that broke mocked S3 paths and also fix typehinting

2.3.0 - 2024-06-27#

Added#

  • Create CLI tools for AWS ECR image upload and Step Function triggering

  • Create tools for DynamoDB in AWS for .pds files (CONS and PDS)

Changed#

  • Update manifest filenames to use ULID instead of timestamp for unique identifiers

  • Change logutil configure_task_logging to optionally log JSON to console

  • Allow configure_task_logging to optionally propagate DEBUG messages from specific loggers

  • Update documentation for how the database is used in the Libera project in DynamoDB

  • Replace the use of PostgreSQL with DynamoDB for the Libera project

2.2.0 - 2023-12-14#

Added#

  • Add AnyFilename polymorphic class

  • Add prefixing to Filename classes for predictable archive paths

  • Add prefixing for manifest files for predictable and navigable paths in s3 buckets

  • Added handling of construction records and pds files appropriately when ingesting

    • This includes reading a construction record and removing the pds file entry for the construction record itself

  • Added output manifest creation from input manifest to match timestamps in filenames of input and output manifests

  • Added error handling to pds ingest

Changed#

  • Change filename of all products to a LiberaDataProductFilename that inherits from AnyFilename

  • Update filenaming convention to be all capital letters

  • Improve API for manifest module

  • Update git to include lfs and move test data to lfs

  • Improve database manager including caching improvements

  • Improve smart_copy_file and bug fixes to smart_open testing

  • Refactoring and improving pds ingest for database entries and integration testing in CDK

  • Improved testing of pds ingest and pds file orm models to more accurately reflect use cases

  • Refactored pds ingest to use AnyPath objects for handling file locations

2.1.1 - 2023-05-16#

Added#

  • Add preliminary instrument kernel

Changed#

  • Update dependency specification to speed up dependency resolution wrt botocore/urllib3

  • Improve database initialization to work with libera_cdk changes

Fixed#

  • Fix bug in Dockerfile that incorrectly set the default entrypoint

2.1.0 - 2023-04-20#

Added#

  • Add manifest filename enforcement to Manifest class

Changed#

  • Improve API to Manifest and Manifest.add_files

  • Update filenaming conventions for product filenames and SPICE kernels

  • Allow adding an s3 bucket/prefix as a basepath for filenames

2.0.1 - 2023-02-07#

Removed#

  • Remove the extras dependency spec because of the way SQLAlchemy imports models

2.0.0 - 2023-01-31#

Added#

  • Add filenaming classes

  • Add manifest file class

  • Add construction record parser

  • Add smart_copy_file function that can copy files to and from S3 and filesystem locations transparently

  • Add quality flag classes

Changed#

  • Update DB schema to store construction records

  • Update kernel generation CLI to use manifest file pattern

  • Shift database and spice related libraries to extras (not installed by default)

  • Change license to BSD3

Removed#

  • Remove HDF-EOS5 filehandling code

1.0.0 - 2022-09-23#

Added#

  • Add build and release processes to readme

  • Add geolocation module

  • Add tools in spiceutil module for caching SPICE kernels from NAIF

  • Add missing unit testing coverage

  • Add spice.md documentation on how the package uses and manages SPICE kernels

  • Add database tooling, dev database, and ORM setup

  • Add smart_open for opening local or S3 objects

  • Add logging utility functions for setting up application logging

Changed#

  • Stub out project structure

  • Switch to Poetry for project dependency configuration and build management