Changelog#
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased#
5.25.0 - 2026-11-11#
Added#
libera_utils.footprint_matching.productassembles and writes the FMATCH products for all four operational modes (assemble_fmatch_dataset,write_fmatch_product,aggregate_external_variables,build_radiometer_footprints,compute_derived_viewing_geometry).Geolocation and viewing angles pass through from L1B, keyed by their L1B RAD-4CH names (
L1B_PASSTHROUGH_VARIABLES), which are also the product names.PSF_Bbox_*,Sunglint_Angle,PSF_Coverage_Fraction, the aggregated external variables and the CERES 75%/95% coverage bits inQuality_Flagare computed.Sunglint_Angleuses the L1B relative azimuth convention, so it is zero at the specular direction.Off-limb records are kept and written as fill with zero coverage. Footprints are weighted with
AngularPSFWeigherby default; passRadialWeigherfor footprints without a cone-angle rate.Missing inputs raise rather than write fill: a variable no input produced raises
ValueErrorunlessallow_placeholders=True, and an unusable ephemeris raisesMissingGeolocationError. Camera-timescale modes cannot be aggregated yet.Integer category and pixel-index variables use -1 (
FMATCH_INTEGER_FILL_VALUE) as fill; bitmasks such asQuality_Flaguse 0.
aggregate_external_variablesruns on a process pool when theFMATCH_WORKERSconfiguration key or environment variable is above one, with the same output as a single-process run.AncillaryReader.preload()loads a reader’s whole-run data before the first query.
Changed#
lasp-curryerrequires>= 0.5.3, <1.0, forcurryer.compute.spatial.relative_azimuth.
5.24.0 - 2026-11-11#
Added#
Add
libera_utils.aws.ecr_upload.ImageTagAlreadyExistsError, aValueErrorthatpush_image_to_ecrandecr-uploadraise when a requested tag other thanlatestalready exists in the ECR repository, whatever image it holds: found by a check before anything is pushed, or refused by ECR during the push when another push took the tag after that check. The message names every such tag and the repository, links the repository’s page in the AWS console, and says to push under a new version tag, or to delete the image under the tag by hand and push again. A refused push is not retried, andecr-uploadregisters nothing.Product definitions for the FMATCH-CAM(-CAMTIME) and FMATCH-IMAGER(-CAMTIME) products. FMATCH-IMAGER also carries the RBSP CLDPIX/FLASHFlux cloud fields and the extended SSF cloud/aerosol/albedo variables (multi-axis SSF flattened to 1-D per footprint, carried by IMAGER alone through their
VariableSpec.modes), and the five ERA5 pressure-level fields and their standard deviations as 2-D variables on(RADIOMETER_TIME, PRESSURE_LEVEL)(e.g.ERA5_Pressure_Temperature) with aPRESSURE_LEVELcoordinate in hPa.PRESSURE_LEVEL(size 37) joins the standard dimensions. A reader declares such stacked product variables withPRODUCT_STACKS(ProductStack), andAncillaryReader.product_stacks()adds each stack’s standard-deviation companion stack; a stack that names a spec the reader does not produce, stacks a spec twice, takes a product spec’s name or mixes members with and without companions raisesTypeErrorwhen the reader class is defined.Cloud_Fraction_Camerain FMATCH-CAM(-CAMTIME) is defined in percent[0, 100]to match the CF-CAM input. FMATCH-CAM(-CAMTIME) carry no ERA5 10 m winds or VIIRS BRDF kernel weights, since no consumer of the CAM products reads them. Every integer category, pixel-index and camera pixel-bound variable declares_FillValue: -1, since 0 is a real value in each; theQuality_Flagbitmask and thePSEUDOFOOTPRINTindex declare none. Dimensionless variables carry nounitsattribute (CF §3.1), andVIIRS_CLDPROP_D3_Cloud_Optical_Thicknesscarries the CLDPROP_D3 source’s combined-phase long_name and itsvalid_min/valid_max[0, 150].libera_utils.footprint_matching.aggregation.stack_product_valuescollects a reader’s stacked specs into one array per stacked product variable and raisesValueErrorwhen a member’s scalar is missing.
Changed#
BREAKING: Before pushing,
push_image_to_ecrandecr-uploadcheck ECR for every requested tag other thanlatestand push nothing if any of them already exists, even in a repository that would accept the push, which used to replace the image under the tag. A session that pushes a version tag now needsecr:DescribeImageson the repository, and a missing repository fails the check before any push. Tags other thanlatestare then pushed beforelatest, so a tag taken by another push since the check leaveslatestwhere it was.BREAKING: SCENE-ID-CAM and SCENE-ID-CAM-CAMTIME variables take their product names (
Scene_ID_ERBE,Scene_ID_Unfiltering,Scene_Bin_ERBE_Cloud_Fraction_Min,Scene_Bin_ERBE_Solar_Zenith_Surface_Min,Surface_Type,IGBP_Surface_Type,Cloud_Fraction,Solar_Zenith_Surface,Camera_Pixel_X_Min, …).FootprintDataholds each variable a product writes under its product name (theFootprintVariablesvalues), socreate_and_write_data_productwrites the declared variables as they are.BREAKING: The VIIRS BRDF reader’s variables drop the repeated
brdf_prefix (brdf_shortwave_fiso→Shortwave_Fiso, and likewise for each band and kernel weight), and the VIIRS AOD reader’saod_550isOptical_Depth_550, so the FMATCH product variables areVIIRS_BRDF_<Band>_<Param>andVIIRS_AOD_Optical_Depth_550.BREAKING: FMATCH reader
READER_KEYs are their product-variable prefixes (CLDPIX,ERA5_Pressure,ERA5_Surface,FLASHFlux,IGBP,NISE,VIIRS_AOD,VIIRS_BRDF,VIIRS_CLDPROP_D3), soReaderRegistry.get,FMATCH_MODE_READERSand thesource_file_pathskeys ofbuild_sourcesuse them. Reader spec names are Camel_Case (Sea_Ice_Concentration,Surface_Type_Primary,Temperature_500hPa), andSTANDARD_DEVIATION_SUFFIXis_Standard_Deviation.BREAKING: The shipped scene definitions are
ERBE.csv,TRMM.csvandUnfiltering.csv. Their columns end in_Min/_Maxand name the classification variablesSurface_Type,Cloud_Fraction,Solar_Zenith_Surface,Viewing_Zenith_SurfaceandRelative_Azimuth_Surface.SceneDefinition.typeis the CSV filename stem as spelled, and its outputs areScene_ID_{type}andScene_Bin_{type}_{variable}_Min/_Max. A column ending in lower-case_min/_maxraisesValueError.SCENE-ID-CAM and SCENE-ID-CAM-CAMTIME take the L1B RAD-4CH and L1B CAM
long_names for the variables they pass through from L1B (Solar_Zenith_Surface,Viewing_Zenith_Surface,Relative_Azimuth_Surface, and in SCENE-ID-CAM-CAMTIMELatitude,Longitude,Altitude), and theirCloud_Fractionlong_namesays the CF-CAM values are in percent rather than scaled to percent. Units and values are unchanged.
Removed#
BREAKING:
OperationalMode.IMAGER_FLASH(FMATCH-IMAGER-FLASH). No FMATCH-IMAGER-FLASH product is planned, so the CERES FLASHFlux reader feeds FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME only.BREAKING: The ERA5 surface reader’s 10 m winds and the VIIRS BRDF reader no longer serve the CAM modes:
ERA5_SurfaceandVIIRS_BRDFfeed FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME only, so they leave the CAM entries ofFMATCH_MODE_READERS, andbuild_sourcesfor a CAM mode rejects theirsource_file_pathskeys.
5.23.0 - 2026-11-05#
Added#
Add the Packet Ordering page to the user guide, which describes how the L1A
PACKETaxis is put inSRC_SEQ_CTRorder, what raises and warns, and the losses the counter cannot size. It replaces the packet axis order section of the L1A Processing page.Add
libera_utils.l1a.packet_ordering.check_packet_order(ds, packet_time_coordinate), which emits aUserWarningwhere packet time steps backward along thePACKETaxis, with the count, the largest step and the first one’s counters and times. It raisesValueErroronly whenSRC_SEQ_CTRor the packet time variable is missing.Add
libera_utils.l1a.packet_ordering.sort_by_sequence_counter(ds, packet_time_coordinate), which orders thePACKETaxis bySRC_SEQ_CTRand drops repeated packets asparse_packets_to_l1a_datasetdoes; it reorders by counter only between packets withinREORDER_WINDOW(10 s) of each other, places packets by packet time across a counter step that repeats or goes backward further apart, and logs each such cut at INFO with the counters and packet times either side of it; when it reorders, it consumes its input, removing thePACKETvariables from the dataset passed in, so use the returned dataset. Also addlibera_utils.l1a.packet_ordering.out_of_order_positions(ds, packet_time_coordinate), which returns thePACKETpositions whereSRC_SEQ_CTRrepeats or steps backward between packets within 10 s of each other.
Changed#
BREAKING:
parse_packets_to_l1a_datasetorders thePACKETaxis bySRC_SEQ_CTRinstead of by packet time, whatever the order of the input files; packet time only resolves which counter wrap a packet falls in. Published{group}_packet_index,CAMERA_PACKET_INDEXandPACKET_IMAGE_IDvalues change, and packet time is not monotonic alongPACKET; sort packet times beforenumpy.searchsortedornumpy.interp. Input files are parsed in file name order, whichinput_fileslists.BREAKING:
parse_packets_to_l1a_datasetdrops duplicate packets bySRC_SEQ_CTRinstead of by packet time, so a distinct packet that shares a packet time with another is kept. Packets that share a packet time but not a counter are no longer treated as duplicates. Packets that share a counter but differ raiseValueErrorunlessground_data=True, which keeps the first. The duplicate warning reads “Dropped N packet(s) received twice: same SRC_SEQ_CTR and contents as a kept packet”, or, whenground_data=Truekeeps the first of packets that differ, “Dropped N packet(s) that repeat a kept packet’s SRC_SEQ_CTR” with the counts of each.BREAKING:
nom_hk_trim.find_obsid_runsraisesValueErrorfor a NOM-HK dataset withoutPACKET_ICIE_TIMEorSRC_SEQ_CTR, or whoseSRC_SEQ_CTRrepeats or steps backward between packets within 10 s of each other, which is the case for a NOM-HK product written in packet-time order by an earlier release; decode its L0 files again.BREAKING:
parse_packets_to_l1a_datasetraisesValueErrorwhere packets adjacent in an input file with consecutiveSRC_SEQ_CTRwould be placed a counter wrap apart, for ground and flight data alike. A single packet whose packet time would place it a wrap away from both such neighbors is instead dropped, with aUserWarningnaming its counter and packet time. This includes a join between two input files whose names sort against the time order of their data, when the file that sorts first, holding the later data, ends one counter below where the next file starts.
Removed#
BREAKING: Remove
packets.SRC_SEQ_CTR_DIMENSION.
Fixed#
LiberaDataProductDefinition.generate_data_product_filenametakes the start and end from the minimum and maximum of the time variable, not its first and last values.
5.22.0 - 2026-11-04#
Changed#
An
output-productsentry in amanual-processingcustom DAG may be a mapping withidandoptional, as well as a bare data product id string.optional, on an input or an output, needs an SDC deployment that includes libera_cdk PR #96 (LIBSDC-892); the Job Creator of an earlier deployment rejects an output mapping and ignoresoptionalon an input.
5.21.0 - 2026-11-02#
Added#
Add
libera_utils.l1a.day_window.day_bounds(day)returns the day and its buffered window,DAY_BUFFEReither side, as timezone-aware UTC datetimes.trim_l1a_to_day_window(ds, day=)keeps the packets in the closed window[D − DAY_BUFFER, D + 1 + DAY_BUFFER](10 minutes) throughslice_l1a_dataset_to_time_window, selecting on every sample group and keeping whole packets.check_data_times(ds)logs a warning for each data-time axis that repeats a time or steps backwards; it does not raise.packet_slicinghandles WFOV SCICAMERA_TIME: an image selected on camera time keeps every packet from SOP through EOP,CAMERA_PACKET_INDEXis renumbered and keeps itsint32dtype,PACKET_IMAGE_IDis renumbered against the survivingCAMERA_TIMEaxis, with-1for a kept packet whose image row was dropped, and packets in no image (PACKET_IMAGE_ID == -1) are dropped. Slicing WFOV SCI withoutPACKET_IMAGE_IDraisesValueError.Add
packet_slicing.data_time_axes(ds): a dataset’s sample axes, or its packet time coordinate when it has none. Slicing, the day trim andcheck_data_timesselect on these axes.Add
libera_utils.l1a.day_coverage.evaluate_day_coveragefor the L1A day-assembly completeness check over the buffered day window[D − DAY_BUFFER, D + 1 + DAY_BUFFER]. Each APID’sDayCoveragePolicyis theday_coverageblock of its entry inl1a_processing_configs.yml:FRACTIONpasses a day when its L0 spans reach both window edges (covers_window) and cover at leastwindow_coverage_fracof the window (window_frac);PRESENCEpasses on any presence in the buffered window, tested byoverlaps_day_window. Span endpoints are timezone-aware; a naive one raisesTypeError. Spans separated by up toDEFAULT_SEAM_TOLERANCE(1 s) merge, and a span within it of a window edge reaches the edge, because a file’s span runs from its first to its last sample timestamp and under-reports its occupancy by one sample period.Add
libera_utils.l1a.day_coverage.measure_time_axis_coverage, which measures day and buffer occupancy from a granule’s own time axis, in one pass: each sample occupies the spacing to the next, except across a gap, a spacing aboveDEFAULT_GAP_FACTOR(5) times the median cadence. A single sample occupies no time. Unlikeevaluate_day_coverage, it sees gaps inside a file.Add
libera_utils.l1a.l1a_packet_configs.configured_apids(), the APIDs with a packet configuration.Add the
l1a-processingCLI (libera_utils.aws.l1a_join.request_l1a_processing), which emits anL1AProcessingRequestevent asking the L1A Preprocessor to assemble an APID’s day for one applicable date, running the completeness check unless--forceis given. It accepts only APIDs with a packet configuration. It builds its session from--profile, or the default profile, and assumes no role._to_dateinmanual_processingmoves tolibera_utils.time.to_date.Add a
max_workersargument (default 1) toparse_packets_to_datasetandparse_packets_to_l1a_dataset, which parses files concurrently inforkworker processes, keeping the packet axis in the order given. A worker reads an S3 path with the client the path carries, or with the default S3 client for a URI string, over its own connections. A worker’s exception is re-raised with the worker’s traceback attached as a note. A worker that dies without reporting raisesRuntimeErrorwith its exit code, andmax_workersbelow 1, or above 1 on a platform withoutfork, raisesValueError.Add
libera_utils.io.smart_open.is_resource_exhaustion(error), which is true forMemoryErrorand for anOSErrorwhose errno isENOSPC,EDQUOT,EMFILEorENFILE. Only the error itself is examined, not its__cause__.
Changed#
BREAKING:
PacketConfigurationrequires aday_coveragefield (DayCoveragePolicy), and every entry inl1a_processing_configs.ymlcarries one.BREAKING:
slice_l1a_dataset_to_time_windowno longer takespacket_time_var; a dataset with no sample axis is sliced on the packet time coordinate its APID’s packet configuration names.parse_packets_to_datasetaccepts a loadedXtcePacketDefinitionas well as a path.parse_packets_to_datasetparses files one at a time onto a preallocated packet axis instead of passing them tocreate_datasettogether, which cost ~800 MiB per two-hour NOM-HK file and ran a full day out of memory. It skips a file with no packets of the APID, and raisesValueErrorwhen files parse to different variable sets.manual-processingandstep-function-triggertake the UTC calendar date of a timezone-aware date or datetime.Document daily L1A assembly (
evaluate_day_coverage→ multi-file parse → trim →check_data_times→measure_time_axis_coverage, then a report file for each L1A granule, not yet written).
Removed#
BREAKING: Remove
packet_slicing.DEFAULT_PACKET_TIME_VAR.BREAKING: Remove the
WFOV_HEADER_PARSE_VALIDvariable from WFOV SCI L1A products, andwfov_image_metadata.WFOV_HEADER_PARSE_VALID_VAR. It was always true: an image whose header cannot be decoded is discarded and counted inHeaderParseErrorCount.
Fixed#
manual_ingest_data_products(s3-utils put) splits its files acrossNewFilesAvailableevents of at mostMAX_FILES_PER_INGEST_EVENT(5) files, keeping a construction record and the PDS files that share its SCID, first APID, created time and numeric ID in one event; PDS files without their construction record are split at the same limit. A single event holding a day of ground CCSDS captures could not be archived inside one 10 minute Data Ingester invocation.BREAKING:
extract_data_time_rangeandscan_ground_ccsds_filelet an error for whichis_resource_exhaustionis true, such as a full local disk while an S3 file downloads to the cache, propagate unchanged instead of raisingDataTimeUndeterminedErroror returningNone.
5.20.0 - 2026-10-30#
Added#
libera_utils.footprint_matching.weighting:RadialWeigherandAngularPSFWeigherweight each native sample where it lies, and eachWeightFieldcarries the footprint’scoverage: the share of the PSF’s energy inside the native cell of at least one finite sample (each sample covers the square of half-widthRESOLUTION_KM / sqrt(2), foreshortened along-scan bycos(VZA)in the angular frame, measured on a reference grid in the weigher’s own frame), so a swath gap or a source beyond its time window reads as missing coverage.AngularPSFWeigherraisesValueErrorwhencone_angle_rateis missing or not finite, or the viewing zenith is not finite.libera_utils.footprint_matching.aggregation.aggregate_sample_variables, one scalar per product variable from a footprint’s weighted samples, raisingValueErrorfor a product variable it cannot map to a read variable or whose aggregation is not a registered strategy, and for values or weights that do not match the reader’s variables and samples.check_coverage(coverage)applies the CERES 75 % / 95 % rule toWeightField.coverage.The
weighted_positive_stdaggregation: the_standard_deviationcompanion of aweighted_log_meanvariable is the spread of the same positive samples the geometric mean averages.Optical depths are aggregated as CERES does: imager-pixel cloud optical depth (CLDPIX
cloud_optical_depth) with the PSF-weighted log mean (CERES SSF-85), and every aerosol optical depth (VIIRS Deep Blueaod_550, SSFaerosol_optical_depth,match_aotand the imager land and ocean AODs) and every optical depth its source has already averaged (SSFcloud_optical_depth_lower/_upper, VIIRS L3cloud_optical_thickness) with the PSF-weighted arithmetic mean (CERES SSF-137).notebooks/05_weighting_aggregation.ipynb, a demonstration of the weighers and the aggregation engine.footprint_matching.geometry.psf_ground_radius_kmandslant_range_km: the PSF’s analytic ground radius and the satellite-to-ground distance at a viewing zenith, used by the PSF weighers.
Changed#
notebooks/04_sources_query.ipynbalso demonstrates coverage.
5.19.0 - 2026-10-30#
Added#
libera_utils.footprint_matching.sources.build_sources, which instantiates the readers anOperationalMode’s product uses over their staged granules and returns them keyed by reader registry key. It raisesKeyErrorwhen the staged mapping is missing a source the mode uses or names one it does not,TypeErrorwhen an entry is a single path rather than a list,ValueErrorwhen an entry holds no granule, andFileNotFoundErrorwhen a staged granule is not an existing file; construction opens no granule, so an unreadable one raises from the firstsamples_inthat loads it. Each reader holds its source once (a static reader its whole indexed body of data, a stepped reader the one hourly step nearest the current footprint, a pass reader the granules in its time window).notebooks/04_sources_query.ipynb, a demonstration of the box query and time matching.
5.18.0 - 2026-10-29#
Added#
The swath readers CERES FLASHFlux (
flashflux,FLASHFluxReader, reading the RBSP FLASHFlux product in the CERES SSF file layout for FMATCH-IMAGER-FLASH, FMATCH-IMAGER and FMATCH-IMAGER-CAMTIME, with its extended cloud/aerosol/albedo fields carried by FMATCH-IMAGER alone) and CERES CLDPIX, and the VIIRS daily L3 cloud-property readerviirs_cldprop_d3(VIIRSCldpropD3Reader,CLDPROP_D3_VIIRS_NOAA20,DAILY), completing the nine-reader registry. Every FLASHFlux and CLDPIX field, geolocation and observation time included, is read with its own_FillValueandvalid_range, and every VIIRS cloud-property field with its own_FillValue; a field without one raisesValueError. CLDPIX cloud phase and mask codes that the file’sflag_meaningslabelbad_dataare dropped before the mode, and a categorical field withoutflag_values/flag_meaningsraisesValueError. A FLASHFlux granule missing a group or variable the reader reads, or whose variable has other dimensions than the reader expects, raisesValueErrornaming it and the file’sproduct_version, as does a CLDPIX granule missing a variable the reader reads. A time variable without aunitsattribute raisesValueError. The FLASHFlux scene and ADM type codes are int16. The FLASHFlux and CLDPIX readers arePassReaders (TIME_MATCH = NEAREST_PASS): they convert CLDPIX colatitude to latitude and normalize 0..360 longitudes to -180..180, raisingValueErrorwhen a CLDPIXLatitudedeclares avalid_rangeother than [0, 180] or a FLASHFlux footprint latitude is not indegrees_north, read each footprint’s (Time_and_Position/time) or pixel’s (Observation_Time, fractional hours each dated to the day nearesttime_coverage_start) observation time, serve every footprint or pixel as its own time-stamped sample from the pass nearest the footprint time, the samples withinPASS_WINDOWof the observation time of the sample nearest the footprint time, so a footprint midway between two passes gets only the nearer (none beyondMAX_TIME_OFFSET), choosing that sample from those inside the footprint’s own box and from the padding only when none falls inside, hold only the hourly granules within that offset of the current footprint, and measure the granule’s own sample spacing (PointSamples.spacing_km, the larger of the along-scan and along-track spacings) for the coverage measure in place of the nominalRESOLUTION_KM.PassReaderjoinsStaticReaderandSteppedGridReaderas a reader base, andReaderRegistry.get_readers_for_modereturns the readers that contribute to anOperationalMode’s product (FMATCH_MODE_READERS, derived from the built-in readers’VariableSpec.modes).notebooks/03_readers_swath.ipynb, a demonstration of the swath readers.Fill and valid-range helpers in
libera_utils.io.netcdf, used by the CERES FLASHFlux, CLDPIX and VIIRS cloud-property readers:fill_in_dtypereturns a_FillValuein a given dtype, orNonewhen no value of that dtype can equal it;field_fill_valueandfield_valid_rangereturn a netCDF4 variable’s own_FillValueandvalid_range, raisingValueErrorwhen it has none;apply_fill_and_valid_rangereturns a raw, optionally masked array as float64 with masked entries, fill values (compared in the raw dtype) and values outside an inclusivevalid_range(either order) set to NaN.
Changed#
BREAKING: An
AncillaryReaderis constructed from a sequence of staged granule paths: a barestrorPathraisesTypeError, an empty sequenceValueError, and a path that is not an existing fileFileNotFoundErrornaming the reader and every such path. A single path was previously accepted, and a missing file failed only when it was read.BREAKING:
VariableSpectakes a requiredmodes, afrozenset[OperationalMode]naming every product that carries the variable, in place ofrequired_mode. An emptymodesraisesValueError, and one that is not a frozenset ofOperationalModemembers raisesTypeError. A standard-deviation companion fromwith_standard_deviation_companionscarries its parent’smodes.
Removed#
BREAKING:
OperationalMode.rank,VariableSpec.required_mode,VariableSpec.only_modesandspec_active_in_mode. A variable belongs to a product exactly whenmode in spec.modes.
5.17.0 - 2026-10-29#
Added#
write_libera_data_producttakesworker_compressed_variables, a tuple of Dask-backed data variables whose chunks are shuffled and deflated on thedask.distributedworkers instead of one at a time under the HDF5 write lock. The writing process stores the compressed chunks with h5pywrite_direct_chunk, and the file reads back identical to a normal write. It requires theh5netcdfengine and an activedistributed.Client, and each named variable’s Dask chunks must be whole multiples of its on-disk chunk shape, its filters deflate with or without shuffle, and its dtype integer or floating point; otherwiseValueErrorbefore any data is computed. A write that fails after creating the file deletes it. The default, an empty tuple, writes withto_netcdfas before.KernelManager.populate_cache()fills the on-disk NAIF and static kernel caches and leaves the SPICE kernel pool empty, for a process to call before starting workers that each load their own kernels.dask[distributed]is a declared dependency; worker-compressed writes import it andh5py.
Changed#
kernel_maker.create_kernels_from_manifestreads the input L1A product once and creates each requested SPICE kernel at the same time, each in its own forked process, so the Az/El and JPSS kernel steps take about as long as their slowest kernel rather than the sum. It requires Linux and raisesValueErroron any other platform before reading the L1A. Kernel filenames and the output manifest are unchanged. TheValueErrorraised after every kernel was attempted when any kernel fails now also gives each failed kernel’s reason: its process’s traceback, or its exit code when the process sent nothing; a kernel process killed by the operating system counts as that kernel failing. A kernel process flushes CloudWatch log handlers before it exits. Peak memory is about the sum of the concurrent kernels’ (an estimated 18 GB for a full-day Az/El run, from about 9 GB measured per kernel). A kernel identifier repeated inkernel_product_idsraisesValueError.BREAKING:
kernel_maker.create_kernels_from_manifestraises a shared-setup failure before any kernel is attempted, with its own exception type, instead of the aggregatedValueErrorlisting every kernel as failed: an unreadable L1A file raisesValueError, a kernel work directory path too long for curryer raisesRuntimeError(setLIBERA_TEMP_DIRto a shorter base path), and a failure populating the kernel cache raisesRuntimeError,FileNotFoundErroror arequests.RequestException.write_libera_data_productwrites an S3 product to a temporary local file, uploads it and removes the file, instead of keeping it in the cloudpathlib cache for a read that the manifest checksum needed. A full-day product no longer stays on local disk after upload, and the write no longer reads cloudpathlib’s private cache path or sends a request to sync the cache file’s modification time.
5.16.0 - 2026-10-29#
Changed#
BREAKING: Manifest files no longer record checksums, input or output.
Manifest.filesis a list of absolute path strings, written as"files": ["s3://...", ...]. A manifest in the previous{"filename": ..., "checksum": ...}record format raisesValidationErrorwhen read. A reader on libera_utils 5.15.0 or earlier treats each listed path as a file to checksum, so it downloads every listed file; move readers and writers of a manifest to this version together.Manifest.add_files()checks that each file exists and is a file, raisingIsADirectoryErrorfor a local directory, an S3 bucket root or an S3 prefix, but no longer reads it, so an S3 file is not downloaded and a file larger than local disk can be listed.BREAKING: The
Manifestconstructor no longer checks that a listed path exists; onlyadd_files()does.BREAKING: A path listed twice in a manifest is an error, where it was skipped with a warning:
Manifest.add_files()raisesValueErrorfor a path already listed or given twice, adding nothing, and the constructor andManifest.from_file()raiseValidationError.create_kernels_from_manifestno longer validates input checksums.
Removed#
BREAKING:
ManifestFileRecord,Manifest.validate_checksums(),Manifest.check_file_structure()andlibera_utils.io.manifest.calculate_checksum.
Fixed#
A file whose content matched another listed file’s was dropped, with a warning, by
Manifest.add_files(), the constructor andManifest.from_file(). Every file is now listed under its own name.
5.15.0 - 2026-10-28#
Added#
The
libera_utils.footprint_matching.readersplugin subsystem: anAncillaryReaderbase with auto-registration (StaticReaderfor a daily composite, map or time-stamped swath held in memory;SteppedGridReaderfor a time series of global grids served one step at a time), aReaderRegistry, and HDF4 and HDF-EOS5 I/O helpers. A reader serves the native samples of its source inside a footprint’s bounding box as aSampleSet(every pixel, footprint or grid point at its own position, no averaging, binning or interpolation), matched to the footprint time by itsTimeMatchpolicy:DAILY(IGBP, NISE, VIIRS BRDF and AOD) orNEAREST_STEP(ERA5). A reader whoseREADER_KEYis already registered to another class is rejected when its module is imported. A query raisesValueErrorwhen a point cloud holds a coordinate outside latitude [-90, 90] or longitude [-180, 180], when aNEAREST_PASSreader serves samples without times, or when the footprint time a time-selecting reader needs is NaT. This release adds the gridded readers:ERA5 single-level (
era5_surface, 7 variables) and 37-level pressure (era5_pressure) fields, serving the hourly step nearest each footprint’s time (none beyond 1 h), rejecting a file with no time axis, and declaringRESOLUTION_KM = 28.0, the 0.25 degree grid spacing.VIIRS Deep Blue AOD (
AERDB_D3_VIIRS_NOAA20) with a categoricalaerosol_type, applying each field’s own_FillValueandvalid_range(AOD 0 to 10).VIIRS BRDF (
VJ143C1), served at the 0.05 degree cell centers (the file’slat/lonare cell edges, checked against the grid’s HDF-EOS5StructMetadata.0corners), applying each field’s own_FillValue,valid_range,scale_factorandadd_offset.IGBP land cover (MCD12Q1), held as its uint8 sinusoidal tiles and geolocated per query, declaring
RESOLUTION_KM = 0.463, the 463.3 m sinusoidal cell.NSIDC NISE sea ice (AMSR-2 provenance), with the unused Extent codes (102, 105-251) and the coastline and suspect-ice quality codes (252, 253) missing (NaN) in every layer. Each hemisphere’s HDF-EOS
StructMetadata.0grid is checked against the 721 x 721 25 km EASE-Grid it is geolocated on (dimensions, corners andGCTP_LAMAZprojection), and a granule that differs raisesValueError.
notebooks/02_readers_gridded.ipynb, a demonstration of the gridded readers.pyhdf(IGBP and NISE HDF4 reads) andh5py(VIIRS BRDF HDF5 reads) are runtime dependencies.pyhdfneeds the HDF4 C library (libhdf4-dev) at build time when no prebuilt wheel is available, as on aarch64; the Docker image installs it.DataLevel.ANCfor ancillary inputs produced outside the SDC and ingested from their upstream providers. It archives tolibera-ancillary-data(LiberaDataBucketName.ANCILLARY_ARCHIVE_BUCKET);AUXkeeps the non-leveled products the SDC produces.Twelve
DataProductIdentifiermembers atDataLevel.ANC, one per external input product, with ProductIDs of the form<SOURCE>-<PRODUCT>, PRODUCT led by the observing platform where the upstream product is made per platform:anc_ceres_noaa20_cldpix(CERES-NOAA20-CLDPIX),anc_ceres_noaa20_flashflux(CERES-NOAA20-FLASHFLUX),anc_ceres_noaa20_ssf(CERES-NOAA20-SSF),anc_nsidc_land_ocean_mask(NSIDC-LAND-OCEAN-MASK),anc_nsidc_ice_snow(NSIDC-ICE-SNOW),anc_modis_igbp(MODIS-IGBP),anc_era5_pressure_level(ERA5-PRESSURE-LEVEL),anc_era5_single_level(ERA5-SINGLE-LEVEL),anc_viirs_noaa20_aod(VIIRS-NOAA20-AOD),anc_viirs_noaa20_brdf(VIIRS-NOAA20-BRDF),anc_viirs_noaa20_cloud_properties(VIIRS-NOAA20-CLOUD-PROPERTIES) andanc_viirs_noaa20_m4_sdr(VIIRS-NOAA20-M4-SDR). No processing step produces them. TheDataProductIdentifierdocstring states the ancillary ProductID pattern.
Changed#
The
h5netcdfdependency is declared with itsh5pyextra (h5netcdf[h5py]), sinceh5netcdfno longer installsh5pyby default.The
devdependency group depends onpyjpeglsinstead ofpillow-jplsfor JPEG-LS decoding in the WFOV integration test.pillow-jpls1.3.2, its latest release, publishes wheels only through cp312, so Python 3.13 and 3.14 installs build it from source, which fails on gcc 14 hosts;pyjpeglsbinds the same CharLS codec.
5.14.0 - 2026-10-27#
Added#
create_product_dataset(array input) andwrite_libera_data_product(Dataset input) set the ACDDdate_createdattribute to the creation time, ISO 8601 UTC at microsecond precision. A value already present is overwritten with a warning.libera_utils.io.product_definition.DATE_CREATED_ATTRIBUTEandALGORITHM_VERSION_ATTRIBUTE, the names of those product attributes.
Changed#
BREAKING: The revision in Libera data product filenames is the UTC creation time at microsecond precision,
R%Y%m%dT%H%M%S%f, e.g.LIBERA_L1B_RAD-4CH_V1-2-3_20270102T112233_20270102T122233_R20270103T080910123456.nc, so files produced within the same second no longer clobber each other in the dropbox bucket. Names with the earlier second-resolutionR%y%j%H%M%Srevision no longer parse:LiberaDataProductFilenameraisesValueErrorfor them. The UMM-G GranuleUR, which is the filename stem, and the.cmr.jsonmetadata name follow the new form.BREAKING:
libera_utils.io.filenaming.REVISION_TS_FORMATis nowR%Y%m%dT%H%M%S%f.BREAKING:
date_createdis a required standard product attribute, andgenerate_data_product_filenametakes the filename revision from it. A strict conformance check on a Dataset built neither bycreate_product_datasetnor bywrite_libera_data_productreports it missing.BREAKING: A product definition YAML that gives
date_createda value raisesValueError.BREAKING: UMM-G generation (
UMMGranule.from_dataset,write_data_product_metadata_ummg) raisesValueErrorwhen a Dataset has nodate_created, and emits it as theDataGranuleProductionDateTimeand theCreateprovider date. It previously read aProductionDateTimeattribute and, without one, warned and used the current time.UMM-G
TemporalExtentno longer falls back to aProductionDateTimeattribute; without theRange*attributes it is omitted.from_filename_partstakesrevision: datetime | None; the defaultNoneuses the current UTC time at the call.L1A packet parsing and the NOM-HK trimmer no longer set
date_created; the trimmer drops the parent product’s value so the writer sets a fresh one.kernel_makerno longer builds a revision for SPICE kernel filenames; the default applies.
Removed#
BREAKING:
.h5is no longer an accepted data product filename extension.
Fixed#
The
revisiondefault infrom_filename_partswasdatetime.now(tz=UTC)evaluated once at import, so every product written by one process shared a single revision. Revisions are now generated per call.LiberaDataProductFilenameraisesValueErrorat construction, or when its path is assigned, when a date or time in the name does not exist, such as month 13. The error previously surfaced at the firstfilename_partsaccess.A stray debug
printinUMMGDatasetTransformer.extract_temporal_extentis removed.
5.13.0 - 2026-10-26#
Added#
The API docs now include
libera_utils.l1a. The package had no__init__.py, so Sphinx skipped it.Link the UMM-G granule page from the user docs index.
Manifest.write()takes a keyword-onlyoverwrite=False, andout_pathmay be a directory, an S3 prefix, or a full manifest file path.Manifest.output_manifest_from_input_manifest()takes an optionalconfiguration=dict, and raisesManifestErrorfor an input manifest with no ULID.
Changed#
The docs build fails on any warning:
doc/Makefilerunssphinx-build -W --keep-going, and.readthedocs.yamlsetsfail_on_warning.ReadTheDocs builds the docs on Python 3.13 (Ubuntu 24.04), and the
docgendependency group requires Sphinx 9.1 or newer on Python 3.12 or newer.BREAKING:
Manifest.ulid_codeis a read-only property derived fromfilename. Assigning to it raises; assignfilenameinstead.ulid_code=is still accepted by the constructor.BREAKING:
Manifest.write()setsfilenameto the path it wrote.BREAKING:
Manifest.write()no longer takes afilenameargument; pass the full manifest file path asout_pathinstead. The old positional formwrite(out_path, filename)raisesTypeError.BREAKING:
Manifest.write()raisesFileExistsErrorfor an existing target unlessoverwrite=Trueis passed, andManifestErrorfor a target whose INPUT/OUTPUT label disagrees withmanifest_typeor that would change the manifest’s ULID, andManifestErrorfor anout_paththat has an extension but is not a valid manifest filename.BREAKING:
Manifest.validate_checksums()raisesManifestErrorinstead ofValueError. A file that cannot be found fails validation.BREAKING:
Manifest.from_file()raisesManifestErrorfor a path that is not a valid manifest filename, or when the filename orulid_codestored in the file disagrees with the ULID in the path.BREAKING:
Manifest.filenameis validated on assignment, and afilenameormanifest_typewhose INPUT/OUTPUT label disagrees with the other raisesValidationError.BREAKING: A file record with a null checksum, such as one read from a manifest file, raises
ValidationErrorinstead of being recomputed.BREAKING:
MANIFEST_FILE_REGEXmust match the whole basename, so names such asLIBERA_INPUT_MANIFEST_<ULID>.json.bakare no longer valid manifest filenames.
Removed#
numpydocfrom thedocgendependency group.sphinx.ext.napoleonparses the numpy style docstrings.BREAKING:
libera_utils.io.manifest.get_ulid_code. UseManifest.ulid_code, orManifestFilename(path).filename_parts.ulid_code.
Fixed#
Fix the docstring, markdown and Sphinx config defects behind the docs build’s 9841 warnings, including cross-references to private or renamed targets (
create_l1a_datasetis nowparse_packets_to_l1a_dataset).Fix the docs build warnings and errors in the
footprint_matchingandscene_identificationdocstrings and thewrite_libera_data_productandLiberaVariableDefinitionRaisessections, which the build now treats as failures.A second
Manifest.write()to a different directory no longer writes to the first location.A manifest with no filename serializes
filenameasnull, not the string"None".
5.12.0 - 2026-10-21#
Added#
The
libera_utils.footprint_matchingpackage, the foundation of the FMATCH (footprint matching) algorithm:footprint_matching.typesholds the shared data classes and enumerations:OperationalMode(the five FMATCH modes, ordered by latency),FmatchCoverageFlag,BoundingBox,RadiometerFootprintandVariableSpec, withspec_active_in_modeandwith_standard_deviation_companions.SampleSet,GridSamples,PointSamplesandTimeMatchdescribe a reader’s native samples inside a footprint’s box (every pixel, footprint or grid point at its own position, no averaging, binning or interpolation), the two native forms a reader loads, and the policy by which a reader selects samples in time (DAILY,NEAREST_STEP,NEAREST_PASS). ARadiometerFootprintcarries its observationtime, which the readers match ancillary samples to, and itsbboxisNoneexactly when it is an off-limb view.footprint_matching.psfevaluates the point spread function (psf_weight) and its 95%-energy angular extent (psf_95_energy_extent), plusstatic_fov_extentandconservative_along_scan_extent. The Libera PSF has not been delivered, so the module uses the heritage CERES analytic PSF (CERES ATBD 4.4) as a stand-in; its CERES constants are isolated in one block so that swapping in the Libera PSF changes only that file.footprint_matching.geometryturns L1B footprint geolocation and viewing geometry into the latitude/longitude bounding box of the ground the radiometer senses (compute_footprint_bounding_box, andcompute_footprint_bounding_boxesfor a batch). It ray-traces the PSF extent onto the WGS84 ellipsoid using curryer’s spatial tools, rounds the box outward so it always encloses the footprint, and handles dateline crossings and footprints that enclose a pole. A boresight that misses the Earth raisesOffLimbError, a footprint whose box corners run off the limb raisesPartialFootprintError, and unusable geolocation or ephemeris raisesMissingGeolocationError. The module also providescompute_viewing_frames/ViewingFrame,surface_cell_ecef_kmandproject_to_angular, which project ground grid cells into the radiometer’s angular frame for PSF weighting.
notebooks/01_geometry_psf.ipynb, a demonstration of the PSF model and footprint geometry.pyprojis a runtime dependency, used for WGS84 geodesic distances.
5.11.5 - 2026-10-14#
Added#
SCENE-ID-CAM (radiometer timescale, written on
RADIOMETER_TIME) and SCENE-ID-CAM-CAMTIME (camera timescale) algorithm runners, each with a product definition and aFootprintData.from_fmatch_cam/from_fmatch_cam_camtimereader. Both run from onelibera-utils scene-id <manifest>command and onelibera-utils-scene-idDocker target; theLIBERA_SCENE_ID_VARIANTenvironment variable (camorcam-camtime) selects the variant, and the runner raisesValueErrorwhen it is unset or unknown. The variants share their manifest/dropbox plumbing throughscene_id_algorithm.run_algorithm, parameterized by a per-variantSceneIdRunnerConfig.SCENE-ID-CAM-CAMTIME stores one image subsection per record on the 2-D
(CAMERA_TIME, PSEUDOFOOTPRINT)grid and passes the FMATCH footprint identifiers (camera time, pixel-index ranges, PSF bounding box, boresight geolocation) straight through.CAMERA_TIMEis a 1-D dimension coordinate (one entry per image) andPSEUDOFOOTPRINTa 0-based image-subsection index (an image is segmented into many, possibly overlapping, subsections that share its time).The
unfilteringscene definition, and thestandard_scene_definitions/default_scene_definitionshelpers.Viewing geometry in scene identification:
solar_zenith_angle,viewing_zenith_angleandrelative_azimuth_angleare classification variables withscene_bin_{type}_{angle}_min/maxbounds. The bins currently span the full physical range as a placeholder, so scene IDs are unchanged.FootprintData.to_time_product()and itsto_radiometer_time_product()wrapper, which promote the observation-time variable to a coordinate and finalize the dataset on its per-footprint axis for writing.
Changed#
BREAKING: Scene identification moved to the
libera_utils.scene_identificationpackage; importscene_idandscene_definitionsfrom there.Default scene classifications are now TRMM, ERBE and unfiltering, and
identify_scenes()with no arguments uses them.
5.11.4 - 2026-10-14#
Changed#
write_libera_data_productalways hands the NetCDF engine a filesystem path, never an open file object. An S3 destination is written to the path’s cloudpathlib cache file and uploaded from there, replacing any existing object at the same key. The cache keeps the product afterwards, as it did when the writer went throughCloudPath.open, so reading it back in the same process (the output manifest’s checksum, for one) does not download it again. Bothh5netcdfandnetcdf4now write to S3;netcdf4previously wrote into the cloudpathlib cache and uploaded nothing.encoding.chunksizesin a product definition is stored as a tuple, so a YAML list is accepted by theh5netcdfengine as well asnetcdf4. Achunksizesthat is not a list of positive integers with one entry per dimension now raisesValidationErrorwhen the definition loads rather than when the product is written.
Fixed#
calculate_checksum, whichManifest.add_filescalls for every output file, reads in 16 MiB blocks instead of reading the whole file into memory, so a product larger than the host’s memory can be added to a manifest.Writing a product under the distributed Dask scheduler with the
h5netcdfengine no longer fails withTypeError: cannot pickle '_io.BufferedRandom' object.
5.11.3 - 2026-10-09#
Added#
All
Filenameclasses inlibera_utils.io.filenamingare hashable.hash()is the hash of the path as given, so filenames can be dictionary keys and set members. Reassigningpathchanges the hash, so do not mutate a filename that is already in a set or a dict.
Changed#
BREAKING:
AbstractValidFilename.__eq__compares paths only and returnsNotImplementedfor non-filename operands, sofilename == "some string"isFalseinstead of raisingAttributeError. Two filenames with the same basename in different directories or buckets are not equal. Previously the parsed filename parts were compared as well, which was redundant because they are derived from the path.
Fixed#
Comparing two
Manifestmodels when one hasfilename=Noneno longer raisesAttributeErrorfrom the filename comparison.
5.11.2 - 2026-10-09#
Changed#
Fix the intermittent
PytestUnraisableExceptionWarningCI failures. Leaked handles (see Fixed), test fixtures that detached root logging handlers without closing them, and anh5py.Filehanded a file object it does not own producedResourceWarnings during garbage collection. pytest drains those from one global queue at each test’s setup, call and teardown, so the failure landed on whichever unrelated test was running at the time. Tests that assert no warnings now use the sharedstrict_warningsmark, which keeps warnings-as-errors but declines to own unraisable exceptions leaked elsewhere in the suite.calculate_checksumno longer decompresses*.gzinput before hashing. A manifest checksum has to cover the bytes as stored so it can be compared against the file a data provider delivered; the previous value hashed content that exists nowhere on disk and would change if the file were recompressed at a different level. Any checksum an earlier version recorded for a*.gzfile must be regenerated.
Fixed#
smart_opennow closes the underlying file object when theGzipFileit returns is closed.GzipFileonly closes a file it opened itself, so reading any*.gzthroughsmart_openleaked an open file handle until the garbage collector finalized it. This affected library callers (calculate_checksum,metadata_writer, SPICE kernel downloads), not just tests.smart_openpasses the requested mode toGzipFilewhen opening a*.gzfile. Writing one no longer emits theFutureWarningGzipFileraises when it infers a write mode from the file object, and'wb+'now opens for writing instead of read-only.smart_openrejects a text mode for a*.gzpath before opening the file, so theOSErrorno longer leaves an open file handle behind.Writing a
*.gzfile throughsmart_openwith anS3PathorAnyPaththat already had a local cached copy (a second write, or a write after a read through the same object) no longer leaves cloudpathlib’s cache marked dirty, which made the next read raiseOverwriteDirtyFileError.
5.11.1 - 2026-09-25#
Added#
Add docker-compose entries for spice kernel making.
5.11.0 - 2026-09-09#
Added#
Add
libera_utils.l1a.data_time_extractorsfor per-file science data-time spans without full L1A assembly, covering everyDATA_TIME_INDEXED_APIDSmember (WFOV SOP FSW image times; RAD/CAL/AXIS sample epoch+period or per-sample times).extract_data_time_rangereturnsNonefor a WFOV file holding noSOPpacket, which is expected when a large image’s mem-dump is chunked across files or downlink passes.jpss_sc_pos(APID 11) is data-time indexed. Its span runs from the earliest to the latest time reported by either of the packet’s independently timestamped sample groups (ADGPSephemeris,ADCFAattitude), so it records the extent of data present rather than the range both groups cover. A consumer needing ephemeris and attitude together must intersect the two sample-time ranges itself.Add demuxed ground-test CCSDS support:
LiberaGroundCcsdsFilenameandDataProductIdentifier.l0_ground_ccsdsfor theLIBERA_SDC_<apid>_ccsds_<yyyy>_<doy>_<hh>_<mm>_<ss>naming (L0 archive prefixGroundCCSDS/<apid>/<yyyy>/<mm>/<dd>/). Each file holds one APID, read from the basename, with no record header before the CCSDS primary header. The name’s time fields are the bin start the file was cut from, not the span of the packets inside it.Add
libera_utils.l1a.ground_ccsds.scan_ground_ccsds_file, returning one file’s packet and science data time spans for File Metadata ingest. It returnsNone, logging the reason, when no packet-time span is possible at all, and returns the packet span withdegraded_reasonset when only the science data times are unavailable.Manual ingest (
s3-utils put/manual_ingest_data_products) accepts demuxed ground CCSDS files.verify_ingestionchecks archival and the base File Metadata row for these; the searchable row is not verified.extract_data_time_rangeandparse_packets_to_l1a_datasetaccept an optionalskip_header_bytes=override, falling back toSKIP_PACKET_HEADER_BYTESwhen omitted, so one process can handle both flight and ground files without mutating shared config.
Changed#
Extracted time spans drop
NaTand timestamps outsideMIN_VALID_TELEMETRY_TIME-MAX_VALID_TELEMETRY_TIME(defaults2020-01-01and2045-01-01, both config-overridable). A ground capture can carry a packet whose onboard clock has not yet been time-synced, decoding to just afterCCSDS_EPOCH(1958-01-01), which would otherwise stretch a span across ~68 years.libera_utils.l1a.packets.expand_sample_timesis now public (was_expand_sample_times), andparse_packets_to_l1a_datasettakesground_dataandverboseas keyword-only arguments.
Fixed#
verify_ingestiontreats its File Metadata record count as a floor rather than an exact match. A file’s data can span more than one day, giving it a searchable record per applicable date, so an L0 PDS with a multi-day span (as ajpss_sc_posfile can have) no longer polls to timeout on a count of 3 against an expected 2.
5.10.11 - 2026-09-08#
Changed#
The production Docker image no longer sets
LEAPSECOND_FILE_ENV. It pointed at a directory containing nonaif*.tls, so curryer’s lookup found nothing there;KernelManager.load_naif_kernelssets the variable at runtime to the directory of the LSK it furnishes.
5.10.10 - 2026-09-08#
Added#
KernelManager.ensure_kernel_coverageverifies that the furnished kernels cover every requested target across a time window, raising before a gap surfaces as an obscure SPICE failure deep in a later computation or as silently wrong numbers. Opt-in: call it after furnishing and before the first computation that reads the kernels.
Changed#
Require
lasp-curryer >= 0.5.2, forcurryer.kernels.coverageand the packaged leapsecond kernel that makesLEAPSECOND_FILE_ENVan override rather than a requirement.
5.10.9 - 2026-09-08#
Fixed#
WFOV_FSW_HEADER_AZIMUTH_ANGLEin the ICIE WFOV SCI L1A product is documented asdegrees, notradiansas it was incorrectly listed before.Whitespace clean up in trim family csv
Test L1A product definition units for Axis-Sample updated to match the actual products
5.10.8 - 2026-09-03#
Changed#
BREAKING:
ecr-uploadnow requires a local image tag instead of defaulting tolatest. Algorithm images are built under an explicitdocker build -t <name>:<version>, so a locallatestusually does not exist and the old default failed later with an opaqueLocal image not found: <name>:latest— after the L2 Team Role assumption round-trip. Omitting the tag now fails immediately with an explanatory error. Passmy-image:latestexplicitly if that really is the image you want. The--ecr-tagsdefault oflatestfor the remote ECR tag is unchanged.The
ecr-upload --image-tagoption is deprecated in favor ofimage-name:image-tag. It still works and logs a warning pointing at the new syntax; supplying both is an error unless they agree.push_image_to_ecracceptsimage_tag=Noneand takes the tag fromimage_nameinstead, which must then carry one.
Fixed#
libera-utils ecr-uploadnow accepts the local image in the usual Docker form,image-name:image-tag(e.g.ecr-upload l2-comp-flux my-image:1.2.3). Previously this positional argument had to be a bare image name and the tag had to be passed separately, so aname:tagvalue silently produced aLocal image not found: name:tag:latesterror. Colons in a registry host:port (localhost:5000/my-image) are not treated as tags.
5.10.7 - 2026-09-03#
Changed#
BREAKING: The four per-channel radiometer frames (
LIBERA_SW_RAD_COORD,LIBERA_SSW_RAD_COORD,LIBERA_LW_RAD_COORD,LIBERA_TOT_RAD_COORD) are replaced by a single co-alignedLIBERA_RAD_COORDframe and oneLIBERA_RADinstrument/FOV in the frame and instrument kernels for both JPSS-4 and NOAA-20. The four channels share a boresight, so the per-channel definitions carried identical geometry. Downstream callers must transform toLIBERA_RAD_COORDinstead of a per-channel frame.BREAKING: The four per-channel fixed-offset SPK configs are replaced by one
libera_rad.fixed_offset.spk.jsonper mission, soLIBERA_KERNEL_STATIC_SPK_CONFIGSindata/config.jsondrops from 8 static SPK configs to 5.
5.10.6 - 2026-09-02#
Added#
Add four radiometer cal-combine
ProcessingStepIdentifierfamily steps (cal-gain-family,cal-swc-family,cal-lwc-family,cal-solar-family), each declaring every CAL product its family’s ObsIDs can yield. Lunar and camera family steps are deferred (TODO[LIBSDC-811]).ProcessingStepIdentifier.ecr_namesupports an optional shared ECR name so all radiometer cal steps resolve tocal-rad-docker-repo. Each job dispatches on the ObsID carried by the TRIMMED input it is handed, so no per-job step identity has to be configured separately.Add
libera_utils.obsids.TRIM_FAMILIESandget_family_specs()so a cal step enumerates the ObsIDs — and CAL products — of its trimmed input family without hand-maintaining the mapping.Add
data/trim_family_inputs.csv(FAMILY_INPUTS/get_family_inputs()) recording the L1A products each family’s cal step consumes besides its own TRIMMED product; the two together are theinput-productsset a libera_cdkcal-*-familynode should declare. The two catalogs are cross-checked at import.Add
libera_utils.l1a.nom_hk_trimto detect contiguous ObsID runs in decoded NOM-HK and write one trimmed file per run, stamped with that run’s family ProductID. Several files per day normally share a family ProductID and are told apart by their filename time ranges.Add
libera_utils.l1a.packet_slicingas the supported way to subset a decoded L1A product.select_packetsandslice_l1a_dataset_to_time_windowtreatPACKETas the driver — whole packets survive or none of them does — and renumber{sample_group}_packet_indexagainst the surviving packet axis, so sample-to-packet traceability holds across a subset. Slicing either axis directly does not.
Changed#
BREAKING: L1A trimmed NOM-HK products are now one per calibration dependency family rather than one per ObsID: 10
NOM-HK-<FAMILY>-FAMILY-TRIMMEDmembers replace the 40NOM-HK-<EVENT>-TRIMMEDmembers added in 5.10.3. Each ObsID keeps its own CAL product, and the ObsID stays readable from theICIE__SW_OBSID_*variable inside each trimmed file. A family never spans both ObsID fields, so RAD and WFOV VIIRS lunar (513/514) are registered as separate families with distinct ProductIDs.BREAKING: Registry validation now rejects a CAL product claimed by more than one ObsID, and a TRIMMED family registered on both the RAD and WFOV ObsID fields. The 5.10.3 check that a TRIMMED product is claimed by only one ObsID is removed, since sharing one is what defines a family.
Add the ObsID Registry user-docs page, covering the
(source, obsid)keying, how downstream repos dispatch cal-combine steps from the registry, and the decision that NOM-HK is the only product the preprocessor trims — cal containers subset the full daily L1A inputs themselves.
Fixed#
parse_packets_to_l1a_datasetnow sorts the packet axis into its final order before samples are expanded, instead of sorting at the end. The trailing sort permutedPACKETwithout permuting the sample axes, silently invalidating the{sample_group}_packet_indexvariables it had already written. Output ordering is unchanged; the indices are now correct.
5.10.5 - 2026-08-31#
Added#
WFOV SCI (APID 1040) L1A processing stitches complete SOP→EOP images onto
CAMERA_TIME, stores compressed JPEG-LS payloads inWFOV_COMPRESSED_IMAGE(uint8/BLOB_BYTE+WFOV_COMPRESSED_IMAGE_LENGTH), and traces packets back to their image withPACKET_IMAGE_ID.Decode the 176-byte WFOV header (FSW + FPGA) onto
CAMERA_TIMEasWFOV_FSW_HEADER_*,WFOV_IMAGE_HEADER_*,WFOV_IMAGE_FOOTER_*, andWFOV_FPGA_STATUS_*; trailing 8-byte NAND footer is validated against a known magic pattern (FooterMismatchCount).Add file-level quality attrs
PacketCountNotUsedInImages,ErrorFlaggedImageCount,FooterMismatchCount,HeaderParseErrorCount, plusFirstImageIncomplete/LastImageIncompleteflagging expected truncation at a packet window’s leading/trailing edge (PacketCountNotUsedInImagescounts only genuine mid-stream anomalies).WFOV SCI processing now raises
ValueErrorwhen no complete image can be stitched. The raise happens beforeICIE__WFOV_DATAis dropped, so the raw packets survive for diagnosis, and the message carries all stitch counters.
Changed#
BREAKING:
CAMERA_TIMEnow contains one row per complete stitched image only (not every qualifying SOP). Removed redundantWFOV_IMAGE_COMPLETE.BREAKING:
ICIE__WFOV_DATAis dropped from the WFOV L1A product entirely — every packet’s raw payload is by then either duplicated inWFOV_COMPRESSED_IMAGEor unusable from this granule alone.PACKET_IMAGE_IDis the only remaining per-packet trace-back.
Fixed#
A structurally complete image whose 176-byte header cannot be decoded is now discarded and counted in
HeaderParseErrorCount, rather than emitted with aNaTCAMERA_TIMEthat crashed filename generation.CAMERA_TIMEis therefore always free ofNaT.SINGLE-flagged WFOV packets (whole image in one packet) are unsupported and were silently misattributed to window-edge truncation. They now log a warning and count towardPacketCountNotUsedInImages; aSINGLEmid-collection abandons the image in progress.
5.10.4 - 2026-08-25#
Added#
Register ADM-specific standard dimensions in
libera_dimensions.yml(viewing-geometry, surface-type, and cloud/wind property bins for ERBE-like and TRMM-like ADM products)
5.10.3 - 2026-08-24#
Added#
Replace the eight alpha calibration
DataProductIdentifiermembers (cal_solar_face{1,2,3}_combined,cal_lw_temp{1,2,3}_combined,cal_gain_combined,cal_sw_combined/SOLAR-FACE1-COMBINED,LW-TEMP1-COMBINED,GAIN-COMBINED,SW-COMBINED, …) with one CAL product per calibration ObsID (cal_gain/GAIN,cal_swc_365nm/SWC-365NM,cal_lwc_310k/LWC-310K,cal_solar_ssw_pri/SOLAR-SSW-PRI, the lunar/VIIRS-lunar and camera cal products, etc.). Downstream consumers must move to the new members; no aliases are kept.Add ObsID-trimmed NOM-HK
DataProductIdentifiers (NOM-HK-<EVENT>-TRIMMED), one per calibration ObsID, produced by L1A Step 1 preprocessing and consumed by the per-ObsID cal-combine steps.Add
libera_utils.obsidsand thelibera_utils/data/obsid_registry.csvcatalog: the ICIE software ObsID registry mapping(NomHkObsidSource, obsid)to its TRIMMED and CAL ProductIDs, withget_obsid_specanditer_trim_eligiblefor downstream dispatch. The CSV is validated at import (unknown member names, wrong data level, malformed rows,kind/sourcemismatches, duplicate ObsIDs, and TRIMMED products claimed by more than one ObsID all raiseValueError).
5.10.2 - 2026-07-31#
Fixed#
enforce_data_array_conformanceno longer emits aUserWarning(or log line) for every encoding key present on a DataArray but absent from the product definition. Extra keys (typically xarray/NetCDF runtime metadata such assource,original_shape,preferred_chunks, compression stubs) are dropped silently; conflicting values for keys that are defined still warn. This removes thousands of spurious warnings when packaging products built from opened L1A NetCDF inputs.
5.10.1 - 2026-07-31#
Added#
ecr-uploadnow registers the algorithm version(s) it uploads. After pushing, it emits aNewAlgorithmImageevent (one per non-latestECR tag) to the SDC event bus so the Registrar creates the corresponding versioned AWS Batch job definition. Adds--verify(blocks until both the ECR image and the Batch job definition are confirmed present, up to--timeoutseconds).Add the
register-algorithm-imageCLI for emitting aNewAlgorithmImageevent for an image already in ECR (--image-digest/--verify/--timeout). The newaws/algorithm_registration.pymodule providesput_new_algorithm_image_eventandverify_algorithm_registration(checks the ECR image exists and an ACTIVE Batch job definition references it).Quieter
ecr-uploadpush logging — per-layer progress ticks are suppressed, a per-tag summary and the resulting image digest are logged at INFO, and-v/--verboseenables full DEBUG push detail.push_image_to_ecrnow returns a{tag: digest}mapping.
Changed#
AWS region for the ECR/EventBridge/Batch operations is now taken from the session’s AWS configuration (profile /
AWS_REGION) instead of a hard-codedus-west-2; a clear error is raised if no region is configured, and--verifywarns when the image URI’s region differs from the session region.Document
register-algorithm-imageand the newecr-uploadflags in the usage docs.
5.10.0 - 2026-07-24#
Added#
Kernel generation applies the measured Libera opto-mechanical frame misalignments from OAV3 ground testing. The Az/El mechanism CKs now encode rotation about the measured axes of rotation, and the radiometer frames carry the measured line-of-sight boresight, so the generated kernels reflect the true instrument geometry instead of nominal placeholders. The three source unit vectors (
LIBERA_AZ_AOR_IN_STAND,LIBERA_EL_AOR_IN_STAND,LIBERA_EL0_Z_IN_STAND) are stored as keywords in the frame kernel; the Az/El CK configs switch from Euler angles to quaternions, andkernel_maker.add_mechanism_ck_quaternionsbuilds the per-sample rotations about the measured axes (applied after the encoder correction).
5.9.3 - 2026-07-24#
Added#
Az/El mechanism CKs now apply deterministic sinusoidal encoder-angle corrections during kernel generation, so the stored quaternions reflect the true mechanism rotation rather than the raw encoder readout. Corrections are applied to the filtered encoder angles in
create_kernel_from_l1a(L1A telemetry is left unchanged), with matching forward/inverse helpers (correct_azimuth/correct_elevation,uncorrect_azimuth/uncorrect_elevation, and DataFrame-levelapply_encoder_corrections/reverse_encoder_corrections)
Fixed#
Remove np.NaN as curryer no longer pins Numpy < 2
5.9.2 - 2026-07-17#
Added#
Expand the RBSP+VIIRS imager
DataProductIdentifierset (l2_unf_rad_imager,l2_nb_bb_imager_camtime,l2_toa_flux_imager, and matching AUX scene-ID/FMATCH/ADM members)
Fixed#
rename camera/imager DPI and
ProcessingStepIdentifiermembers for consistency with product strings:l2_unf_cam→l2_unf_rad_cam/l2-unf-rad-cam,l2_cf_rad_time/l2_cf_rad→l2_cf_cam/l2-cf-cam,l2_cf_cam_time/l2_cf_cam→l2_cf_cam_camtime/l2-cf-cam-camtime,l2_nb_bb_cam_time/NB-BB-CAM-TIME→l2_nb_bb_cam_camtime/NB-BB-CAM-CAMTIME/l2-nb-bb-cam-camtime,l2_unf_imager→l2_unf_rad_imager/l2-unf-rad-imager,l2_nb_bb_imager→l2_nb_bb_imager_camtime/l2-nb-bb-imager-camtime
5.9.1 - 2026-07-16#
Added#
Scene Identification now reports the property bin min/max bounds alongside the scene ID. For each footprint and classification variable,
identify_scenesaddsscene_bin_{type}_{variable}_min/_maxvariables giving the bounds of the matched scene’s bin (unbounded sides and unmatched footprints reported asNaN). Controlled by the newreport_bin_boundsflag (defaultTrue) onFootprintData.identify_scenesandSceneDefinition.identify_and_update; existingscene_id_{type}columns are unchanged.Add
SceneDefinition.get_bin_bounds_for_scene_idandScene.get_bin_boundsto look up the property bin bounds for a given scene ID.
5.9.0 - 2026-07-14#
Changed#
BREAKING: Rename
DataLevel.ANCtoDataLevel.AUXand split ancillary archive routing into separate AUX and CAL archive buckets.BREAKING: Replace provisional L2/ANC
DataProductIdentifierandProcessingStepIdentifiervalues with the camera cloud-fraction and RBSP+VIIRS imager product sets for July delivery (renamed product strings, new AUX/L2 members, and updated L2 team IAM role mappings).Update user docs for the new product/level identifiers.
5.8.6 - 2026-07-06#
Added#
Manual processing now runs through the SDC event bus instead of triggering Step Functions directly.
step-function-triggeremits aManualProcessingevent for a single step, and a newmanual-processingCLI submits arbitrary custom DAGs (or the default DAG) across one or more dates, with an optional--verifythat polls the Coordination Table.ecr-uploadassumes the algorithm’s per-team L2 Team Role for L2/ADM images so non-admin L2 developers can push to their ECR repos; non-L2 steps use the default/--profilesession unchanged.
5.8.5 - 2026-06-29#
Added#
s3-utils putnow performs manual SDC data ingest instead of a direct archive upload. It accepts multiple file paths, stages each file to the SDC Ingest Dropbox bucket, and emits a singleNewFilesAvailableevent to the SDC event bus; the SDC Data Ingester then handles archiving and metadata/data-availability records. Accepts L0 and data product filenames (manifests rejected).Add
get_l2_team_role_session(inaws/utils.py) which builds a boto session from a profile/AWS_PROFILEand assumes an L2 team role (defaultL2Developer/LiberaUtils), raisingValueErrorif the base profile is not permitted to assume it.s3_put_cli_handlernow uses it as its session source.Add
--verify/--timeouttos3-utils put. With--verify, the CLI blocks after emitting the ingest event and polls (read-only) until each file is confirmed in its archive bucket and recorded in the File Metadata table (and, for non-L0 products, the Data Availability table), logging a per-file summary and raising on timeout (default 5 minutes). Addsfind_dynamodb_table_in_account_by_partial_nametoaws/utils.py.
Changed#
Remove the unused
libera_utils/db/dynamodb_utils.pymodule (stale write-side helpers and an unusedget_dynamodb_table); the ingest-verification DynamoDB reads are inlined inverify_ingestion.
5.8.4 - 2026-06-08#
Added#
KernelManager.load_naif_kernelsalso loads an extended Earth PCK (earth_*_predict.bpcviaNAIF_EARTH_EXTENDED_PCK_REGEX) so ITRF93 orientation is available beyond the short high-precisionearth_000101_*.bpcwindow (required for JPSS geolocation and future-epoch processing).
5.8.3 - 2026-06-04#
Note: NOM-HK L1A changes the shape and dtype of five variables (replacing per-index fields with indexed arrays). Treat this as breaking for external consumers of those variables; a patch bump was chosen during early development. Changed variables: ICIE__SW_FP_WP_ST_WP, ICIE__SW_SEQ_EXEC_BUF_OP,
ICIE__SW_SEQ_EXEC_POS_OP, ICIE__SW_SEQ_ST_OP, ICIE__SW_SEQ_STOP_CD_OP
Added#
Add NOM-HK array groups to stack waypoint status and sequence execution fields into indexed arrays (
ARRAY_128andARRAY_8)
Fixed#
Normalize Unicode packet string fields during aggregation to ensure deterministic byte sizing
5.8.2 - 2026-06-03#
Added#
Faster ICIE WFOV SCI (APID 1040) packet-to-L1A by refreshing
icie_xtce_tlm.xmland combining the 972-field WFOVaggregation_groupsblock fromicie_wfov_sciinl1a_processing_configs.yml.
5.8.1 - 2026-06-03#
Added#
Added alpha calibration combination constants for data products
5.8.0 - 2026-05-15#
Added#
KernelFileCachecan materialize kernels from local filesystem paths (Pathor non-HTTPstr), with documented resolution rules for relative paths and stable cache freshness after copy.KernelManagercaches Libera static kernels under the versioned user cache viaKernelFileCacheafter building them under the existing short temporary directory. When every required artifact is already cached (and withincache_timeout_days), static kernel creation is skipped and kernels are furnished from the cache, consistent with NAIF generic kernels.
Changed#
BREAKING:
KernelManager.load_libera_dynamic_kernelsaccepts only aSequence[str | pathlib.Path | S3Path](manifest-ordered path list). Directory expansion, scalarPath, and scalar localstrare not accepted (use[path]for a single file). Materialization is inlined with a single normalization pass per entry;_materialize_dynamic_kernel_paths/_is_dynamic_sources_sequenceare removed from the dispatch path.BREAKING: Renamed
load_libera_dynamic_kernelsargumentdynamic_kernel_directorytodynamic_kernel_sources(broader semantics; no compatibility alias).Improve
KernelFileCachedocstrings andspice_utilstype annotations (includingensure_spiceoverloads and tighter return types on kernel helpers).
5.7.1 - 2026-05-13#
Added#
Added
ground dataoption for deduplication to allow ground data processing to continue
Fixed#
Fix packet parsing to l1a of datasets with deduplicated timestamps for final data assembly
With
ground_data=True, non-identical duplicate coordinates no longer raise whenverbose=False; warnings only ifverbose=True.Duplicate-coordinate diagnostics no longer cite the source sequence counter when that coordinate is absent from the dataset slice (e.g. inside sample groups).
5.7.0 - 2026-05-01#
Added#
Added metadata writer to make UMM-G metadata files from NetCDF-4 data files
5.6.0 - 2026-03-19#
Added#
Add dimension validation to data product definition
Support safer enforcement of product definition specs on data products
Changed#
Improve logging and warning visibility in data product writer
5.5.6 - 2026-03-19#
Added#
Adding support for APIDs 1000, 1002, 1035, 1043, and 1044 to support calibration and status telemetry packets
5.5.5 - 2026-03-02#
Changed#
Update the ICIE TLM XTCE to latest version
5.5.4 - 2026-03-02#
Changed#
Port Make_Kernel and SPICE-based time conversions to spice_utils to avoid circular imports
Fixed#
Implement KernelManager for Make_Kernel, furnishing necessary kernels
5.5.2 - 2026-01-27#
Fixed#
Improve resiliency of JSON logging system for cloudwatch logs
5.5.1 - 2026-01-27#
Fixed#
Fixed the netcdf writing needed for L1B camera product and clarifying netcdf engine usage
5.5.0 - 2026-01-27#
Added#
Allows the usage of the profile argument in the cli tools for specifying AWS credentials profiles
Changed#
Adjusted CLI interface for AWS tools to improve usability and consistency
Changing the s3 utilities for list and put to use the new constants tools more effectively
5.4.8 - 2026-01-22#
Added#
Add lookups between L1a, APID, and L1a product definitions
5.4.7 - 2026-01-22#
Added#
Update Space Packet Parser dependency to take advantage of packet filtering functionality
5.4.6 - 2026-01-22#
Fixed#
update xarray version due to breaking change in netcdf writing
5.4.5 - 2026-01-16#
Added#
Add enforcement of valid versioning in Filename classes
5.4.4 - 2026-01-13#
Added#
Allow passing dynamic product level attributes to data product writer
5.4.3 - 2026-01-10#
Added#
Allow adding the archive prefix to the path when calling write_libera_data_product
5.4.2 - 2026-01-09#
Changed#
Change xarray netcdf engine to h5netcdf because netcdf4 does not support writing to filelike objects (e.g. opened files).
5.4.1 - 2026-01-09#
Added#
Added a KernelManager class to handle SPICE kernel caching and retrieval
5.4.0 - 2025-12-08#
Changed#
Refactor kernel_maker to use L1a Dataset object. Packet-based kernel creation is still possible via
create_kernels_from_packets, which runs L1a processing internallyUpdate Curryer kernel configs to match Libera AXIS_SAMPLE packets fields
Move the L1A processing configurations to YAML instead of Python objects
Fixed#
Fix bug in
packets.py::_drop_duplicatesto allow dropping based on coordinate or dimension
5.3.0 - 2025-12-08#
Added#
scene_id module alpha release capable of identifying scenes from footprint data in Ceres SSF files.
5.2.2 - 2025-12-03#
Changed#
Updated the Curryer version pin to use at least 0.2.0, which is when the problematic subdependencies were removed
5.2.1 - 2025-11-14#
Added#
Added scene_id module for processing footprint data to produce datasets with assigned scene IDs according to libera_utils standard definitions.
5.2.0 - 2025-11-12#
Added#
Implement processing to L1A Dataset from packets
Changed#
Add full ICIE XTCE to Libera Utils package
5.1.0 - 2025-11-07#
Added#
Added UMMGranule Pydantic class to validate datasets against the UMM-G schema
Added methods to convert Libera datasets to UMMGranule instances
5.0.0 - 2025-11-07#
Added#
Add
write_libera_data_producttonetcdf.pyfor one-shot product writes from numpy arrays
Changed#
BREAKING: Refactor
DataProductDefinition(previouslyDataProductConfig) and product definition validationRemove
hdf.pyand associated tests
Removed#
BREAKING: Remove L1A
ProcessingStepIdentifier(node) enum constants
4.0.0 - 2025-09-18#
Added#
New DataLevel enum: Added structured processing level enumeration with built-in archive bucket name mapping
Added the Libera APID values we care about for science processing to the LiberaApid enum class
Changed#
BREAKING: Refactored identifier classes: Moved DataProductIdentifier and ProcessingStepIdentifier from libera_utils.aws.constants to libera_utils.constants
BREAKING: SPICE kernel filename consolidation: Merged separate SPICE kernel filename classes into LiberaProductFilename
Enhanced DataProductIdentifier: Now includes embedded metadata with processing level information and simplified lookup methods
Enhanced ProcessingStepIdentifier: Now includes embedded product relationships and improved level derivation
Filename validation: Updated L0 filename patterns to remove chunk number validation
Improved type safety: Better type hints and validation throughout identifier classes
Updated documentation: Refreshed user documentation to reflect new file naming conventions
Removed#
BREAKING: Removed L0 file chunking support: The chunk_number parameter and related methods have been removed from L0 data product identifiers
BREAKING: Removed deprecated classes: ProductName, CkObject, and SpkObject enums have been removed and consolidated into DataProductIdentifier
3.2.4 - 2025-09-11#
Added#
Added the SPICE kernel creation Docker image
Added the Geolocation Tier-1 integration test case and data files
Changed#
Changed the SPICE kernel creation API
3.2.3 - 2025-09-11#
Added#
Added step function and policy names to ProcessingStepIdentifier
Changed#
Improved ECR authentication issues with the
libera-utils ecr-uploadCLI command for working with SSO authenticationUpdated S3 list tool to print by default
Improved tooling of the aws constants module for both DataProductIdentifier and ProcessingStepIdentifier
Updated constants for L2 data products
Updated step function trigger tool with url output and naming improvements
Exposed useful L2 tools at highest level of the
libera_utilspackage
3.2.2 - 2025-06-27#
Added#
Added dimension handling for data product configuration
Added internal DataArray data storage matching to known dimensions
Added jenkins file for publishing to PyPI
Added contact listserv for project
Added json and yaml linting to the pre-commit hooks
Added prettier configuration for formatting markdown, json, and yaml files
Added internal Dataset object to the Configurable NetCDF-4 metadata object
Added writing of output file for the Configurable NetCDF-4 data product object
3.2.1 - 2025-06-13#
Added#
Added automatic formatting to the repository using
pre-commithooksAdded a list-serv for ci/cd output as part of the Jenkins files
Fixed#
Fixed filenaming changes from 3.2.0 that affected the CDK deployment and usage
3.2.0 - 2025-05-19#
Added#
Added pydantic models for a Configurable NetCDF-4 metadata object
Changed#
Includes static metadata that is populated and known by these tools
Includes dynamic metadata that must be provided by user of these tools
Changed the naming scheme of the SPICE data files to include SPK or CK
3.1.1 - 2025-05-08#
Fixed#
Testing improvement for mocking docker image building
3.1.0 - 2025-05-08#
Added#
Added Curryer (lasp-curryer) library dependency for SPICE kernel creation and geolocation
Added static SPICE kernels and configuration files for geolocating NOAA-20 / CERES
Removed#
BREAKING: Removed geolocation submodule, replaced by interfaces within Curryer
3.0.0 - 2025-04-25#
Changed#
BREAKING: Updated the aws constants ProcessingStepIdentifier dump method to be to_str_with_chunk_number
Improved the standardization of CLI commands to call cli_handler functions that wrap the main functionality
Removed#
BREAKING: Removed support for python version 3.9 and 3.10
2.5.2 - 2025-03-03#
Added#
Added the s3-utils cli interface with subcommands put, list, and cp for ease of use s3 interactions
Added the Libera Archive bucket naming pattern to the aws constants module
Changed#
The cli subcommand
libera-utils s3-utils putwill upload a file to the correct S3 archive bucket given an algorithmThe cli subcommand
libera-utils s3-utils listwill list all files in a given S3 archive bucket for a given algorithmThe cli subcommand
libera-utils s3-utils cpwill copy a file from one S3 archive bucket to another locationImproved type hinting for the smart_open module
Improved readability of cli testing
Removed#
Removing testing against python 3.9 and 3.10
2.5.1 - 2025-01-22#
Fixed#
missing pyyaml dependency that prevented usage of logutil module
2.5.0 - 2024-12-14#
Changed#
Reimplement Manifest class as a pydantic model and integrate with dependent code
BREAKING: Rename outpath to out_path on the Manifest class write method
Removed#
BREAKING: Remove the deprecated add_file_to_manifest method from Manifest class
2.4.5 - 2024-11-27#
Added#
Add
--ecr-image-tagsoption tolibera-utils ecr-uploadCLI for tagging remote algorithm images in an ECRAdd validation and serialization methods to the
DataProductIdentifierandProcessingStepProductIdentifierenums
2.4.4 - 2024-10-21#
Changed#
Allow overriding the standard docker config.json file with a minimal file for ECR uploading to prevent ECR upload permission failures from cached docker login credentials from CDK deployments
2.4.3 - 2024-10-16#
Changed#
Changes to ecr_upload to support programmatic building and pushing of Docker images
Removed#
Remove DynamoDB docs (moved to
libera_cdk)
2.4.2 - 2024-10-14#
Removed#
BREAKING: Remove the
AnyFilenamepolymorphic class. Please useAbstractValidFilename.from_file_path()
2.4.1 - 2024-10-04#
Changed#
Updating requirements of methods to use keyword arguments rather than positional arguments
Adding ProcessingStepIdentifier and DataProductIdentifier to the filenaming classes
Updating ecr names to work with the completion checker testing in libera_cdk
2.4.0 - 2024-09-11#
Added#
Add properties to filenaming classes to retrieve
data_product_idandprocessing_step_idAdd ProcessingStepIdentifier and DataProductIdentifier standardization to be used by downstream repos
2.3.1 - 2024-08-28#
Fixed#
Fix os.path.join bug in filenaming module that broke mocked S3 paths and also fix typehinting
2.3.0 - 2024-06-27#
Added#
Create CLI tools for AWS ECR image upload and Step Function triggering
Create tools for DynamoDB in AWS for .pds files (CONS and PDS)
Changed#
Update manifest filenames to use ULID instead of timestamp for unique identifiers
Change logutil configure_task_logging to optionally log JSON to console
Allow configure_task_logging to optionally propagate DEBUG messages from specific loggers
Update documentation for how the database is used in the Libera project in DynamoDB
Replace the use of PostgreSQL with DynamoDB for the Libera project
2.2.0 - 2023-12-14#
Added#
Add AnyFilename polymorphic class
Add prefixing to Filename classes for predictable archive paths
Add prefixing for manifest files for predictable and navigable paths in s3 buckets
Added handling of construction records and pds files appropriately when ingesting
This includes reading a construction record and removing the pds file entry for the construction record itself
Added output manifest creation from input manifest to match timestamps in filenames of input and output manifests
Added error handling to pds ingest
Changed#
Change filename of all products to a LiberaDataProductFilename that inherits from AnyFilename
Update filenaming convention to be all capital letters
Improve API for manifest module
Update git to include lfs and move test data to lfs
Improve database manager including caching improvements
Improve smart_copy_file and bug fixes to smart_open testing
Refactoring and improving pds ingest for database entries and integration testing in CDK
Improved testing of pds ingest and pds file orm models to more accurately reflect use cases
Refactored pds ingest to use AnyPath objects for handling file locations
2.1.1 - 2023-05-16#
Added#
Add preliminary instrument kernel
Changed#
Update dependency specification to speed up dependency resolution wrt botocore/urllib3
Improve database initialization to work with libera_cdk changes
Fixed#
Fix bug in Dockerfile that incorrectly set the default entrypoint
2.1.0 - 2023-04-20#
Added#
Add manifest filename enforcement to Manifest class
Changed#
Improve API to Manifest and Manifest.add_files
Update filenaming conventions for product filenames and SPICE kernels
Allow adding an s3 bucket/prefix as a basepath for filenames
2.0.1 - 2023-02-07#
Removed#
Remove the extras dependency spec because of the way SQLAlchemy imports models
2.0.0 - 2023-01-31#
Added#
Add filenaming classes
Add manifest file class
Add construction record parser
Add smart_copy_file function that can copy files to and from S3 and filesystem locations transparently
Add quality flag classes
Changed#
Update DB schema to store construction records
Update kernel generation CLI to use manifest file pattern
Shift database and spice related libraries to extras (not installed by default)
Change license to BSD3
Removed#
Remove HDF-EOS5 filehandling code
1.0.0 - 2022-09-23#
Added#
Add build and release processes to readme
Add geolocation module
Add tools in spiceutil module for caching SPICE kernels from NAIF
Add missing unit testing coverage
Add spice.md documentation on how the package uses and manages SPICE kernels
Add database tooling, dev database, and ORM setup
Add smart_open for opening local or S3 objects
Add logging utility functions for setting up application logging
Changed#
Stub out project structure
Switch to Poetry for project dependency configuration and build management