libera_utils.io.filenaming#

Module for file naming utilities

Functions

`check_version_number_format`(version)	Ensures that a version string is in VM-m-p format for Libera filenaming.
`format_from_semantic_version`(semantic_version)	Formats a semantic version string X.Y.Z into a filename-compatible string like VX-Y-Z, for X = major version, Y = minor version, Z = patch.
`get_current_version_str`(package_name)	Retrieve the current version of a (algorithm) package and format it for inclusion in a filename

Classes

`AbstractDataProductFilename`(args, *kwargs)	Abstract base class for data product filenames.
`AbstractValidFilename`(args, *kwargs)	Filename class that ensures validity of a filename based on regex pattern.
`L0Filename`(args, *kwargs)	Filename validation class for L0 Production Data Set (PDS) files from EDOS.
`LiberaDataProductFilename`(args, *kwargs)	Filename validation class for Libera SDC data products.
`LiberaGroundCcsdsFilename`(args, *kwargs)	Filename validation class for demuxed ground-test CCSDS files.
`ManifestFilename`(args, *kwargs)	Class for naming manifest files

class libera_utils.io.filenaming.AbstractDataProductFilename(*args, **kwargs)#

Abstract base class for data product filenames.

This class adds the data product specific requirements that all data products must have: a processing step ID and a data product ID. For example, an L0Filename or a LiberaDataProductFilename are both AbstractDataProductFilenames.

Attributes:

archive_prefix: Property that contains the generated prefix used for archiving, when applicable
data_product_id: Property that contains the DataProductIdentifier for this file type
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(args, *kwargs)	Abstract method that must be implemented to provide hinting for required parts
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

abstract property data_product_id: DataProductIdentifier#: Property that contains the DataProductIdentifier for this file type

class libera_utils.io.filenaming.AbstractValidFilename(*args, **kwargs)#

Filename class that ensures validity of a filename based on regex pattern.

Notes

This is an abstract base class that must be inherited by concrete filename classes.
This class internally stores a CloudPath or Path object in the path property (composition).

Attributes:

archive_prefix: Property that contains the generated prefix used for archiving, when applicable
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(args, *kwargs)	Abstract method that must be implemented to provide hinting for required parts
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

abstractmethod classmethod _format_filename_parts(*args: Any, **kwargs: Any)#

Format parts into a filename

Note: When this is implemented by concrete classes, *args and **kwargs become specific parameters

classmethod _from_filename_parts(*, basepath: str | Path | S3Path | None = None, **parts: Any)#

Create instance from filename parts.

The part kwarg names are named according to the regex for the file type.

Parameters:

basepath (Union[str, Path, S3Path], Optional) – Allows prepending a basepath or prefix.
parts (Any) – Passed directly to _format_filename_parts. This is a dict of variable kwargs that will differ in each filename class based on the required parts for that particular filename type.

Return type:

AbstractValidFilename

abstractmethod _parse_filename_parts()#

Parse the filename parts into objects from regex matched strings

Returns:: namespace object containing filename parts as parsed objects
Return type:: types.SimpleNamespace

abstract property archive_prefix: str#: Property that contains the generated prefix used for archiving, when applicable

property filename_parts#: Property that contains a namespace of filename parts

classmethod from_file_path(*args, **kwargs)#: Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)

abstractmethod classmethod from_filename_parts(*args: Any, **kwargs: Any)#: Abstract method that must be implemented to provide hinting for required parts

generate_prefixed_path(parent_path: str | CloudPath | Path) → CloudPath | Path#

Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)

Parameters:: parent_path (Union[str, Path, S3Path]) – Absolute path to the parent directory or S3 bucket prefix. The generated path prefix is appended to the parent path and followed by the file basename.
Return type:: pathlib.Path or cloudpathlib.s3.s3path.S3Path

property path: CloudPath | Path#: Property containing the file path

regex_match(path: CloudPath | Path)#

Parse and validate a given path against class-attribute defined regex

Parameters:: path (Union[Path, CloudPath]) – Path to validate
Returns:: Match group dict of filename parts
Return type:: dict

class libera_utils.io.filenaming.L0Filename(*args, **kwargs)#

Filename validation class for L0 Production Data Set (PDS) files from EDOS.

Attributes:

archive_prefix: Property that contains the generated prefix for L0 archiving
data_product_id: Property that contains the DataProductIdentifier for this file type
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(*, id_char, scid, ...[, ...])	Create instance from filename parts
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

classmethod _format_filename_parts(*, id_char: str, scid: int, first_apid: int, fill: str, created_time: datetime, numeric_id: int, file_number: int, extension: str, signal: str | None = None)#

Construct a path from filename parts

Parameters:

id_char (str) – Either P (for PDS files, Construction Records) or X (for Delivery Records)
scid (int) – Spacecraft ID
first_apid (int) – First APID in the file
fill (str) – Custom string up to 14 characters long
created_time (datetime.datetime) – Creation time of the file
numeric_id (int) – Data set ID, 0-9, one digit
file_number (str) – File number within the data set. Construction records are always file number zero.
extension (str) – File name extension. Either PDR or PDS
signal (Optional[str], Optional) – Optional signal suffix. Always ‘.XFR’

Returns:

Formatted filename

Return type:

str

_parse_filename_parts()#

Parse the filename parts into objects from regex matched strings

Returns:: namespace object containing filename parts as parsed objects
Return type:: types.SimpleNamespace

property archive_prefix: str#: Property that contains the generated prefix for L0 archiving

property data_product_id: DataProductIdentifier#: Property that contains the DataProductIdentifier for this file type

classmethod from_filename_parts(*, id_char: str, scid: int, first_apid: int, fill: str, created_time: datetime, numeric_id: int, file_number: int, extension: str, signal: str | None = None, basepath: str | Path | S3Path | None = None)#

Create instance from filename parts

This method exists primarily to expose typehinting to the user for use with the generic _from_filename_parts. The part names are named according to the regex for the file type.

Parameters:

id_char (str) – Either P (for PDS files, Construction Records) or X (for Delivery Records)
scid (int) – Spacecraft ID
first_apid (int) – First APID in the file
fill (str) – Custom string up to 14 characters long
created_time (datetime.datetime) – Creation time of the file
numeric_id (int) – Data set ID, 0-9, one digit
file_number (str) – File number within the data set. Construction records are always file number zero.
extension (str) – File name extension. Either PDR or PDS
signal (Optional[str]) – Optional signal suffix. Always ‘.XFR’
basepath (Optional[Union[str, Path, S3Path]]) – Allows prepending a basepath or prefix.

Return type:

L0Filename

class libera_utils.io.filenaming.LiberaDataProductFilename(*args, **kwargs)#

Filename validation class for Libera SDC data products.

Attributes:

applicable_date: Property that returns the applicable date based on the midpoint of start and end times.
archive_prefix: Property that contains the generated prefix for L1B and L2 archiving
data_product_id: Property that contains the DataProductIdentifier for this file type
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path
processing_step_id: Property that contains the ProcessingStepIdentifier that generates this file
ummg_metadata_filename: Property that returns the corresponding UMM-G metadata filename for this data product file.

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(*, product_name, ...[, ...])	Create instance from filename parts.
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

classmethod _format_filename_parts(*, data_level: str, product_name: str, version: str, utc_start: datetime, utc_end: datetime, revision: datetime, extension: str)#

Construct a path from filename parts

Parameters:

data_level (str) – L1B or L2
product_name (str) – Libera instrument, cam or rad for L1B and cloud-fraction etc. for L2. May contain anything except for underscores.
version (str) – Software version that the file was created with. Corresponds to the algorithm version as determined by the algorithm software.
utc_start (datetime.datetime) – First timestamp in the SPK
utc_end (datetime.datetime) – Last timestamp in the SPK
revision (datetime.datetime) – Time when the file was created.
extension (str) – File extension (.nc or .h5)

Returns:

Formatted filename

Return type:

str

_parse_filename_parts()#

Parse the filename parts into objects from regex matched strings

Returns:: namespace object containing filename parts as parsed objects
Return type:: types.SimpleNamespace

property applicable_date: date#

Property that returns the applicable date based on the midpoint of start and end times.

Issues a warning if the time range covers more than 24 hours.

Returns:: The date of the midpoint between utc_start and utc_end
Return type:: datetime.date

property archive_prefix: str#: Property that contains the generated prefix for L1B and L2 archiving

property data_product_id: DataProductIdentifier#: Property that contains the DataProductIdentifier for this file type

classmethod from_filename_parts(*, product_name: str | DataProductIdentifier, version: str, utc_start: datetime, utc_end: datetime, data_level: str | DataLevel | None = None, revision: datetime = datetime.datetime(2026, 9, 25, 20, 10, 24, 986831, tzinfo=datetime.timezone.utc), extension: str | None = None, basepath: str | Path | S3Path | None = None)#

Create instance from filename parts. All keyword arguments other than basepath are required!

This method exists primarily to expose typehinting to the user for use with the generic _from_filename_parts. The part names are named according to the regex for the file type.

Parameters:

data_level (str | DataLevel | None) – L1B or L2 identifying the level of the data product. Default None will infer the data level from the product name (DataProductIdentifier)
product_name (str | DataProductIdentifier) – Product type. e.g. CF-CAM for L2 or RAD-4CH for L1B. May contain anything except for underscores.
version (str) – Software version that the file was created with. Corresponds to the algorithm version as determined by the algorithm software.
utc_start (datetime.datetime) – First timestamp in the SPK
utc_end (datetime.datetime) – Last timestamp in the SPK
revision (datetime.datetime) – Time when the file was created. Default is now in UTC time.
extension (str | None) – File extension. Default None will infer extension based on product_name.
basepath (Optional[Union[str, Path, S3Path]]) – Allows prepending a basepath or prefix.

Return type:

LiberaDataProductFilename

property processing_step_id: ProcessingStepIdentifier | None#: Property that contains the ProcessingStepIdentifier that generates this file

property ummg_metadata_filename: Path | S3Path#

Property that returns the corresponding UMM-G metadata filename for this data product file.

Returns:: Same base filename with a Common Metadata Repository(CMR) JSON extension.
Return type:: Path | S3Path

class libera_utils.io.filenaming.LiberaGroundCcsdsFilename(*args, **kwargs)#

Filename validation class for demuxed ground-test CCSDS files.

Canonical form: LIBERA_SDC_<apid>_ccsds_<yyyy>_<doy>_<hh>_<mm>_<ss> (no extension), e.g. LIBERA_SDC_1057_ccsds_2026_191_14_00_00. Each file holds packets for the single APID named in the basename, with no record header before the CCSDS primary header.

The time fields are the start of the time bin the file was cut from, not the span of the packets inside it: packet times can fall outside the bin. The archive prefix GroundCCSDS/<apid>/<yyyy>/<mm>/<dd>/ is an expansion of these fields and carries no guarantee about packet or data times. Searchable times come from File Metadata at ingest.

Attributes:

apid: CCSDS APID encoded in the filename.
archive_prefix: L0 archive prefix from the filename’s APID and bin start.
bin_start: UTC start of the time bin encoded in the filename.
data_product_id: Property that contains the DataProductIdentifier for this file type.
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(*, apid, year, doy, ...)	Create instance from filename parts.
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

classmethod _format_filename_parts(*, apid: int, year: int, doy: int, hour: int, minute: int, second: int)#: Construct a basename from filename parts.

_parse_filename_parts()#: Parse the filename parts into objects from regex matched strings.

property apid: int#

CCSDS APID encoded in the filename.

Not necessarily a defined LiberaApid member; ingest accepts any legal APID.

property archive_prefix: str#: L0 archive prefix from the filename’s APID and bin start.

property bin_start: datetime#: UTC start of the time bin encoded in the filename.

property data_product_id: DataProductIdentifier#: Property that contains the DataProductIdentifier for this file type.

classmethod from_filename_parts(*, apid: int, year: int, doy: int, hour: int, minute: int, second: int, basepath: str | Path | S3Path | None = None)#

Create instance from filename parts.

Parameters:

apid (int) – CCSDS APID of the packets in the file (0-2047).
year (int) – Four-digit UTC year of the bin start.
doy (int) – Day of year (1-366).
hour (int) – Bin start time-of-day in UTC.
minute (int) – Bin start time-of-day in UTC.
second (int) – Bin start time-of-day in UTC.
basepath (Optional[Union[str, Path, S3Path]]) – Optional directory or S3 prefix prepended to the basename.

Return type:

LiberaGroundCcsdsFilename

property path: CloudPath | Path#: Property containing the file path

class libera_utils.io.filenaming.ManifestFilename(*args, **kwargs)#

Class for naming manifest files

Attributes:

archive_prefix: Manifests are not archived like data products, but for convenience and ease of debugging they will be kept in the dropbox bucket by input/output and day they were made.
filename_parts: Property that contains a namespace of filename parts
path: Property containing the file path

Methods

`from_file_path`(args, *kwargs)	Factory method to produce an AbstractValidFilename from a valid Libera file path (str or Path)
`from_filename_parts`(manifest_type, ulid_code)	Create instance from filename parts.
`generate_prefixed_path`(parent_path)	Generates an absolute path of the form {parent_path}/{prefix_structure}/{file_basename} The parent_path can be an S3 bucket or an absolute local filepath (must start with /)
`regex_match`(path)	Parse and validate a given path against class-attribute defined regex

classmethod _format_filename_parts(manifest_type: ManifestType, ulid_code: ULID)#

Construct a path from filename parts

Parameters:

manifest_type (ManifestType) – Input or output
ulid_code (ulid.ULID) – ULID code for use in filename parts

Returns:

Formatted filename

Return type:

str

_parse_filename_parts()#

Parse the filename parts into objects from regex matched strings

Returns:: namespace object containing filename parts as parsed objects
Return type:: types.SimpleNamespace

property archive_prefix: str#: Manifests are not archived like data products, but for convenience and ease of debugging they will be kept in the dropbox bucket by input/output and day they were made. This is used by the step function clean up function in the CDK. # Generate prefix structure # <manifest_type>/<year>/<month>/<day>

classmethod from_filename_parts(manifest_type: ManifestType, ulid_code: ULID, basepath: str | Path | S3Path | None = None)#

Create instance from filename parts.

This method exists primarily to expose typehinting to the user for use with the generic _from_filename_parts. The part names are named according to the regex for the file type.

Parameters:

manifest_type (ManifestType) – Input or output
ulid_code (ulid.ULID) – ULID code for use in filename parts
basepath (Optional[Union[str, Path, S3Path]]) – Allows prepending a basepath or prefix.

Return type:

ManifestFilename

libera_utils.io.filenaming._ensure_utc_timezone(dt_obj: datetime) → datetime#

Ensure datetime object has UTC timezone info.

If the datetime is timezone-naive, assume it is in UTC and add timezone info. If the datetime is timezone-aware, convert it to UTC.

Parameters:: dt_obj (datetime) – Input datetime object
Returns:: Timezone-aware datetime in UTC
Return type:: datetime

libera_utils.io.filenaming._parse_ground_ccsds_bin_start(year: int, doy: int, hour: int, minute: int, second: int) → datetime#

Build the UTC bin start for a ground CCSDS file from its filename fields.

Raises:: ValueError – If the fields do not name a real instant. This includes DOY 366 in a common year, which strptime rolls into 1 January of the following year rather than rejecting.

libera_utils.io.filenaming._validate_ccsds_apid(apid: int) → int#: Return apid if it fits the 11-bit CCSDS APID field, else raise ValueError.

libera_utils.io.filenaming.check_version_number_format(version: str) → bool#

Ensures that a version string is in VM-m-p format for Libera filenaming. M, m, and p are integers representing Major, minor, and patch respectively.

Parameters:: version (str) – Version string to validate
Returns:: True if version string is in VM-m-p format, False otherwise
Return type:: bool

libera_utils.io.filenaming.format_from_semantic_version(semantic_version: str) → str#

Formats a semantic version string X.Y.Z into a filename-compatible string like VX-Y-Z, for X = major version, Y = minor version, Z = patch.

Result is uppercase. Release candidate suffixes are allowed as no strict checking is done on the contents of X, Y, or Z. e.g. 1.2.3rc1 becomes V1-2-3RC1

Parameters:: semantic_version (str) – String matching X.Y.Z where X, Y and Z are integers of any length
Return type:: str

libera_utils.io.filenaming.get_current_version_str(package_name: str) → str#

Retrieve the current version of a (algorithm) package and format it for inclusion in a filename

Parameters:: package_name (str) – Package for which to retrieve a version string. This should be your algorithm package and it must use a semantic versioning scheme, configured in project metadata.
Returns:: Version string in format V1-2-3
Return type:: str