libera_utils.io.manifest.Manifest#

class libera_utils.io.manifest.Manifest(*, manifest_type: ~libera_utils.aws.constants.ManifestType, files: ~typing.List[~libera_utils.io.manifest.ManifestFileRecord] = <factory>, configuration: ~typing.Dict[str, ~typing.Any] = <factory>, filename: ~libera_utils.io.filenaming.ManifestFilename | None = None, ulid_code: ~ulid.ULID | None = <factory>)#

Bases: BaseModel

Pydantic model for a manifest file.

Attributes:
model_extra

Get extra fields set during validation.

model_fields_set

Returns the set of fields that have been explicitly set on this model instance.

Methods

add_desired_time_range(start_datetime, ...)

Add a time range to the configuration section of the manifest.

add_files(*files)

Add files to the manifest from filename

check_file_structure(file_structure, ...)

Check file structure, returning True if it is good.

copy(*[, include, exclude, update, deep])

Returns a copy of the model.

from_file(filepath)

Read a manifest file and return a Manifest object (factory method).

model_construct([_fields_set])

Creates a new instance of the Model class with validated data.

model_copy(*[, update, deep])

Usage docs: https://docs.pydantic.dev/2.10/concepts/serialization/#model_copy

model_dump(*[, mode, include, exclude, ...])

Usage docs: https://docs.pydantic.dev/2.10/concepts/serialization/#modelmodel_dump

model_dump_json(*[, indent, include, ...])

Usage docs: https://docs.pydantic.dev/2.10/concepts/serialization/#modelmodel_dump_json

model_json_schema([by_alias, ref_template, ...])

Generates a JSON schema for a model class.

model_parametrized_name(params)

Compute the class name for parametrizations of generic classes.

model_post_init(_BaseModel__context)

Override this method to perform additional initialization after __init__ and model_construct.

model_rebuild(*[, force, raise_errors, ...])

Try to rebuild the pydantic-core schema for the model.

model_validate(obj, *[, strict, ...])

Validate a pydantic model instance.

model_validate_json(json_data, *[, strict, ...])

Usage docs: https://docs.pydantic.dev/2.10/concepts/json/#json-parsing

model_validate_strings(obj, *[, strict, context])

Validate the given object with string data against the Pydantic model.

output_manifest_from_input_manifest(...)

Create Output manifest from input manifest file path, adds input files to output manifest configuration

serialize_filename(filename, _info)

Custom serializer for the manifest filename.

transform_filename(raw_filename)

Convert raw filename to ManifestFilename class if necessary.

transform_files(raw_list)

Allow for the incoming files list to have varying types.

validate_checksums()

Validate checksums of listed files

write(out_path[, filename])

Write a manifest file from a Manifest object (self).

construct

dict

from_orm

json

parse_file

parse_obj

parse_raw

schema

schema_json

update_forward_refs

validate
__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Methods

add_desired_time_range(start_datetime, ...)

Add a time range to the configuration section of the manifest.

add_files(*files)

Add files to the manifest from filename

check_file_structure(file_structure, ...)

Check file structure, returning True if it is good.

from_file(filepath)

Read a manifest file and return a Manifest object (factory method).

output_manifest_from_input_manifest(...)

Create Output manifest from input manifest file path, adds input files to output manifest configuration

serialize_filename(filename, _info)

Custom serializer for the manifest filename.

transform_filename(raw_filename)

Convert raw filename to ManifestFilename class if necessary.

transform_files(raw_list)

Allow for the incoming files list to have varying types.

validate_checksums()

Validate checksums of listed files

write(out_path[, filename])

Write a manifest file from a Manifest object (self).

Attributes

model_computed_fields

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_extra

Get extra fields set during validation.

model_fields

model_fields_set

Returns the set of fields that have been explicitly set on this model instance.

manifest_type

files

configuration

filename

ulid_code

_generate_filename() → ManifestFilename#

Generate a valid manifest filename

add_desired_time_range(start_datetime: datetime, end_datetime: datetime)#

Add a time range to the configuration section of the manifest.

Parameters:

  • start_datetime (datetime.datetime) – The desired start time for the range of data in this manifest

  • end_datetime (datetime.datetime) – The desired end time for the range of data in this manifest

Return type:

None

add_files(*files: str | Path | S3Path)#

Add files to the manifest from filename

Parameters:

files (Union[str, Path, S3Path]) – Path to the file to add to the manifest.

Return type:

None

classmethod check_file_structure(file_structure: ManifestFileRecord, existing_names: set[str], existing_checksums: set[str]) → bool#

Check file structure, returning True if it is good.

classmethod from_file(filepath: str | Path | S3Path)#

Read a manifest file and return a Manifest object (factory method).

Parameters:

filepath (Union[str, Path, S3Path]) – Location of manifest file to read.

Returns:

Pydantic model built from the json of the given manifest file.

Return type:

Manifest

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None#

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or None if config.extra is not set to “allow”.

property model_fields_set: set[str]#

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,

i.e. that were not filled from defaults.

classmethod output_manifest_from_input_manifest(input_manifest: Path | S3Path | Manifest) → Manifest#

Create Output manifest from input manifest file path, adds input files to output manifest configuration

Parameters:

input_manifest (Union[Path, S3Path, 'Manifest']) – An S3 or regular path to an input_manifest object, or the input manifest object itself

Returns:

output_manifest – The newly created output manifest

Return type:

Manifest

serialize_filename(filename: str | Path | S3Path | ManifestFilename | None, _info) → str#

Custom serializer for the manifest filename.

classmethod transform_filename(raw_filename: str | ManifestFilename | None) → ManifestFilename | None#

Convert raw filename to ManifestFilename class if necessary.

classmethod transform_files(raw_list: list[dict | str | Path | S3Path | ManifestFileRecord] | None) → list[ManifestFileRecord]#

Allow for the incoming files list to have varying types. Convert to a standardized list of ManifestFileStructure.

validate_checksums() → None#

Validate checksums of listed files

write(out_path: str | Path | S3Path, filename: str = None) → Path | S3Path#

Write a manifest file from a Manifest object (self).

Parameters:

  • out_path (Union[str, Path, S3Path]) – Directory path to write to (directory being used loosely to refer also to an S3 bucket path).

  • filename (str, Optional) – must be a valid manifest filename. If not provided, the method uses the objects internal filename attribute. If that is not set, then a filename is automatically generated.

Returns:

The path where the manifest file is written.

Return type:

Union[Path, S3Path]