libera_utils.aws.s3_utilities#

Module for S3 cli utilities

Functions

s3_copy_cli_handler(parsed_args)

CLI handler function for s3-utils cp CLI subcommand.

s3_list_archive_files(processing_step, *[, ...])

List all files in an archive S3 bucket for a given processing step.

s3_list_cli_handler(parsed_args)

CLI handler function for s3-utils list CLI subcommand.

s3_put_cli_handler(parsed_args)

CLI handler function for s3-utils put CLI subcommand.

s3_put_in_archive_for_processing_step(...[, ...])

Upload a file to the archive S3 bucket associated with a given processing step.
libera_utils.aws.s3_utilities.s3_copy_cli_handler(parsed_args: Namespace) → None#

CLI handler function for s3-utils cp CLI subcommand.

libera_utils.aws.s3_utilities.s3_list_archive_files(processing_step: str | ProcessingStepIdentifier, *, account_suffix: str | LiberaAccountSuffix | None = LiberaAccountSuffix.STAGE, print_out: bool | None = False) → list#

List all files in an archive S3 bucket for a given processing step.

Parameters:

  • processing_step (str) – Processing step ID string. Used to infer the S3 archive bucket name.

  • account_suffix (Union[str, constants.LiberaAccountSuffix], optional) – Account suffix for the bucket name, by default constants.LiberaAccountSuffix.STAGE

  • print_out (bool, optional) – Print the list of files to the console, by default False

Returns:

bucket_objects – S3Path objects for each file in the bucket

Return type:

list

libera_utils.aws.s3_utilities.s3_list_cli_handler(parsed_args: Namespace) → None#

CLI handler function for s3-utils list CLI subcommand.

libera_utils.aws.s3_utilities.s3_put_cli_handler(parsed_args: Namespace) → None#

CLI handler function for s3-utils put CLI subcommand.

libera_utils.aws.s3_utilities.s3_put_in_archive_for_processing_step(path_to_file: Path | S3Path, processing_step: str | ProcessingStepIdentifier, *, account_suffix: str | LiberaAccountSuffix | None = LiberaAccountSuffix.STAGE)#

Upload a file to the archive S3 bucket associated with a given processing step.

Parameters:

  • path_to_file (Path) – Local path to the file to upload

  • processing_step (str) – processing_step : Union[str, constants.ProcessingStepIdentifier] Processing step ID string or object. Used to infer the S3 archive bucket name.

  • account_suffix (Union[str, constants.LiberaAccountSuffix], optional) – Account suffix for the bucket name, by default constants.LiberaAccountSuffix.STAGE