libera_utils.aws.manual_processing.start_manual_processing#

libera_utils.aws.manual_processing.start_manual_processing(applicable_dates: list[str | date | datetime], *, boto_session: Session, custom_dag_config: dict | None = None, start_processing_step_ids: list[str | ProcessingStepIdentifier] | None = None, process_downstream: bool = True, job_ids: list[ULID] | None = None, verify: bool = False, wait_time: float = 60) → list[ULID]#

Start a manual processing job by emitting a single ManualProcessing event to the SDC event bus.

One job graph is created per applicable date. The SDC Job Creator builds the job(s) from the event and the rest of the SDC orchestration runs them. Input products are assumed to already be ingested into the SDC.

Parameters:

  • applicable_dates (list of str, date, or datetime) – The applicable dates to process data for. A separate job is created for each date.

  • boto_session (boto3.Session) – Boto3 session used for all AWS interactions. Created once by the CLI handler (with the LiberaUtils role assumed) and passed in so the same authenticated session is used throughout.

  • custom_dag_config (dict, optional) – A custom DAG configuration ({"nodes": {...}} with kebab-case node keys). When None (default) the SDC’s static default DAG is used.

  • start_processing_step_ids (list of str or ProcessingStepIdentifier, optional) – Entry nodes for the job graph. When None (default) the Job Creator starts from the DAG’s root nodes.

  • process_downstream (bool, optional) – When True (default) the job graph includes all downstream descendants of the start nodes. When False, only the start nodes themselves are included – the simple way to run a single node from the default DAG.

  • job_ids (list[ULID], optional) – Caller-supplied job ids, paired positionally with applicable_dates (lengths must match). When None and verify is True, fresh ulids are minted so the created jobs can be polled for. When None and verify is False, the Job Creator mints its own ids and none are sent.

  • verify (bool, optional) – When True, poll the Coordination Table to confirm each job was created. This requires job ids to be known ahead of time, so they are minted when not supplied. Defaults to False. Note that the step function console URL(s) are logged regardless of this flag.

  • wait_time (float, optional) – Maximum seconds to wait when verifying job creation, by default 60.

Returns:

The job ids associated with the submitted jobs (empty if none were supplied or minted).

Return type:

list[ULID]