libera_utils.l1a.packets.parse_packets_to_dataset#

libera_utils.l1a.packets.parse_packets_to_dataset(packet_files: list[PathLike | str], packet_definition: str | PathLike | XtcePacketDefinition, apid: int, *, max_workers: int = 1, **generator_kwargs) → Dataset#

Parse packets from files into an xarray Dataset using specified packet definition.

This function does not make any changes to the packet data other than filtering by a single APID.

Files are parsed one at a time and concatenated, rather than handed to create_dataset together. create_dataset retains a Python object per parsed field until it converts the whole batch to numpy, which costs roughly 800 MiB per two-hour NOM-HK file; a full day of files in one batch exhausts an 8 GiB container partway through. Parsing per file bounds that at one file’s worth.

Parsed files are copied onto a preallocated packet axis and released one at a time, rather than held and handed to xr.concat, which allocates its result while every input is still referenced.

A file holding no packets of apid is skipped with a warning.

Parameters:

  • packet_files (list[PathLike | str]) – List of filepaths to packet files.

  • packet_definition (str | PathLike | space_packet_parser.xtce.definitions.XtcePacketDefinition) – Path to the XTCE packet definition file, or an already loaded definition.

  • apid (int) – Application Process Identifier to filter for.

  • max_workers (int, optional) – Files to parse concurrently, in worker processes. Default 1 (in-process). Parsing is CPU-bound single-threaded Python, so concurrency is the only way to use a container sized for more than one vCPU. Peak memory rises by one in-flight parse per worker (about 800 MiB for a two-hour NOM-HK file), which is the practical limit rather than the core count. Must be at least 1. Workers start with fork, which Windows does not provide, so the caller needs no if __name__ == "__main__": guard; a caller that already runs threads risks a worker inheriting a held lock (Python 3.12+ warns with DeprecationWarning). A worker reads an S3 path with the client the path carries, or with the default S3 client for a URI string, over its own connections.

  • **generator_kwargs – Additional keyword arguments passed to the packet generator.

Returns:

xarray Dataset containing parsed packet data, with files concatenated in the order given.

Return type:

xr.Dataset

Raises:

  • ValueError – If no file holds any packet of apid, if a file yields an APID other than apid, if the files do not all carry the same set of variables, if max_workers is below 1, or if max_workers above 1 with more than one file is combined with a platform that has no fork start method.

  • RuntimeError – If a worker process exits without reporting a result, such as when it is killed for running out of memory.

  • Exception – Any exception raised while parsing a file in a worker process is re-raised in the caller, with the worker’s traceback attached as a note.