libera_utils.logutil#

Logging utilities

Functions

configure_static_logging(config_file)

Configure logging based on a static logging configuration yaml file.

configure_task_logging(task_id, *[, ...])

Configure logging for a specific task (e.g. a processing algorithm).

flush_cloudwatch_logs()

Force flush of all cloudwatch logging handlers.

Classes

JsonLogFormatter(*args[, ...])

Altered version of the CloudWatchLogFormatter provided in the watchtower library

class libera_utils.logutil.JsonLogFormatter(*args, add_log_record_attrs: Tuple[str, ...] | None = None, add_asctime: bool = True, **kwargs)#

Altered version of the CloudWatchLogFormatter provided in the watchtower library

Methods

converter([seconds])

Convert seconds since the Epoch to a time tuple expressing local time.

format(record)

Format log message to a string

formatException(ei)

Format and return the specified exception information as a string.

formatStack(stack_info)

This method is provided as an extension point for specialized formatting of stack information.

formatTime(record[, datefmt])

Return the creation time of the specified LogRecord as formatted text.

usesTime()

Check if the format uses the creation time of the record.

formatMessage

format(record: LogRecord) str#

Format log message to a string

Parameters:

record (logging.LogRecord) – Log record object containing the logged message, which may be a dict (Mapping) or a string

libera_utils.logutil._json_serialize_default(o: Any) str#

A standard ‘default’ json serializer function.

  • Serializes datetime objects using their .isoformat() method.

  • Serializes all other objects using repr().

libera_utils.logutil.configure_static_logging(config_file: str | Path | S3Path)#

Configure logging based on a static logging configuration yaml file.

The yaml is interpreted as a dict configuration. There is no ability to customize this logging configuration at runtime.

Parameters:

config_file (cloudpathlib.anypath.AnyPath or str) – Location of config file.

See also

configure_task_logging

Runtime modifiable logging configuration.

libera_utils.logutil.configure_task_logging(task_id: str, *, limit_debug_loggers: Iterable[str] | str | None = None, console_log_level: str | int = 20, console_log_json: bool = False, log_dir: str | Path | S3Path | None = None, cloudwatch_log_group: str | None = None)#

Configure logging for a specific task (e.g. a processing algorithm).

File-based logging is always done at the DEBUG level. Watchtower-based cloudwatch logging is always done at the DEBUG level. Console logging level defaults to INFO but can be set with console_log_level.

Examples

Example 1: The following will configure DEBUG console-only logging for anything in your script but all other loggers will be limited to INFO level.

`python configure_task_logging("my-script", limit_debug_loggers=("__main__",), console_log_level=logging.DEBUG) `

Example 2: This will allow all debug messages through from all loggers and sets up file-based logging and a custom cloudwatch log group. Also console messages will be logged in serialized JSON.

```python configure_task_logging(“my-script”,

console_log_level=logging.DEBUG, log_dir=Path(“/tmp/my-script”), console_log_json=True, cloudwatch_log_group=”custom-log-group”)

```

Parameters:
  • task_id (str) – Unique identifier by which to name the log file and cloudwatch log stream.

  • limit_debug_loggers (Optional[Union[Iterable[str] | str]]) – A list of logger name prefixes from which you want to allow debug messages (blocks debug from all others). For example, if you are working on a package called my_app and using module level logging, all your loggers will be named like my_app.module_name.submodule_name. By setting this to (my_app,), all loggers that are named my_app.* will propagate debug messages while preventing spammy debug messages from installed libraries like boto3. If this is empty or None, all debug messages will propagate. To use this in scripts, either leave it unset or use limit_debug_loggers=(“__main__,).

  • console_log_level (str or int, Optional) – Log level for console logging. If not specified, defaults to INFO

  • console_log_json (bool, Optional) – If True, console logs will be JSON formatted. This is suitable for setting up loggers in AWS services that are automatically monitored by cloudwatch on stdout and stderr (e.g. Lambda or Batch)

  • log_dir (str or Path or S3Path, Optional) – Log directory, which may be a local or S3Path. Default is None and results in no file-based logging.

  • cloudwatch_log_group (str, Optional) – Override optional environment variable log group name. Default is None and will result in falling back to the LIBERA_LOG_GROUP environment variable. If that is not set, no cloudwatch JSON logging will be configured.

Notes

Even in the absence of cloudwatch JSON logging, all stdout/stderr messages generated by a Lambda will be logged to CloudWatch as string messages. Embedded JSON strings in log message text can still be queried in CloudWatch.

See also

configure_static_logging

Static logging configuration based on yaml file.

libera_utils.logutil.flush_cloudwatch_logs()#

Force flush of all cloudwatch logging handlers.

If you are missing the last few log messages in a log stream, this may help get those logs ingested before the process shuts down the logging system.

Return type:

None