Skip to content

canvod.ops API reference

Configurable preprocessing pipeline for GNSS observation datasets.

Pipeline

canvod.ops: Preprocessing operations pipeline for GNSS VOD data.

Op

Bases: ABC

Abstract base class for a preprocessing operation.

Each Op is a callable carrying config set at construction time. At call time it is a pure Dataset -> (Dataset, OpResult) transform.

settings abstractmethod property

The configured parameters, as the preprocessing record holds them.

Known before the operation runs, so a run can compare them with the records of a store before it processes any data.

OpResult dataclass

Immutable record of a single preprocessing operation.

step()

This operation's entry in the preprocessing record.

settings are the configured parameters, result what the operation measured or derived (see canvod.config.models.preprocessing_record).

Pipeline

Ordered chain of :class:~canvod.ops.base.Op instances.

__len__()

Number of operations.

settings()

Operations and settings this pipeline records, before it runs.

Same layout as canvod.config.models.preprocessing_settings gives for a stored record: [{"op", "settings"}, ...] in run order.

add(op)

Append an operation and return self for chaining.

PipelineResult dataclass

Aggregated result from running a full pipeline.

record()

Preprocessing record (JSON) of the operations, in the order they ran.

to_metadata_dict()

Serialize to a dict suitable for ds.attrs.

build_default_pipeline(config=None)

Build the pipeline that processing.preprocessing describes.

Temporal aggregation runs first, then the grid cell assignment; each only if its section is set and enabled.

Parameters:

Name Type Description Default
config PreprocessingConfig | None

Explicit config. If None, reads processing.preprocessing via load_config(); an unset section gives an empty pipeline.

 None

Returns:

Type Description
Pipeline

Ready-to-call pipeline (empty if nothing is set).

preprocess_files(parts, config)

Apply processing.preprocessing to the files of one receiver and day.

The files are joined along epoch before the operations run, so a time bin that spans two files is aggregated from the observations of both. The result is split back into one dataset per file: each bin goes to the file that holds its earliest observation. Every dataset keeps its own attributes (e.g. the file hash) and carries the preprocessing record of the day (see canvod.config.models.preprocessing_record) under :data:~canvod.config.models.PREPROCESSING_ATTR, also when nothing is applied. The record is only carried in memory: the stores move it into the preprocessing column of their log book and never write it into the data.

Parameters:

Name Type Description Default
parts Sequence[tuple[K, Dataset]]

(key, dataset) per file, e.g. (path, dataset); datasets have (epoch, sid) dimensions.

 required
config PreprocessingConfig | None

The processing.preprocessing section; None applies nothing.

 required

Returns:

Type Description
list[tuple[K, Dataset]]

(key, dataset) per file, in time order. A file whose observations all fall into bins of an earlier file is left out.

Temporal

Temporal aggregation operation.

TemporalAggregate

Bases: Op

Aggregate an (epoch, sid) dataset to regular time bins.

Each bin starts at a multiple of freq counted from 00:00 of the day and is labeled with its start. Every variable along epoch is aggregated per signal; the others pass unchanged. Missing values (NaN, -1 for integers) are ignored; a bin without any value stays missing. Floating-point variables take the method of their values in the bin. Integer variables (flags and codes such as LLI and SSI) cannot be combined: they take one of their values in the bin, the first, last or middle one in time order (integer_method). The azimuth phi (radians) is aggregated as an angle, so a satellite crossing north (0 rad) does not average to south: each value is taken relative to the first value of its bin.

The result has the variables, dimensions, data types, attributes and encodings of the input. The input sampling and the bin length (in seconds) are reported in the result of the returned :class:~canvod.ops.base.OpResult, which the stores keep in their log book.

Parameters:

Name Type Description Default
freq str

Bin length as a pandas offset alias (e.g. "1min", "30s").

 '1min'
method str

Aggregation method of floating-point variables: "mean" or "median".

 'mean'
integer_method str

Value of the bin integer variables take: "first", "last" or "middle".

 required

temporal_aggregate(ds, freq='1min', method='mean', *, integer_method)

Convenience function: temporally aggregate a dataset.

Parameters:

Name Type Description Default
ds Dataset

Input dataset with (epoch, sid) dimensions.

 required
freq str

Target frequency.

 '1min'
method str

"mean" or "median", for floating-point variables.

 'mean'
integer_method str

"first", "last" or "middle", for integer variables.

 required

Returns:

Type Description
Dataset

Aggregated dataset.

Grid

Grid cell assignment operation.

GridAssignment

Bases: Op

Assign each (epoch, sid) observation to a grid cell.

Parameters:

Name Type Description Default
grid_type str

Grid builder name (e.g. "equal_area").

 'equal_area'
angular_resolution float

Grid resolution in degrees.

 2.0

grid_assign(ds, grid_type='equal_area', angular_resolution=2.0)

Convenience function: assign grid cells to a dataset.

Parameters:

Name Type Description Default
ds Dataset

Input dataset with phi(epoch, sid) and theta(epoch, sid) coords.

 required
grid_type str

Grid builder name.

 'equal_area'
angular_resolution float

Resolution in degrees.

 2.0

Returns:

Type Description
Dataset

Dataset with cell_id_<grid_name>(epoch, sid) variable added.