Preprocessing pipeline¶

Before interpolation, auxiliary data must be restructured to match the (epoch × sid) layout of the observation datasets. This pipeline bridges the sv (satellite) indexing of SP3/CLK files and the sid (signal ID) indexing of the observations.

In a run, AuxDataPipeline calls prep_aux_ds() with the signal IDs kept by the sids: settings, so the orbits and clocks carry the same signals as the observations before they are interpolated.

The dimension mismatch problem¶

sp3_data.dims
# {'epoch': 96, 'sv': 32}

rinex_data.dims
# {'epoch': 2880, 'sid': 384}

Satellite position is frequency-independent (all signals leave the same antenna), so the sv \(\rightarrow\) sid mapping simply replicates each position across all signal IDs for that satellite.

Pipeline functions¶

Step 1: map_aux_sv_to_sid()¶

Each satellite position is replicated across all its signal IDs:

from canvod.auxiliary.preprocessing import map_aux_sv_to_sid

sp3_sid = map_aux_sv_to_sid(sp3_data)

# G01 position is identical for every signal ID of G01
sp3_sid["X"].sel(sid="G01|L1|C")  # 12 345 678.9 m
sp3_sid["X"].sel(sid="G01|L2|W")  # 12 345 678.9 m  (same)
sp3_sid["X"].sel(sid="G01|L5|I")  # 12 345 678.9 m  (same)

Signal IDs generated for GPS G01 (28 total):

G01|L1|C  G01|L1|L  G01|L1|M  G01|L1|N  G01|L1|P  G01|L1|S  G01|L1|W
G01|L1|X  G01|L1|Y  G01|L1|p  G01|L1|u
G01|L2|C  G01|L2|D  G01|L2|L  G01|L2|M  G01|L2|N  G01|L2|P  G01|L2|S
G01|L2|W  G01|L2|X  G01|L2|Y  G01|L2|l  G01|L2|p  G01|L2|u
G01|L5|I  G01|L5|Q  G01|L5|X  G01|L5|u

Signal ID format: "{SV}|{BAND}|{CODE}", for example "G01|L1|C", "E08|E5a|Q". Uppercase codes are RINEX 3 attributes; the lowercase codes are the RINEX 2 tracking-code markers for signals whose code RINEX 2 does not record.

Step 2: pad_to_global_sid()¶

Pads the Dataset to include all possible Signal IDs across all supported constellations: every \(\text{SV} \times \text{band} \times \text{code}\) in the constellation BAND_CODES (4 429 at the time of writing). Signal IDs outside that set are dropped. Required for Icechunk storage, where sequentially appended datasets must share the same coordinate space.

from canvod.auxiliary.preprocessing import pad_to_global_sid

sp3_global = pad_to_global_sid(sp3_sid)
sp3_global.sizes["sid"]  # 4429

Added SIDs are filled with NaN.

Step 3: normalize_sid_dtype()¶

Converts the sid coordinate to object dtype for Zarr/Icechunk compatibility. Fixed-length Unicode string types (<U12, etc.) cause dtype conflicts when datasets from different files are concatenated and appended.

from canvod.auxiliary.preprocessing import normalize_sid_dtype

ds = normalize_sid_dtype(ds)
ds["sid"].dtype  # dtype('O')

Step 4: strip_fillvalue()¶

Removes _FillValue attributes that conflict with Icechunk's internal missing-data handling. NaN is the standard missing-value marker throughout the pipeline.

from canvod.auxiliary.preprocessing import strip_fillvalue

ds = strip_fillvalue(ds)
# No variable in ds.data_vars has a "_FillValue" encoding key

Scientific note¶

Replicating satellite positions across signal IDs is scientifically valid: