Upgrading to v1.0.0¶

This page is for everyone who used canVODpy 0.x. Do the four steps before your first run with v1.0.0; the rest of the page lists what changed, what is deprecated and what is new.

Before your first run¶

1. Check the settings file¶

canvodpy config validate

It names every setting that is missing or no longer allowed, with its dotted key. A settings file from 0.x typically needs:

Settings that are deprecated still work and print a warning that says what to do; they are listed below.

2. Back up your stores¶

The migration commits to the stores; earlier snapshots stay readable, but keep a copy until you've checked the result. A store is a folder: copy it (cp -a, rsync -a).

3. Migrate each site's stores¶

One command per site:

canvodpy store migrate SITE

It shows what it will do, asks once (--yes skips the question), and then takes these steps, skipping each one that is done already:

1. The folders. A store's folder is named by its kind, gnss or vod, optionally with a suffix (vod-recompute, Store folders). The GNSS store's old folder rinex moves to gnss. Any other folder whose name isn't its kind needs its new name: --rename vod_2024=vod-2024. The store id is taken from the new name. A site that has both rinex and gnss is refused: decide which store to keep. The next steps migrate the stores your settings name (gnss_store_name, vod_store_name). Another store, such as vod-2024, is only moved, and the command says so: to migrate it, set vod_store_name: vod-2024 and run the command again.
2. The GNSS store, then the VOD store, each with the steps of migrate-metadata and migrate-layout (described below).
3. The clock. A GNSS store written with CLK files holds a clock variable. v1.0.0 reads no CLK files by default, and VOD doesn't use the clock: the variable is removed and the store records fetch_clock: false. The VOD store records the GNSS store's new content-settings hash, so VOD runs go on.

If a step stops, fix the cause and run the command again. migrate-metadata and migrate-layout run single steps of one store (--store gnss or --store vod):

• migrate-metadata moves the store to repository metadata: its description, a store id and UUID, the time scale of its epochs, and its content settings, the settings that shape its data and that every later run must match (Content settings). Metadata the store lacks, such as the publisher and license, comes from the settings file, marked (from the settings). A store that is already migrated is left as it is.
• migrate-layout writes the groups again as day blocks, the layout v1.0.0 runs write (Moving a store to day blocks). An epoch stored twice keeps the entry written last.

Both show what they change and ask first. Each one is a single commit. The details are in Move an older store to the current form.

Settings the store doesn't record. migrate-metadata takes the content settings from the settings recorded in the store's first commit. A value that commit doesn't record isn't guessed: the command stops, lists it, and you give it with --gnss-set (or --vod-set for the VOD store), as your settings file had it when the store was written. For a GNSS store from canVODpy 0.4.0 these are typically:

canvodpy store migrate SITE \
  --gnss-set processing.params.store_sbf_metadata=true \
  --gnss-set processing.params.rinex_v3_parser=validated \
  --gnss-set "sites.SITE.receivers.reference_01.paired_canopies=[canopy_01]"

Some settings are read from the data instead, because canVODpy 0.x recorded settings its runs didn't follow: the preprocessing (none, unless the data show a step) and whether the stores hold r, delta_snr and radial_diff. canVODpy 0.x always wrote delta_snr to the VOD store, so a migrated store records store_delta_snr: true: set it in your settings file too.

Stores from SBF or NMEA files. canVODpy 0.x stored their epochs in UTC, 18 s behind the GPS time v1.0.0 uses for every epoch. The migration refuses these stores before it writes anything, and so does the VOD store computed from them: rebuild them from the raw files with canvodpy run. Stores from RINEX files hold GPS time and migrate as they are.

Receivers with a naming recipe. A GNSS store records each receiver's naming recipe and the sha256 of the recipe file, and every run compares them with the settings. Stores of canVODpy 0.x record the recipe, if the receiver had one, without its sha256; a receiver that used the naming block and now has a recipe records none. The migration doesn't add them: it can't tell whether today's recipe file is the one the store was built with. For each receiver with a naming recipe, check that its recipe file is the one the store was built with, then record it:

canvodpy store amend SITE --store gnss --update-recipe RECEIVER \
    --reason "recipe of the stored days, recorded after the migration"

Without this, the first run stops and prints the command. Sites whose files have canonical canVOD names have no recipes and skip this.

4. Check the result¶

canvodpy store info SITE            # content settings: set, with their hash
canvodpy run --site SITE --start 2025001 --end 2025001

A run compares your settings with the store's content settings before it reads a file, and stops if they differ, listing each setting with both values. Change your settings to the store's values, or write a settings file from the store with canvodpy config init --from-store <store>. A dry run (--dry-run) lists the files a run would process, but doesn't compare the settings.

What changed¶

Deprecated¶

Deprecated code and settings still work in every v1.x release, warn when used, and are removed in v2.0.0 (ADR 0005). The rest of the documentation leaves them out.

Ways to run¶

All of these are replaced by one entry point: canvodpy run, or canvodpy.Site(<site>).pipeline() in Python (see the command line and Python).

Deprecated settings¶

Python API¶

New in v1.0.0¶

• Stores on S3, experimental: runs write GNSS and VOD stores to S3-compatible object storage, tested against S3 emulators only (Stores on S3).
• Store metadata after DataCite and ACDD, checked against the standards, and a STAC catalog of all stores (canvodpy run --write-catalog; Store metadata).
• Settings from a link: --config-dir and CANVOD_CONFIG_DIR take the raw GitHub link of a settings file; naming recipes are read from the same repository (Settings from a GitHub repository).
• Join or reproduce a store: canvodpy config init --from-store writes the settings file of a store (Settings file).
• Run history and dashboard: every run records itself, its stage timings and resource use (Diagnostics and monitoring).
• Data check: canvodpy config validate checks the receiver data the way a run finds it, and runs warn about files they don't process and about incomplete receiver-days.
• Memory budget and units of days: max_memory_gb sizes the workers and the units of days a run commits at once, and a run reads the next unit while it computes VOD (Parallel processing).
• A complete command line: canvodpy run, vod, vod-reconcile, config, store, stac, doctor and dashboard (Command line).
• Plugins: your own readers and VOD calculators from installed packages (Your own readers and VOD calculators).
• Comparing stores within fixed tolerances, and canvodpy config explain.
• Stores that don't record the computer: processing.metadata.record_machine: false leaves the operating system, architecture, CPU count, memory and free disk out of every commit, for stores that are published (Settings). It is true by default, so nothing changes for you.
• Demo data: canvodpy.demo_data() downloads the demo day; the notebooks use it.

The full list is in the changelog.