Development guide¶

Quick reference for day-to-day canVODpy development.

Initial setup¶

git clone https://github.com/TUW-GEO/canvodpy.git
cd canvodpy
git submodule update --init --recursive   # test data + demo data
uv sync                                    # install all packages (editable)
just hooks                                 # install pre-commit hooks
just test                                  # verify everything works

Prerequisites¶

Two tools must be installed outside uv:

just check-dev-tools   # verify both are present

Full installation guide

Configuration management¶

canVODpy uses three YAML files in config/:

All three sections live in a single config/canvod-settings.yaml. Template: config/canvod-settings.yaml.example, a copy of the template the package ships (packages/canvod-config/src/canvod/config/templates/); edit that one and copy it.

canvodpy config init                # scaffold canvod-settings.yaml from template
# edit config/canvod-settings.yaml (or: canvodpy config init --interactive)
canvodpy config validate            # check for errors
canvodpy config show                # print resolved config

just config-validate    # after editing
just config-show        # inspect current values
canvodpy doctor         # environment + config diagnostics when something's off

Testing¶

just test                            # all tests
just test-package canvod-readers     # specific package
just test-coverage                   # with HTML coverage report
just test-network                    # tests that reach the download servers

Tests live in packages/<pkg>/tests/. Integration tests are marked @pytest.mark.integration and excluded from the default run.

Network tests are marked @pytest.mark.network and skipped unless just test-network runs them; GitHub CI never does. They check every product of the registry on the FTP servers, download the COD final orbits and clocks of 2025-001 and compare them with the test data, and run a day of the test data that downloads its own orbits and clocks. Run them before a release and after changing the product registry. NASA CDDIS is checked only when the settings hold credentials.nasa_earthdata_acc_mail.

Code Quality¶

just check          # lint + format + type-check (run before committing)
just check-lint     # ruff linting only
just check-format   # ruff formatting only

Type checking (ty)¶

ty check must pass with zero diagnostics. It blocks git push (local pre-push hook) and runs again in CI (type_consistency job). There's no budget or ratchet; if you introduce a diagnostic, fix it before pushing.

Most real diagnostics mean the annotation is actually wrong: fix the annotation, not the checker. Suppress with a targeted comment only when the code is correct and a limitation of ty is the real issue (for example, a runtime-built enum.StrEnum, a deliberately dynamic attribute set on a function object, a third-party stub that's too strict for a legitimately partial dict):

_excepthook._logger_holder = logger_holder  # ty: ignore[unresolved-attribute]

Always include the specific rule name in brackets (uv run ty check prints it in the diagnostic, for example error[invalid-return-type]). A bare # type: ignore (mypy syntax) does not suppress ty diagnostics, since ty matches on its own rule codes.

Logging: add it generously¶

canvodpy runs unattended on remote machines. When something goes wrong there, whatever you logged is the only evidence there is: there's no attaching a debugger to a cron job after the fact. Log more than feels necessary, especially around: file I/O, external calls (downloads, Icechunk writes), anything with a fallback/degraded path, and any place a silent skip could hide data loss. See docs/guides/diagnostics.md for the full picture (the log files, run_id, the failure events, stage_timer, the performance dashboard). This section is the short version: how to add a log line.

Which logger to use: every module, in every package, gets its logger directly from structlog:

import structlog

log = structlog.get_logger(__name__)

Only canvodpy configures logging; the canvod-* packages never do. configure_logging() installs structlog's processor chain globally, so any structlog.get_logger() call anywhere in the process, regardless of which package it's in, automatically gets routed through the two-track logging setup (human/agent files) with run_id injected, with zero per-call-site wiring. Never reach for print() or the standard library logging directly.

Event style: first positional argument is a short snake_case event name, everything else is a structured keyword, not string interpolation:

# Good
log.info("rinex_preprocessing_started", file=str(rnx_file.name), site=site_name)
log.warning("sids_dropped_no_ephemeris", count=len(dropped), sids=sorted(dropped))

# Avoid: loses structure, can't be grepped/filtered on a field
log.info(f"Started preprocessing {rnx_file.name} for {site_name}")

Timing: use stage_timer()/timed_stage() from canvod.utils.logging rather than hand-rolling t0 = time.perf_counter(); ...; duration = ...: it emits the canonical stage_timing event the performance dashboard reads, and still emits (with status="error") if the block raises.

from canvod.utils.logging import stage_timer

with stage_timer("icechunk.write", group=group_name, size_mb=size_mb):
    to_icechunk(dataset, session, group=group_name)

When one block has several sequential checkpoints, each a dashboard stage of its own, measure them yourself and emit each with record_stage_timing(stage, seconds, ...) (see _preprocess_file in processor.py). Checkpoints that are not dashboard stages belong in one specific, well-named event with their own rich context (see _preprocess_aux_data_with_hermite in processor.py), which is more useful than forcing everything through one generic name.

Only the main process adds stages to the run_summary; stages timed in a worker process reach the log and the dashboard only.

Colours in the terminal¶

The CLI's colours come from one module, canvodpy.cli.theme; its docstring gives the rules and the reasons. In short: structure (commands, names, descriptions, borders) uses the brand's greens by lightness, and status uses three added colours, each always with its symbol. There is no red, and the colours keep apart with red-green colour blindness.

In a command, print through the shared console and use the roles, not colour names:

from canvodpy.cli.theme import console

console.print(f"[canvod.ok]✓ Wrote {path}[/canvod.ok]")
console.print("[canvod.warning]⚠️ No recipe for canopy_01[/canvod.warning]")
console.print(f"[canvod.error]❌ File not found: {path}[/canvod.error]")

test_cli_theme.py fails on a colour name ([red], style="green") or an unknown role in a CLI module, and checks each colour's contrast to the background and the distance between the colours as seen with red-green colour blindness. To change a colour, change PALETTES in theme.py and keep those tests passing for both backgrounds.

The logo of the banner (canvodpy.cli.banner) is drawn from the SVG in docs/assets/logo by scripts/render_logo.py. When the logo changes, draw its frames again:

uv run scripts/render_logo.py docs/assets/logo/canvodpy_logo_dark_mode.svg \
    canvodpy/src/canvodpy/cli/logo_frames.txt.gz

Keeping your fork in sync¶

If you are working from a fork (rather than a direct clone), periodically pull updates from the upstream repository:

git checkout main
git fetch upstream
git merge upstream/main
git push origin main

# Then update your feature branch:
git checkout feature/my-feature
git rebase main

If upstream is not configured, add it once:

git remote add upstream git@github.com:TUW-GEO/canvodpy.git

Detailed fork sync guide

Pre-commit hooks¶

just hooks installs Git hooks that run automatically on every commit. If any hook fails, the commit is rejected: your changes stay staged but no commit is created.

What runs¶

When your commit is rejected¶

# Most common: ruff auto-fixed your code. Stage the fixes and retry:
git add -u && git commit -m "feat(readers): your message"

# If commitizen rejects your message, use the correct format:
git commit -m "type(scope): description"
# types: feat, fix, docs, refactor, test, chore, perf, ci
# scopes: readers, aux, grids, vod, store, viz, utils, naming, ops, docs, ci, deps

# If uv-lock is out of date:
uv sync && git add uv.lock && git commit -m "feat(readers): your message"

Full troubleshooting guide

Contributing workflow¶

git checkout -b feature/my-feature
# … make changes in packages/<pkg>/src/ …
# … add tests in packages/<pkg>/tests/ …
just check && just test
git add packages/<pkg>/src/... packages/<pkg>/tests/...
git commit -m "feat(readers): add RINEX 4.0 support"
git push origin feature/my-feature
# → open pull request on GitHub

Conventional commit scopes¶

readers · aux · grids · vod · store · viz · utils · naming · ops · docs · ci · deps

Continuous Integration¶

Every push and PR triggers GitHub Actions workflows:

Coverage reports are posted as PR comments and tracked on Coveralls over time. To run coverage locally:

just test-coverage    # generates HTML report

Private or local add-on packages¶

You may want to develop an add-on package alongside canVODpy that is not part of the public repository, for example a private algorithm, a site-specific extension, or an experimental module you are not yet ready to publish.

The recommended pattern keeps the public workspace and lock file clean while letting you use the package locally without friction.

The pattern¶

1. Place the package inside packages/

packages/
  canvod-myprivatepackage/
    pyproject.toml
    src/
      canvod/myprivatepackage/

2. Add it to .gitignore (so it is never accidentally committed)

packages/canvod-myprivatepackage

3. Exclude it from the uv workspace in the root pyproject.toml

[tool.uv.workspace]
members  = ["packages/*", "canvodpy"]
exclude  = ["packages/canvod-myprivatepackage"]

This keeps uv.lock identical for everyone else. CI never sees the directory, so it resolves the same 240 packages with or without your local copy.

4. Install locally with uv pip install

uv pip install -e packages/canvod-myprivatepackage/

uv pip install installs directly into the virtual environment. It does not modify pyproject.toml or uv.lock. After a fresh uv sync, re-run the command to restore your local install.

Upgrade path¶

When the package is ready to become public:

1. Remove it from .gitignore and from the exclude list.
2. Commit the package directory.
3. Run uv lock and commit the updated uv.lock.

It becomes a first-class workspace member and appears in the lock for everyone.

Why not git submodules?¶

You could also host the private package in its own repository and add it as a git submodule. actions/checkout@v4 does not initialize submodules by default, so CI would still see an empty directory. The trade-off is that the uv.lock would diverge between users who have initialized the submodule and those who have not, requiring everyone to keep the exclude in place anyway. For a single-developer private add-on the simpler approach is preferred.

All just commands¶

just                     # list all commands
just check               # lint + format + type-check
just test                # run all tests
just sync                # install/update dependencies
just clean               # remove build artifacts
just hooks               # install pre-commit hooks
canvodpy config init     # scaffold canvod-settings.yaml from template
just config-init-interactive  # scaffold canvod-settings.yaml via guided wizard
just config-validate     # validate configuration
just config-show         # view resolved configuration
just doctor              # environment + config diagnostics
just store-list          # list every site's gnss/vod stores + status
just store-info SITE     # branches/groups/stats for one site's store
just store-log SITE      # commit history graph for one site's store
just store-migrate-metadata SITE  # move an older store to repository metadata
just store-migrate-layout SITE    # move a store written by appending to day blocks
just docs                # preview documentation (opens the browser)
just docs-lint           # language check of the docs (Vale, Google style)
just docs-links          # link check of the built docs (lychee)
just build-all           # build all packages
just deps-report         # dependency metrics report
just deps-mermaid        # mermaid dependency graph

Troubleshooting¶

uv sync --all-extras

git ls-tree HEAD packages/   # shows only committed packages
ls packages/                 # shows all local packages (including untracked)