`canvod.store_metadata` API reference¶

Rich provenance metadata for Icechunk stores, aligned with DataCite 4.5, ACDD 1.3, and STAC 1.1.

Package¶

canvod-store-metadata: store-level provenance for Icechunk stores.

This package manages store metadata: identity, creator, environment, software provenance, and standards compliance (DataCite/ACDD/STAC).

Not to be confused with the file registry in canvod.store ({group}/metadata/table), which tracks individual ingested files.

`StoreMetadata` ¶

Bases: BaseModel

Root metadata model composing all 14 sections.

`to_root_attrs()` ¶

Serialize to a JSON-compatible dict for Zarr root attrs.

`StoreLocation` ¶

Bases: ABC

Where a store's repository lives.

`needs_commit_lock` `abstractmethod` `property` ¶

Whether commits here must hold :func:~.io.commit_lock.

`local_path` `abstractmethod` `property` ¶

The directory on this machine, or None for remote storage.

`access_hint` `abstractmethod` `property` ¶

What to check when the storage can't be reached.

`storage()` `abstractmethod` ¶

The Icechunk storage for this location.

`truediv(relative)` `abstractmethod` ¶

The location relative below this one, of the same kind.

Raises:

Type	Description
`ValueError`	If `relative` is an absolute path.

`LocalStoreLocation` `dataclass` ¶

Bases: StoreLocation

A store in a directory on a local file system.

`needs_commit_lock` `property` ¶

Yes: Icechunk on a local file system can lose one of two concurrent commits without an error.

`S3StoreLocation` `dataclass` ¶

Bases: StoreLocation

A store under a prefix in an S3 bucket.

Credentials come only from the environment (the AWS_* variables, AWS_PROFILE) or the shared AWS files (~/.aws/credentials, ~/.aws/config), as do region and endpoint (AWS_REGION, AWS_ENDPOINT_URL): never from the settings or the store metadata, and they are never logged. CANVOD_S3_PATH_STYLE=1 puts the bucket in the request path, for S3 services that expect it.

`needs_commit_lock` `property` ¶

No: Icechunk's conditional writes on S3 keep two commits from overwriting each other, and there is no local directory to hold a lock file.

`storage()` ¶

The Icechunk storage, configured from the environment.

Raises:

Type	Description
`ValueError`	If `CANVOD_S3_PATH_STYLE` is set to something other than yes or no.

`truediv(relative)` ¶

The location relative below this one, .. resolved.

S3 keys are plain names: S3 itself doesn't resolve .., so a catalog item's relative href is resolved here.

Raises:

Type	Description
`ValueError`	If `relative` is an absolute path, or leads to the bucket's root or out of the bucket.

`collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None)` ¶

Collect all metadata sections into a StoreMetadata object.

This is the main entry point for metadata collection. It gives a new store its id, UUID and catalog link.

Raises:

Type	Description
`ValueError`	If the site name gives no valid store id, the naming authority is missing or not a reverse DNS name, or the site's store description, the publisher or the license is missing.

`write_metadata(store_path, metadata, branch='main')` ¶

Write a new store's metadata.

The repository metadata (fixed and amendable fields, ADR 0014) is written once; a first commit writes its root copy and the snapshot summary, with metadata.config as the settings in the commit metadata.

Returns:

Type	Description
`str`	Snapshot ID from the commit.

Raises:

Type	Description
`StoreCreatedByAnotherRunError`	If the store already has repository metadata, or another run committed its creation first. A store on S3 has no commit lock, so two runs may both find it without metadata; the creation commit does not rebase, so only one of them creates it (#358).
`ValueError`	If `metadata` lacks a fixed field (build it with `collect_metadata`), or a field breaks a rule of its standard (e.g. a license outside SPDX).

`read_metadata(store_path, branch='main', *, snapshot_id=None)` ¶

Read the metadata of a snapshot as it was when it was committed.

One :class:StoreMetadata from the snapshot's root copy, snapshot summary and commit metadata. A snapshot written before the store kept repository metadata is read from its old canvod_metadata attribute and marked written_before_migration.

Parameters:

Name	Type	Description	Default
`branch`	`str`	Read the tip of this branch.	`'main'`
`snapshot_id`	`str \| None`	Read this snapshot instead of a branch tip (e.g. one a release pins).	`None`

Raises:

Type	Description
`KeyError`	If no metadata found in store.

`amend_metadata(store_path, changes, *, reason, who, config=None, receiver_content=None)` ¶

Change amendable fields of a store's repository metadata.

Parameters:

Name	Type	Description	Default
`changes`	`dict[str, Any]`	New values by dotted field path, e.g. `{"identity.title": "..."}`. A new receiver or VOD analysis is added by its whole entry, e.g. `{"instruments.receivers.canopy_02": {"type": "canopy", "directory": "..."}}`.	required
`reason`	`str`	Why; recorded with each changed field.	required
`who`	`str`	Who changes the fields, e.g. `"NFB"`; recorded with each changed field.	required
`config`	`Any`	The settings recorded in the commit metadata (see :func:`~canvod.store_metadata.io.commit_with_rebase`).	`None`
`receiver_content`	`dict[str, dict[str, Any]] \| None`	The content settings of each receiver added (`type`, `reader_format`, `paired_canopies`, ADR 0019), by name. Needed to add a receiver to a store that has content settings: they are added with it, and never changed later.	`None`

Returns:

Type	Description
`str`	Snapshot ID of the commit that carries the new root copy.

Raises:

Type	Description
`ValueError`	If there is no reason, no `who` or no change, a field is fixed, computed or unknown, a value is invalid, the store has no repository metadata, or a receiver is added to a store with content settings without its `receiver_content`.

`migrate_metadata(store_path, *, who, naming_authority=None, values=None, variable_attrs=None, branch='main', set_option='--set')` ¶

Move a store's metadata to repository metadata (ADR 0014).

Parameters:

Name	Type	Description	Default
`who`	`str`	Who migrates the store, e.g. `"NFB"`; recorded in the migration record.	required
`naming_authority`	`str \| None`	The naming authority (reverse DNS) for a store whose metadata has none; otherwise the store's own is kept.	`None`
`values`	`Mapping[str, Any] \| None`	Amendable fields to set while migrating, by dotted path, e.g. `{"identity.description": "..."}`: the fields an old store lacks to pass the standards checks.	`None`
`variable_attrs`	`Mapping[str, Mapping[str, Any]] \| None`	New attributes by variable name, e.g. `{"VOD": VOD_ATTRS}`. Every array of that name gets them in place of its own; xarray's encoding attributes (`_FillValue`, `coordinates`) are kept. The changed arrays are listed in the migration record.	`None`
`branch`	`str`	The branch whose metadata is migrated.	`'main'`
`set_option`	`str`	The command-line option an error names for giving a missing field: `--gnss-set` or `--vod-set` under `canvodpy store migrate`.	`'--set'`

Returns:

Type	Description
`str`	Snapshot ID of the migration commit.

Raises:

Type	Description
`ValueError`	If `who` is blank, the store is already migrated, holds no store metadata, has no valid naming authority and none is given, a value names a field that cannot be amended, or the migrated store would fail the standards checks (the message lists the failures). Nothing is written then.

`metadata_exists(store_path, branch='main')` ¶

Check if the store has repository metadata.

branch is not used: repository metadata is on no branch. It stays for callers that pass it.

`open_repository(location, *, create=False, config=None, expected_uuid=None)` ¶

Open the repository of the store at location.

Opening never creates a repository, and creating never opens one, so a mistyped location fails instead of leaving an empty store behind.

Parameters:

Name	Type	Description	Default
`create`	`bool`	Create a new repository instead of opening one.	`False`
`config`	`RepositoryConfig \| None`	The repository configuration (compression, caching, concurrency). `None` creates a store with the defaults of the `processing.icechunk` settings (:func:`repository_config`), so it matches a store a run creates, and opens one with the configuration Icechunk kept with the store.	`None`
`expected_uuid`	`str \| None`	The UUID of the store the caller expects, for example from a catalog item. A store with another UUID, or without repository metadata, is refused, so a store replaced behind the same location is never taken for the old one.	`None`

Raises:

Type	Description
`FileNotFoundError`	If `create` is false and no repository is at `location`.
`FileExistsError`	If `create` is true and a repository is already at `location`.
`ValueError`	If `create` is true and `location` holds files but no repository, or if `expected_uuid` is given and the store has another UUID or no repository metadata.

`store_exists(location)` ¶

Whether a store's repository is at location.

A run creates a site's stores when it first writes data, so checks made before that, such as a resume or a comparison with stored results, ask this first instead of opening a store that isn't there.

Raises:

Type	Description
`StorageUnreachableError`	If the storage at `location` can't be reached.

`read_store_uuid(location)` ¶

The UUID of the store at location, from its repository metadata.

Raises:

Type	Description
`ValueError`	If the store has no repository metadata.

`repository_config(settings, *, compression_level=None)` ¶

The repository configuration processing.icechunk asks for.

A store's repository and the step that creates it use this one configuration, so a new store records the settings, not Icechunk's defaults.

Parameters:

Name	Type	Description	Default
`settings`	`IcechunkConfig`	The `processing.icechunk` settings.	required
`compression_level`	`int \| None`	A compression level other than the one in `settings`.	`None`

`store_location(location)` ¶

The store location location names.

A :class:StoreLocation is returned as it is; s3://bucket/prefix is a store on S3; any other path or string is a local directory (~ expanded).

Raises:

Type	Description
`ValueError`	If an `s3://` URL lacks the bucket or the prefix (Icechunk creates no repository at a bucket's root), or the URL has another scheme.

`validate_all(metadata, *, recommended=True, store_path=None)` ¶

Run all validators, return issues grouped by standard.

rules lists the fields that break a rule of their standard (e.g. a license outside SPDX), which a store read from disk may hold.

Parameters:

Name	Type	Description	Default
`recommended`	`bool`	Also list ACDD recommendations, FAIR advice and warnings. With `False` only failures are listed: a store created in the current form has none.	`True`
`store_path`	`Path \| None`	The local store `metadata` was read from. Given, the STAC check also warns when the store's catalog link has no item file yet.	`None`

`scan_stores(root_dir, recursive=True)` ¶

List the Icechunk stores under a directory, one row per store.

A store written before stores kept repository metadata is listed with needs_migration and the command that migrates it.

Returns:

Type	Description
`DataFrame`	One row per store with metadata columns.

Schema¶

Pydantic models for store metadata (14 sections).

Field names follow DataCite 4.5, ACDD 1.3, and STAC 1.1; validate.py checks the fields against them and the FAIR principles.

`MetadataRuleWarning` ¶

Bases: UserWarning

Metadata read from a store breaks a rule of its standard.

`RuleBreak` `dataclass` ¶

A rule broken by a metadata field.

`field` `instance-attribute` ¶

The field, as a dotted path (e.g. publisher.license).

`Publisher` ¶

Bases: BaseModel

`ReceiverHardware` ¶

Bases: BaseModel

The GNSS receiver itself (optional, from the settings).

`Antenna` ¶

Bases: BaseModel

The receiver's antenna (optional, from the settings).

`CadenceRange` ¶

Bases: BaseModel

A time range of a receiver's data at one cadence.

`VodAnalysisInfo` ¶

Bases: BaseModel

One VOD analysis a VOD store holds: the receivers it pairs.

`Amendment` ¶

Bases: BaseModel

One recorded change of an amendable field.

`MigrationRecord` ¶

Bases: BaseModel

How a store written before ADR 0014 was moved to repository metadata.

`StoreMetadata` ¶

Bases: BaseModel

Root metadata model composing all 14 sections.

`to_root_attrs()` ¶

Serialize to a JSON-compatible dict for Zarr root attrs.

`RepositoryMetadata` ¶

Bases: BaseModel

What describes a store as a whole: fixed and amendable fields.

`from_store_metadata(metadata, *, existing=False)` `classmethod` ¶

Take the repository part of metadata, in the current form.

Parameters:

Name	Type	Description	Default
`metadata`	`StoreMetadata`	Store metadata, in any form.	required
`existing`	`bool`	`metadata` was read from a store: a value breaking a rule is kept (reading warned about it already).	`False`

`spdx_license_ids()` `cached` ¶

The SPDX license identifiers of the list vendored here.

`rule_breaks(error)` ¶

The broken rules among the errors of error.

`describe_rule_breaks(breaks)` ¶

breaks in one line, for a message.

`check_rules(metadata)` ¶

The rules metadata breaks (one read from a store may break some).

`validate_existing(model, data, *, changed=())` ¶

Validate metadata of an existing store.

A rule broken by a field in changed fails, as for new metadata; a rule broken by another field warns, and the value is kept.

Parameters:

Name	Type	Description	Default
`model`	`type[M]`	The model to validate with.	required
`data`	`Any`	The metadata, as read from the store (and changed, for an amendment).	required
`changed`	`Iterable[str]`	Dotted fields being changed now (an amendment); `()` when the metadata is only read.	`()`

Returns:

Type	Description
`M`	The validated metadata, keeping values that break a rule.

Raises:

Type	Description
`ValueError`	If a field in `changed` breaks a rule; naming each.
`ValidationError`	If `data` is invalid for another reason.

Collectors¶

Runtime collectors: pure functions that gather metadata.

`collect_software_versions()` ¶

Collect versions of key packages via importlib.metadata.

`collect_python_info()` `cached` ¶

Return the Python version number and implementation ("3.14.8 CPython").

Not sys.version: it adds the build date and the compiler, and the store records no machine information.

`collect_uv_version()` `cached` ¶

Return the uv version number, or None if uv is not installed.

Only the number is kept ("0.13.1"): uv --version may add how uv was built and the platform, for example "uv 0.13.1 (Homebrew 2026-10-14 aarch64-apple-darwin)", and the store records no machine information. Asked once per process.

`collect_environment(store_path=None, dask_workers=None, dask_threads_per_worker=None, record_machine=True)` ¶

Collect runtime environment information.

With record_machine=False the computer (operating system, architecture, CPU count, memory, free disk) is not recorded: those fields are None (processing.metadata.record_machine).

`records_machine(config)` ¶

processing.metadata.record_machine of config: settings, a plain dict of them or a :class:ConfigSnapshot; true when not set.

`collect_config_snapshot(config)` ¶

Serialize config sections + compute SHA256 hash.

Left out (and out of the hash), because stores are shared: processing.credentials, the account details for data servers, and the settings of one computer (:data:~.machine.MACHINE_SETTINGS, and the stores' root when it is a local folder). The same settings on two computers give the same hash.

`collect_creator(metadata_config)` ¶

Build Creator from MetadataConfig.

`collect_publisher(metadata_config)` ¶

Build Publisher from MetadataConfig.

`collect_spatial(site_config, site_name)` ¶

Build SpatialExtent from SiteConfig.

`collect_receiver(receiver_config, site_name=None)` ¶

A receiver's entry in the store metadata, from its settings.

With site_name, a receiver with a naming recipe also gets the sha256 of its recipe file (:func:recipe_sha256_of). A recipe given as a URL is recorded by its file name only.

`recipe_sha256_of(site_name, recipe)` ¶

sha256 of the naming recipe file recipe of site_name.

The file is found the way a run finds it (canvod-filemap's find_recipe: a local recipe in the configuration directory, or a URL recipe, fetched).

Returns:

Type	Description
`str \| None`	The hexadecimal sha256, `None` if canvod-filemap is not installed or the recipe cannot be found (a run stops there before any store is written).

`collect_instruments(site_config, *, hardware=True, site_name=None)` ¶

Build Instruments from SiteConfig receivers.

hardware=False leaves out the receiver and antenna details: a VOD store reads them from the GNSS store's entries instead of copying them.

`collect_vod_analyses(site_config)` ¶

The VOD analyses of a site, for its VOD store.

`collect_processing_provenance(store_type, source_format, facility=None)` ¶

Build ProcessingProvenance.

facility is the institution that processed the data (processing.metadata.institution), never the computer.

`collect_references(config)` ¶

Build References from config if available.

`missing_store_settings(site_name, site_config, config)` ¶

Return the settings a new store needs but does not have.

A store records these when it is created (DataCite, STAC): the site's store description, the publisher, the license and the naming authority. Each returned name carries an example value, as the store-creation error lists them. The deprecated shared processing.metadata.store_description still counts as the site's store description.

The naming authority is checked with check_naming_authority: it is listed as missing when it is absent or not a reverse DNS name.

`missing_store_settings_message(missing, *, stores=None, store_label=None, store_path=None)` ¶

Return the store-creation error message for missing settings.

missing is the list from :func:missing_store_settings. stores names each of the new stores as (label, path) pairs and reads "the new at ", joined with " and ", with a plural verb: the form for the one error that names several new stores. A single store is described by store_path ("The new at ") or store_label ("A new "), and with none of them the message reads "A store", as collect_metadata prints it.

`collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None)` ¶

Collect all metadata sections into a StoreMetadata object.

This is the main entry point for metadata collection. It gives a new store its id, UUID and catalog link.

Raises:

Type	Description
`ValueError`	If the site name gives no valid store id, the naming authority is missing or not a reverse DNS name, or the site's store description, the publisher or the license is missing.

Store locations¶

Store locations and the one function that opens a store's repository.

Every repository in canVODpy is opened through :func:open_repository, so a new kind of storage is added in one place: a new :class:StoreLocation.

`StorageUnreachableError` ¶

Bases: OSError

A store's storage can't be reached; the message says what to check.

`StoreLocation` ¶

Bases: ABC

Where a store's repository lives.

`needs_commit_lock` `abstractmethod` `property` ¶

Whether commits here must hold :func:~.io.commit_lock.

`local_path` `abstractmethod` `property` ¶

The directory on this machine, or None for remote storage.

`access_hint` `abstractmethod` `property` ¶

What to check when the storage can't be reached.

`storage()` `abstractmethod` ¶

The Icechunk storage for this location.

`truediv(relative)` `abstractmethod` ¶

The location relative below this one, of the same kind.

Raises:

Type	Description
`ValueError`	If `relative` is an absolute path.

`LocalStoreLocation` `dataclass` ¶

Bases: StoreLocation

A store in a directory on a local file system.

`needs_commit_lock` `property` ¶

Yes: Icechunk on a local file system can lose one of two concurrent commits without an error.

`S3StoreLocation` `dataclass` ¶

Bases: StoreLocation

A store under a prefix in an S3 bucket.

Credentials come only from the environment (the AWS_* variables, AWS_PROFILE) or the shared AWS files (~/.aws/credentials, ~/.aws/config), as do region and endpoint (AWS_REGION, AWS_ENDPOINT_URL): never from the settings or the store metadata, and they are never logged. CANVOD_S3_PATH_STYLE=1 puts the bucket in the request path, for S3 services that expect it.

`needs_commit_lock` `property` ¶

No: Icechunk's conditional writes on S3 keep two commits from overwriting each other, and there is no local directory to hold a lock file.

`storage()` ¶

The Icechunk storage, configured from the environment.

Raises:

Type	Description
`ValueError`	If `CANVOD_S3_PATH_STYLE` is set to something other than yes or no.

`truediv(relative)` ¶

The location relative below this one, .. resolved.

S3 keys are plain names: S3 itself doesn't resolve .., so a catalog item's relative href is resolved here.

Raises:

Type	Description
`ValueError`	If `relative` is an absolute path, or leads to the bucket's root or out of the bucket.

`store_location(location)` ¶

The store location location names.

A :class:StoreLocation is returned as it is; s3://bucket/prefix is a store on S3; any other path or string is a local directory (~ expanded).

Raises:

Type	Description
`ValueError`	If an `s3://` URL lacks the bucket or the prefix (Icechunk creates no repository at a bucket's root), or the URL has another scheme.

`repository_config(settings, *, compression_level=None)` ¶

The repository configuration processing.icechunk asks for.

A store's repository and the step that creates it use this one configuration, so a new store records the settings, not Icechunk's defaults.

Parameters:

Name	Type	Description	Default
`settings`	`IcechunkConfig`	The `processing.icechunk` settings.	required
`compression_level`	`int \| None`	A compression level other than the one in `settings`.	`None`

`store_exists(location)` ¶

Whether a store's repository is at location.

A run creates a site's stores when it first writes data, so checks made before that, such as a resume or a comparison with stored results, ask this first instead of opening a store that isn't there.

Raises:

Type	Description
`StorageUnreachableError`	If the storage at `location` can't be reached.

`open_repository(location, *, create=False, config=None, expected_uuid=None)` ¶

Open the repository of the store at location.

Opening never creates a repository, and creating never opens one, so a mistyped location fails instead of leaving an empty store behind.

Parameters:

Name	Type	Description	Default
`create`	`bool`	Create a new repository instead of opening one.	`False`
`config`	`RepositoryConfig \| None`	The repository configuration (compression, caching, concurrency). `None` creates a store with the defaults of the `processing.icechunk` settings (:func:`repository_config`), so it matches a store a run creates, and opens one with the configuration Icechunk kept with the store.	`None`
`expected_uuid`	`str \| None`	The UUID of the store the caller expects, for example from a catalog item. A store with another UUID, or without repository metadata, is refused, so a store replaced behind the same location is never taken for the old one.	`None`

Raises:

Type	Description
`FileNotFoundError`	If `create` is false and no repository is at `location`.
`FileExistsError`	If `create` is true and a repository is already at `location`.
`ValueError`	If `create` is true and `location` holds files but no repository, or if `expected_uuid` is given and the store has another UUID or no repository metadata.

`read_store_uuid(location)` ¶

The UUID of the store at location, from its repository metadata.

Raises:

Type	Description
`ValueError`	If the store has no repository metadata.

Reading and writing¶

Read/write metadata to/from Icechunk store root attrs.

`StoreCreatedByAnotherRunError` ¶

Bases: ValueError

Another run created the store after this run found it without metadata.

`wait_after_conflict(retry)` ¶

Wait before try retry + 1 of a commit that lost a conflict.

Returns:

Type	Description
`float`	The seconds waited: they double with every retry and have a random part.

`commit_lock(store_path)` ¶

Let only one commit at a time change the store at store_path.

Icechunk on a local file system is not safe for concurrent commits: when two commits run at the same time, both can return successfully while one of them is lost. This lock, a file in the store directory, makes every other commit wait, in this process (other threads) and in other processes on the same machine. A commit that waits then sees the commit before it, and :func:commit_with_rebase rebases onto it.

The lock is re-entrant in one thread. The operating system releases it when the process that holds it ends.

`commit_with_rebase(session, message, metadata=None, *, store_path, config=None, repository=None, rebase=True, keep_summary=False)` ¶

Commit a session, rebasing it if the branch moved since it was opened.

The commit holds :func:commit_lock for store_path, the store the session belongs to, so commits to one store run one after the other. Several writers may still work on one store at the same time (e.g. two receivers of a site): when the branch moved since a session was opened, icechunk.ConflictDetector rebases changes that do not overlap (e.g. different groups) onto the new tip; overlapping changes fail with icechunk.RebaseFailedError.

A commit of data checks the run's content settings against the store's at the tip, after the rebase (ADR 0019): another run may have created the store or written first. A commit that changes the repository metadata (creation, amendment, migration) is not checked.

The commit also writes the root copy and the snapshot summary of the rebased session (ADR 0014); the summary is updated from the log-book rows added since this process's last commit to the branch, and rebuilt from the whole store at its first (ADR 0021, :mod:~canvod.store_metadata.summary_state). It records software, environment and settings as commit metadata, next to metadata. The lockfile text is stored once per hash: before a commit of data, in a commit of its own when the branch does not hold it yet (:func:store_lockfile), so two runs' data commits don't both write it and conflict (#352); a commit that changes the repository metadata carries it itself.

It refuses a store without repository metadata, so no store holds data without it, and a root group that differs from the repository metadata.

Every commit in canVODpy goes through this function, except manifest compaction, which goes through :func:rewrite_manifests.

Parameters:

Name	Type	Description	Default
`config`	`Any`	The settings recorded in the commit metadata: a settings object or a :class:`~canvod.store_metadata.schema.ConfigSnapshot`. `None` loads the settings.	`None`
`repository`	`dict[str, Any] \| None`	The repository metadata the commit carries, when it is about to change (creation, amendment, migration): the caller writes it with `Repository.set_metadata` only after the commit succeeded, so a failed commit leaves the repository metadata as it was. `None` uses the store's current repository metadata.	`None`
`rebase`	`bool`	`False` commits only onto the snapshot the session was opened on: if the branch moved, the commit fails with `icechunk.ConflictError` instead of rebasing. The creation of a store uses it, so of two runs that create one store at the same time only one commit lands (:func:`write_metadata`).	`True`
`keep_summary`	`bool`	`True` keeps the snapshot summary of the snapshot before, with only its time updated, instead of computing it from the data: for a commit that writes no data, such as growing day blocks (`MyIcechunkStore.grow_day_blocks`). It writes the same summary a computation would (ADR 0014), without reading every group's log book. The commit is refused unless its only changes are new day blocks in day-block groups: epoch axes grown at the end, epoch chunks and new block-index entries. It is also refused together with `repository`, and if the root group holds no summary.	`False`

Returns:

Type	Description
`str`	Snapshot ID from the commit.

Raises:

Type	Description
`ValueError`	If the store has no repository metadata and `repository` is not given, or the root group last committed differs from the repository metadata (written by an older canvodpy or edited by hand), or `keep_summary` is set and the commit is more than new day blocks.
`ContentSettingsError`	If the run's content settings differ from the store's.

`store_lockfile(repo, branch, repository, *, store_path, config)` ¶

Store the lockfile text in use in a commit of its own, unless branch has it.

The text is stored once per hash (ADR 0014). Two runs that start with the same new lockfile may both find it missing: the second commit then conflicts, finds the text stored by the first, and goes on.

Returns:

Type	Description
`str \| None`	Snapshot ID of the new commit; `None` if the text was stored already, or no lockfile is in use.

`rewrite_manifests(store_path, message, *, branch='main', config=None)` ¶

Compact a branch's manifests in a commit that carries commit metadata.

Repository.rewrite_manifests commits without a session, so it can write no lockfile text: when the lockfile in use is not stored in the store yet, a commit through :func:commit_with_rebase stores it first. In a store without repository metadata the compaction records only Icechunk's own metadata, as :func:commit_with_rebase does.

Parameters:

Name	Type	Description	Default
`config`	`Any`	The settings recorded in the commit metadata; `None` loads them.	`None`

Returns:

Type	Description
`str`	Snapshot ID of the compaction commit.

`read_repository_metadata(store_path)` ¶

The store's repository metadata as stored (empty for a store without).

`write_metadata(store_path, metadata, branch='main')` ¶

Write a new store's metadata.

The repository metadata (fixed and amendable fields, ADR 0014) is written once; a first commit writes its root copy and the snapshot summary, with metadata.config as the settings in the commit metadata.

Returns:

Type	Description
`str`	Snapshot ID from the commit.

Raises:

Type	Description
`StoreCreatedByAnotherRunError`	If the store already has repository metadata, or another run committed its creation first. A store on S3 has no commit lock, so two runs may both find it without metadata; the creation commit does not rebase, so only one of them creates it (#358).
`ValueError`	If `metadata` lacks a fixed field (build it with `collect_metadata`), or a field breaks a rule of its standard (e.g. a license outside SPDX).

`read_metadata(store_path, branch='main', *, snapshot_id=None)` ¶

Read the metadata of a snapshot as it was when it was committed.

One :class:StoreMetadata from the snapshot's root copy, snapshot summary and commit metadata. A snapshot written before the store kept repository metadata is read from its old canvod_metadata attribute and marked written_before_migration.

Parameters:

Name	Type	Description	Default
`branch`	`str`	Read the tip of this branch.	`'main'`
`snapshot_id`	`str \| None`	Read this snapshot instead of a branch tip (e.g. one a release pins).	`None`

Raises:

Type	Description
`KeyError`	If no metadata found in store.

`amendments_since(store_path, metadata)` ¶

Amendments made to the store after the snapshot metadata was read from.

`metadata_exists(store_path, branch='main')` ¶

Check if the store has repository metadata.

branch is not used: repository metadata is on no branch. It stays for callers that pass it.

`check_who(who)` ¶

Return who changes a store's metadata, refusing a missing name.

Raises:

Type	Description
`ValueError`	If `who` is missing or blank.

`store_command(command, site_name, store_type)` ¶

A canvodpy store command line for one site's store.

`ensure_metadata(store_path, collect, *, site_name, store_type, config=None)` ¶

Return a store's repository metadata; create a missing store first.

Writers call this before they write data, so a store never holds data without its metadata, and this is the only step that creates a store. collect builds the metadata of a new store (collect_metadata); it is called only for a store that has none.

Parameters:

Name	Type	Description	Default
`config`	`RepositoryConfig \| None`	The repository configuration a new store is created with (:func:`~.location.repository_config`); Icechunk keeps it with the store. `None` uses the defaults of the `processing.icechunk` settings.	`None`
`Two`			required
`commit`			required
`and`			required
`settings`			required

Raises:

Type	Description
`ValueError`	If the store holds data or old-form metadata but no repository metadata: it predates ADR 0014 and must be migrated first. Also if the location holds files but no store, or another run created the store but wrote no repository metadata in time.

`refuse_old_gnss_folder(store_path, site_name)` ¶

Refuse a GNSS store write while the site's old GNSS folder holds a store.

Before v1.0.0 the GNSS store's folder was rinex. A run writing to the GNSS store next to it would start a second GNSS store, or write to one while the old one waits for its migration (#456).

Raises:

Type	Description
`ValueError`	If `rinex` next to the GNSS store holds a store; names `canvodpy store migrate SITE`.

`ensure_store_metadata(store_path, *, config, site_name, store_type, source_format, names, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None)` ¶

Make a store ready for a run's data: the step every run takes first.

Gives a new store its repository metadata from the settings, then refuses names the metadata doesn't list: receivers and their pair groups for a GNSS store, VOD analyses for a VOD store. Then refuses settings that differ from the store's content settings, and a store without them (ADR 0019). Runs from the command line, Python and Airflow all take this step, so they behave the same.

Parameters:

Name	Type	Description	Default
`config`	`Any`	The settings of the run.	required
`source_format`	`str \| None`	The format of the reader that read the data (GNSS store) or the VOD calculator (VOD store) a new store records. `None` when it is unknown: then only a store that already has its metadata is accepted.	required
`names`	`Iterable[str]`	The groups the run will write.	required

Returns:

Type	Description
`dict`	The store's repository metadata.

Raises:

Type	Description
`ValueError`	If the site is not in the settings, the store is new and `source_format` is `None`, the store predates repository metadata (names the migrate command), a name is not listed in the metadata, or a GNSS store's naming recipe differs from the settings' (both name the amend command).
`ContentSettingsError`	If the settings differ from the store's content settings, or the store has none (names the migrate command).

`check_receivers(repository, receivers, *, store_path)` ¶

Refuse receivers the store's repository metadata does not list.

Raises:

Type	Description
`ValueError`	Naming the amend command that adds a receiver.

`check_recipes(repository, site_config, site_name, *, store_path)` ¶

Refuse naming recipes that differ from the ones the store was made with.

A store records, per receiver, its naming recipe and the sha256 of the recipe file. A run finds files with the recipe of the settings; a different recipe, or the same one with other content, would find other files than the days already stored. Receivers without a recipe in the settings and in the store are not checked.

Raises:

Type	Description
`ValueError`	Naming the receiver, both sha256s and the amend command that records the new recipe.

`current_recipes(site_config, site_name)` ¶

Each receiver's naming recipe in the settings and the sha256 of its file.

The recipe is given as a store records it: a recipe URL by its file name.

Raises:

Type	Description
`ValueError`	If a recipe is not found, so its sha256 is unknown.

`check_vod_analyses(repository, analyses, *, store_path)` ¶

Refuse VOD analyses the VOD store's repository metadata does not list.

Raises:

Type	Description
`ValueError`	Naming the amend command that adds a VOD analysis.

Validation¶

Validate store metadata against zarr conventions, ACDD 1.3, DataCite, STAC, FAIR.

A failure breaks a standard; a store created in the current form has none. Recommendations ([recommended]) and FAIR advice are listed with recommended=True. CF and the ACDD variable attributes are checked on a netCDF export by IOOS compliance-checker in the tests.

`root_attributes(metadata)` ¶

The root group attributes a commit writes for metadata.

Raises:

Type	Description
`ValueError`	If `metadata` is in the form from before the migration (no UUID, naming authority or STAC link).

`validate_datacite(metadata)` ¶

Check DataCite 4.5 mandatory fields.

`validate_acdd(metadata, *, recommended=True)` ¶

Check the root attributes against the ACDD 1.3 global attributes.

A missing highly recommended attribute is a failure; a missing recommended one is listed with recommended=True.

`validate_zarr_conventions(metadata)` ¶

Check the zarr conventions in the root attributes with zarr-cm.

`validate_stac(metadata, *, recommended=True)` ¶

Check STAC 1.1 Collection required fields.

`validate_fair(metadata)` ¶

Check FAIR data principles compliance.

Checks each sub-principle (F1–F4, A1–A2, I1–I3, R1.1–R1.3) and returns actionable issues for anything missing.

`validate_all(metadata, *, recommended=True, store_path=None)` ¶

Run all validators, return issues grouped by standard.

rules lists the fields that break a rule of their standard (e.g. a license outside SPDX), which a store read from disk may hold.

Parameters:

Name	Type	Description	Default
`recommended`	`bool`	Also list ACDD recommendations, FAIR advice and warnings. With `False` only failures are listed: a store created in the current form has none.	`True`
`store_path`	`Path \| None`	The local store `metadata` was read from. Given, the STAC check also warns when the store's catalog link has no item file yet.	`None`

`cf_standard_names()` `cached` ¶

The names and aliases of the CF standard name table vendored here.

Catalog¶

The STAC catalog of all stores under a data root (ADR 0015).

A static catalog beside the stores, never inside one: catalog.json, one collection per product (gnss-observations, vod, gnss-geodesy) and one item per store, all links relative, so the data root can be copied or moved. The catalog is derived from the stores and is written only on request.

Each asset names what opening its dataset needs: the store (href), the group (xarray:open_kwargs) and the snapshot (canvod:snapshot_id).

The catalog never leaves a store out silently: every store under the root is read first, and if any cannot be described, one error lists them all and nothing is written.

Each write replaces the latest catalog fully: the catalog, collection and item files of the previous write that the new one has no use for are removed, and only those. Any other file stays, and releases/ is never touched. Each file is replaced in one step, so a reader or a second writer never sees half a file; of two writers at once, the last one wins.

A release (:func:create_release) tags main in every store with the same Icechunk tag and writes the catalog of that tag under releases/<tag>/: the same tree, each asset pinned to the tag and the snapshot it points at. A release catalog is never changed afterwards.

`store_item(store_path)` ¶

The catalog item of one store, describing main now.

Parameters:

Name	Type	Description	Default
`store_path`	`Path \| str`	The store's directory.	required

Returns:

Type	Description
`Item`	Item `{naming_authority}.{store id}`: a point at the site, the time range of the data, the receivers with their positions, and one asset per stored dataset whose href is the store itself.

Raises:

Type	Description
`ValueError`	If the store cannot be described (see :func:`build_catalog`).

`build_catalog(root, *, release=None)` ¶

The catalog of every store under root, in memory.

Parameters:

Name	Type	Description	Default
`root`	`Path \| str`	The data root (`storage.stores_root_dir`).	required
`release`	`str \| None`	A release tag: the catalog of every store at that tag, each asset naming the tag. `None`: the latest catalog, of `main`.	`None`

Raises:

Type	Description
`ValueError`	Listing every store that cannot be described, with its reason: it cannot be opened, has no metadata, needs migration (with the migrate command), has no site position, its license is missing or not usable (not SPDX, or `other` without `license_uri`), or it has no tag `release`; or `root` is on S3.

`write_catalog(root, *, release=None)` ¶

Write the latest catalog of every store under root, beside them.

Parameters:

Name	Type	Description	Default
`root`	`Path \| str`	The data root (`storage.stores_root_dir`).	required
`release`	`str \| None`	A release tag the stores already have: write the catalog of that release under `releases/<tag>/` instead. :func:`create_release` tags the stores and calls this.	`None`

Returns:

Type	Description
`Path`	The written `catalog.json`.

Raises:

Type	Description
`ValueError`	Listing every store that cannot be described (see :func:`build_catalog`); nothing is written. With `release`, also if the tag is not a single folder name or its catalog exists already: a release catalog is never changed. If `root` is on S3.

`create_release(root, tag)` ¶

Tag main in every store under root and write the release catalog.

Every store gets the Icechunk tag tag on the snapshot of main the release describes, and releases/<tag>/ gets the catalog of that tag: the latest catalog's tree, each asset naming the tag and its snapshot. If the stores have the tag already (a release whose catalog write failed), only the catalog is written.

Parameters:

Name	Type	Description	Default
`root`	`Path \| str`	The data root (`storage.stores_root_dir`).	required
`tag`	`str`	The release tag, for example `v2026.1`. It holds no `/`: that keeps it apart from the keeper tags (`keep/<receiver>/<date>`), and `releases/<tag>/` one folder deep.	required

Returns:

Type	Description
`Path`	The release's `catalog.json`.

Raises:

Type	Description
`ValueError`	Before anything is tagged or written: the tag holds a `/`, the release catalog exists already, a store cannot be described (see :func:`build_catalog`, all of them listed), no store is under `root`, `root` is on S3, or some stores have the tag and others don't (those that have it listed).

Inventory¶

Store catalog builder: scan directories for Icechunk stores.

`scan_stores(root_dir, recursive=True)` ¶

List the Icechunk stores under a directory, one row per store.

A store written before stores kept repository metadata is listed with needs_migration and the command that migrates it.

Returns:

Type	Description
`DataFrame`	One row per store with metadata columns.

Display¶

Human-readable metadata display and query tool.

Usage

from canvod.store_metadata import show_metadata
show_metadata("/path/to/store")                  # full report
show_metadata("/path/to/store", section="env")   # just environment
show_metadata("/path/to/store", section="uv")    # dump uv.lock

`format_content_settings(meta)` ¶

The store's content settings (ADR 0019): every run must have them.

`format_validation(meta, store_path=None)` ¶

Failures per standard, then the recommendations, FAIR advice and warnings.

store_path, the local store meta comes from, adds the warning about a catalog link without an item file (:func:validate_all).

`format_metadata(meta, section=None, *, store_path=None)` ¶

Format metadata as a human-readable string.

Parameters:

Name	Type	Description	Default
`meta`	`StoreMetadata`	The metadata to format.	required
`section`	`str \| None`	If given, show only this section. Special values: - "uv" / "uv.lock": dump raw uv.lock content - "toml" / "pyproject": dump raw pyproject.toml content - "env-reproduce": print instructions to reproduce env - Any section name of :func:`_sections`: show that section only If None, show full report.	`None`
`store_path`	`Path \| None`	The local store `meta` comes from, for the validation section (:func:`format_validation`).	`None`

`extract_env(store_path, output_dir, branch='main')` ¶

Extract pyproject.toml + uv.lock from a store for reproduction.

Parameters:

Name	Type	Description	Default
`store_path`	`Path`	Path to the Icechunk store.	required
`output_dir`	`Path`	Directory to write the files into.	required
`branch`	`str`	Store branch.	`'main'`

Returns:

Type	Description
`Path`	The output directory (ready for `uv sync --frozen`).

`show_metadata(store_path, section=None, branch='main', *, snapshot_id=None)` ¶

Print store metadata to stdout.

Parameters:

Name	Type	Description	Default
`store_path`	`str \| Path`	Path to an Icechunk store.	required
`section`	`str \| None`	Section to show (None = full report).	`None`
`branch`	`str`	Show the tip of this branch.	`'main'`
`snapshot_id`	`str \| None`	Show this snapshot instead of a branch tip.	`None`

canvod.store_metadata API reference¶

Package¶

StoreMetadata ¶

to_root_attrs() ¶

StoreLocation ¶

needs_commit_lock abstractmethod property ¶

local_path abstractmethod property ¶

access_hint abstractmethod property ¶

storage() abstractmethod ¶

__truediv__(relative) abstractmethod ¶

LocalStoreLocation dataclass ¶

needs_commit_lock property ¶

S3StoreLocation dataclass ¶

needs_commit_lock property ¶

storage() ¶

__truediv__(relative) ¶

collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None) ¶

write_metadata(store_path, metadata, branch='main') ¶

read_metadata(store_path, branch='main', *, snapshot_id=None) ¶

amend_metadata(store_path, changes, *, reason, who, config=None, receiver_content=None) ¶

migrate_metadata(store_path, *, who, naming_authority=None, values=None, variable_attrs=None, branch='main', set_option='--set') ¶

metadata_exists(store_path, branch='main') ¶

open_repository(location, *, create=False, config=None, expected_uuid=None) ¶

store_exists(location) ¶

read_store_uuid(location) ¶

repository_config(settings, *, compression_level=None) ¶

store_location(location) ¶

validate_all(metadata, *, recommended=True, store_path=None) ¶

scan_stores(root_dir, recursive=True) ¶

Schema¶

MetadataRuleWarning ¶

RuleBreak dataclass ¶

field instance-attribute ¶

Publisher ¶

ReceiverHardware ¶

Antenna ¶

CadenceRange ¶

VodAnalysisInfo ¶

Amendment ¶

MigrationRecord ¶

StoreMetadata ¶

to_root_attrs() ¶

RepositoryMetadata ¶

from_store_metadata(metadata, *, existing=False) classmethod ¶

spdx_license_ids() cached ¶

rule_breaks(error) ¶

describe_rule_breaks(breaks) ¶

check_rules(metadata) ¶

validate_existing(model, data, *, changed=()) ¶

Collectors¶

collect_software_versions() ¶

collect_python_info() cached ¶

collect_uv_version() cached ¶

collect_environment(store_path=None, dask_workers=None, dask_threads_per_worker=None, record_machine=True) ¶

records_machine(config) ¶

collect_config_snapshot(config) ¶

collect_creator(metadata_config) ¶

collect_publisher(metadata_config) ¶

collect_spatial(site_config, site_name) ¶

collect_receiver(receiver_config, site_name=None) ¶

recipe_sha256_of(site_name, recipe) ¶

collect_instruments(site_config, *, hardware=True, site_name=None) ¶

collect_vod_analyses(site_config) ¶

collect_processing_provenance(store_type, source_format, facility=None) ¶

collect_references(config) ¶

missing_store_settings(site_name, site_config, config) ¶

missing_store_settings_message(missing, *, stores=None, store_label=None, store_path=None) ¶

collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None) ¶

Store locations¶

StorageUnreachableError ¶

StoreLocation ¶

needs_commit_lock abstractmethod property ¶

local_path abstractmethod property ¶

access_hint abstractmethod property ¶

storage() abstractmethod ¶

__truediv__(relative) abstractmethod ¶

LocalStoreLocation dataclass ¶

needs_commit_lock property ¶

S3StoreLocation dataclass ¶

needs_commit_lock property ¶

`canvod.store_metadata` API reference¶

`StoreMetadata` ¶

`to_root_attrs()` ¶

`StoreLocation` ¶

`needs_commit_lock` `abstractmethod` `property` ¶

`local_path` `abstractmethod` `property` ¶

`access_hint` `abstractmethod` `property` ¶

`storage()` `abstractmethod` ¶

`truediv(relative)` `abstractmethod` ¶

`LocalStoreLocation` `dataclass` ¶

`needs_commit_lock` `property` ¶

`S3StoreLocation` `dataclass` ¶

`needs_commit_lock` `property` ¶

`storage()` ¶

`truediv(relative)` ¶

`collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None)` ¶

`write_metadata(store_path, metadata, branch='main')` ¶

`read_metadata(store_path, branch='main', *, snapshot_id=None)` ¶

`amend_metadata(store_path, changes, *, reason, who, config=None, receiver_content=None)` ¶

`migrate_metadata(store_path, *, who, naming_authority=None, values=None, variable_attrs=None, branch='main', set_option='--set')` ¶

`metadata_exists(store_path, branch='main')` ¶

`open_repository(location, *, create=False, config=None, expected_uuid=None)` ¶

`store_exists(location)` ¶

`read_store_uuid(location)` ¶

`repository_config(settings, *, compression_level=None)` ¶

`store_location(location)` ¶

`validate_all(metadata, *, recommended=True, store_path=None)` ¶

`scan_stores(root_dir, recursive=True)` ¶

`MetadataRuleWarning` ¶

`RuleBreak` `dataclass` ¶

`field` `instance-attribute` ¶

`Publisher` ¶

`ReceiverHardware` ¶

`Antenna` ¶

`CadenceRange` ¶

`VodAnalysisInfo` ¶

`Amendment` ¶

`MigrationRecord` ¶

`StoreMetadata` ¶

`to_root_attrs()` ¶

`RepositoryMetadata` ¶

`from_store_metadata(metadata, *, existing=False)` `classmethod` ¶

`spdx_license_ids()` `cached` ¶

`rule_breaks(error)` ¶

`describe_rule_breaks(breaks)` ¶

`check_rules(metadata)` ¶

`validate_existing(model, data, *, changed=())` ¶

`collect_software_versions()` ¶

`collect_python_info()` `cached` ¶

`collect_uv_version()` `cached` ¶

`collect_environment(store_path=None, dask_workers=None, dask_threads_per_worker=None, record_machine=True)` ¶

`records_machine(config)` ¶

`collect_config_snapshot(config)` ¶

`collect_creator(metadata_config)` ¶

`collect_publisher(metadata_config)` ¶

`collect_spatial(site_config, site_name)` ¶

`collect_receiver(receiver_config, site_name=None)` ¶

`recipe_sha256_of(site_name, recipe)` ¶

`collect_instruments(site_config, *, hardware=True, site_name=None)` ¶

`collect_vod_analyses(site_config)` ¶

`collect_processing_provenance(store_type, source_format, facility=None)` ¶

`collect_references(config)` ¶

`missing_store_settings(site_name, site_config, config)` ¶

`missing_store_settings_message(missing, *, stores=None, store_label=None, store_path=None)` ¶

`collect_metadata(*, config, site_name, site_config, store_type, source_format, store_path=None, dask_workers=None, dask_threads_per_worker=None, gnss_repository=None)` ¶

`StorageUnreachableError` ¶

`StoreLocation` ¶

`needs_commit_lock` `abstractmethod` `property` ¶

`local_path` `abstractmethod` `property` ¶

`access_hint` `abstractmethod` `property` ¶

`storage()` `abstractmethod` ¶

`truediv(relative)` `abstractmethod` ¶

`LocalStoreLocation` `dataclass` ¶

`needs_commit_lock` `property` ¶

`S3StoreLocation` `dataclass` ¶

`needs_commit_lock` `property` ¶

`storage()` ¶

`truediv(relative)` ¶

`store_location(location)` ¶

`repository_config(settings, *, compression_level=None)` ¶