Stores on S3¶

A run writes its stores into an S3 bucket when processing.storage.stores_root_dir is an s3://bucket/prefix URL (settings file). This page covers what to check before you choose a provider, and how to set up the bucket and the credentials.

Credentials¶

canVODpy reads credentials, region and endpoint only from the environment and the shared AWS files, never from the settings file or the store metadata. They're never logged.

Give each machine that runs canVODpy its own key, so you can withdraw one machine's access without touching the others.

Access policies¶

A run needs to list the store prefix and to read and write objects below it. A run never deletes objects: the commits of a run, and the refusal of a conflicting commit, use only the S3 operations below. This was checked against the S3 emulator with a demo run on S3 that included two runs writing to one store at the same time.

Writer: a machine that runs canVODpy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-bucket",
      "Condition": {"StringLike": {"s3:prefix": "stores/*"}}
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::example-bucket/stores/*"
    }
  ]
}

Reader: someone who opens the stores and reads their data. Use the same policy without s3:PutObject.

Replace example-bucket and stores with your bucket and the prefix of stores_root_dir. Providers other than AWS name these permissions differently or support only part of this policy language; check their documentation.

Conditional writes¶

Icechunk, the storage engine of the stores, commits with conditional writes: it creates an object only if it doesn't exist yet (If-None-Match) and replaces one only if nobody changed it since it was read (If-Match). This is what keeps two runs that commit to one store at the same time from overwriting each other's commit.

A provider that ignores these conditions accepts both commits, and one of them is lost without an error. Ask the provider whether it supports conditional PUT with If-Match and If-None-Match before you choose it, and don't let two runs write to one store until you know. AWS S3 supports both. On a local folder canVODpy uses a lock file instead, so this question doesn't arise there.

Bucket settings that break a store¶

A store is many objects that refer to each other. Leave the objects below the store prefix to canVODpy:

• No lifecycle rule that expires or moves objects below the store prefix. Expiry deletes objects that the store still refers to, and the store can no longer be read. Moving them to an archive storage class makes reads fail until they're restored.
• No bucket versioning on the store prefix. Icechunk keeps its own history of every commit (snapshots), so versioning adds no safety, and objects deleted by maintenance would stay as old versions you still pay for.
• Don't delete, rename or copy single objects of a store. To move a store, copy the whole prefix while no run writes to it.

Costs¶

A store on S3 only grows: every commit adds objects, and canvodpy store maintain doesn't delete anything on S3 yet (see Limits). Plan the bucket's size for the full history of every run. canvodpy store maintain SITE --store gnss reports how much maintenance would free.

Limits¶

These work on a local folder but not yet on S3:

• canvodpy store maintain and maintain-due only report what they would delete. --execute is refused, because S3 has no lock file that keeps a run from writing during maintenance.
• No STAC catalog is written for stores on S3.
• The run history stays in the log folder of the machine that runs, so each machine has its own.
• Only s3:// URLs: Google Cloud Storage (gs://) and Azure Blob Storage (az://) are refused.

Stores on S3 in the canvod-store-metadata docs shows how to open a store on S3 from Python.