Backup Storage#

Plan required

Backups are available only on the Premium and Enterprise plans. Freemium accounts cannot back up, restore or schedule backups.

Volume backups for a service location are executed by the Ametnes agent and stored in a backup repository that you provide. This page explains how to choose and configure that repository, distribute credentials, and operate the backup jobs.

The control plane stores the backup ledger and schedules. It never reads or writes the repository and never stores repository or cloud credentials. The agent creates standard Kubernetes Jobs in the resource's namespace; no CRDs or operators are installed.

For the conceptual overview (manual runs, schedules, retention, and deletion), see Backups. For console workflows, see Backup Operations.

Before you begin#

A running agent installed with Helm in the location cluster.
A repository that every node can reach at the same path (mount target) or an S3-compatible bucket (S3 target).
The repository password stored in a Kubernetes Secret, or a password file in the mounted repository.

Backups are disabled by default (agent.config.backup.enabled: false). While disabled, agents do not process INIT runs. DELETING runs are always processed so resource deletion can clean up snapshots.

Choose a target#

Target	Use when	Trade-offs
`mount`	You already run NFS or another shared filesystem and want the repository on storage you control.	The same export must be mounted at the same path on every node. The repository directory is opaque; copy it by snapshot, not `rsync`.
`s3`	You want the repository reachable from every node without filesystem setup.	Requires S3 credentials (or workload identity) and a repository password Secret in every service namespace.

The default repository path for a mount target is <mount_path>/ametnes/<location>, so one repository holds every resource's snapshots for that location.

Mount target#

agent:
  config:
    backup:
      enabled: true
      target:
        type: mount
        password_secret:
          name: backup-repository-password
          key: RESTIC_PASSWORD
        mount:
          host_path: /var/lib/ametnes-backup

Expose the repository in one of two ways:

Key	Meaning
`mount.host_path`	A path on each node, for example an NFS export mounted at `/var/lib/ametnes-backup`. The Job can schedule on any node, so the same export must be mounted at the same path on every node.
`mount.host_path_type`	Host path type for `mount.host_path`. Defaults to `Directory`; leave it at `Directory` (see the warning below).
`mount.claim_name`	A PVC in the service namespace. Pods cannot mount a PVC from another namespace. Use an RWX NFS PVC so Jobs on any node can mount it.
`mount.mount_path`	Path where the repository appears inside the backup Job. Defaults to `/repo`.
`mount.repository`	Explicit repository path. Defaults to `<mount_path>/ametnes/<location>`.

Keep the host path type as Directory. With DirectoryOrCreate, a missing mount silently becomes an empty local directory and the agent initialises a wrong, node-local repository.

When using mount.claim_name, a PV binds to exactly one PVC. Sharing one export across namespaces therefore means one static PV/PVC pair per namespace, or a per-namespace subdirectory created by an NFS provisioner.

For NFS, mount with NFSv4.1 or later, hard, noatime, a low actimeo, _netdev, and never nolock. Export with sync, and keep clocks synchronised (NTP) on nodes and the NFS server.

The repository directory is opaque. To take a copy, snapshot the export (or stop writes) and copy the snapshot. Never rsync a live repository during a backup or maintenance. A copy of the repository directory is directly usable and is the portable, customer-owned artifact.

S3 target#

agent:
  config:
    backup:
      enabled: true
      target:
        type: s3
        password_secret:
          name: backup-repository-password
          key: RESTIC_PASSWORD
        s3:
          endpoint: https://s3.eu-west-2.amazonaws.com
          bucket: ametnes-backups
          prefix: prod-euw2
          region: eu-west-2
          secret_name: backup-s3-credentials

Key	Meaning
`s3.endpoint`	S3-compatible endpoint.
`s3.bucket`	Bucket that holds the repository.
`s3.prefix`	Key prefix inside the bucket.
`s3.region`	Region for the S3 client.
`s3.secret_name`	Secret containing the access key and secret key. Leave empty to use workload identity.
`s3.access_key_key`	Key for the access key ID. Defaults to `AWS_ACCESS_KEY_ID`.
`s3.secret_key_key`	Key for the secret access key. Defaults to `AWS_SECRET_ACCESS_KEY`.

prefix is not location-scoped. When several locations share an endpoint and bucket, set a distinct prefix per location. Otherwise they share one repository.

An S3 target requires target.password_secret. target.mount.password_file is mount-only and is rejected for S3 because an S3 Job has no repository mount from which to read the file.

Repository password#

Setting	Target	Meaning
`target.password_secret`	mount and s3	Secret in each service namespace containing the repository password. Required for S3, optional for mount.
`target.mount.password_file`	mount only	Path inside the mounted repository, for example `/repo/.backup-password`. Takes precedence over the Secret when both are set.

Password settings belong at the target level, not inside mount or s3. The agent rejects mount.password_secret and s3.password_secret and tells you to move them. target.password_file is accepted as a deprecated alias for mount targets and rejected for S3.

Distribute credentials#

Backup Jobs run in the service namespace, and a Secret cannot be referenced across namespaces. Keep the source Secrets in the agent namespace and let the agent copy them:

kubectl -n ametnes-system create secret generic backup-repository-password \
  --from-literal=RESTIC_PASSWORD='<repository password>'

kubectl -n ametnes-system label secret backup-repository-password \
  ametnes.io/distribute=scheduled

kubectl -n ametnes-system create secret generic backup-s3-credentials \
  --from-literal=AWS_ACCESS_KEY_ID='<key>' \
  --from-literal=AWS_SECRET_ACCESS_KEY='<secret>'

kubectl -n ametnes-system label secret backup-s3-credentials \
  ametnes.io/distribute=scheduled

The marker can be a label (preferred) or an annotation:

Value	Behaviour
`scheduled`	Create the copy in every agent-managed namespace and keep it in sync. Updating the source rotates the copies.
`once`	Create the copy where missing. Never update or delete it.
`true`	Alias for `scheduled`.

The agent reconciles shared Secrets every five minutes and immediately after a resource namespace is created. Managed namespaces are ametnes-<account>-<project> and account-<account>. Copies carry ametnes.io/distributed=true and ametnes.io/distributed-from. A scheduled copy is deleted when its source Secret is deleted or loses the marker.

TLS certificate and CA copies written by the certificate and CA tasks also carry ametnes.io/distributed=true, but with ametnes.io/distributed-mode=control-plane, so this reconciler never rewrites or deletes them.

Then point the target at the copies:

agent:
  config:
    backup:
      target:
        password_secret:
          name: backup-repository-password
          key: RESTIC_PASSWORD
        s3:
          secret_name: backup-s3-credentials

The Secrets are readable by principals that can read Secrets in the tenant namespace, like the TLS Secrets already distributed there. For stronger tenant isolation, use a repository per tenant with tenant-owned credentials.

Configuration reference#

Agent chart values under agent.config.backup:

Key	Default	Meaning
`enabled`	`false`	Process `INIT` runs. `DELETING` runs are always processed.
`timeout`	`3600`	Job timeout and lease duration base, in seconds.
`retention`	`30`	Fallback retention window, in days, when a run is not created by a schedule.
`scale_wait`	`10`	Seconds to wait after scaling down before capturing volumes.
`poll_interval`	`5`	Job status polling interval, in seconds.
`max_concurrent`	`4`	Concurrent per-volume Jobs.
`target.*`	-	Repository target described above.

Control plane (control/config/<profile>/config.yml):

Key	Default	Meaning
`backup.stale_timeout`	`21600`	A run stuck in `INIT`, or a deletion stuck in `DELETING`, is marked `ERROR` after this many seconds.

On startup the agent logs the effective backup configuration: target type, repository, S3 region, credential sources (Secret names or workload identity), timeout, retention, and concurrency. Secret values are never logged.

Concurrency and locking#

A Kubernetes Lease per resource prevents two operations on the same resource across nodes and agent replicas.
A repository Lease serializes repository maintenance across agents writing the same repository. It lives in the agent namespace.
The backup engine holds repository locks. Repository maintenance waits for an interrupted backup to finish before it continues.

Backups of different resources may run concurrently on different nodes. Only repository maintenance is exclusive per repository.

Operations#

# Backups in flight
kubectl get jobs -A -l app.kubernetes.io/managed-by=ametnes-agent

# A specific run's Jobs
kubectl get pods -n <service-namespace> -l ametnes.io/backup-run=<run-id>

Use your backup tooling to inspect the snapshot inventory for a resource or to list repository locks.

Common failures:

Symptom	Cause	Action
Job `Pending`, `persistentvolumeclaim ... not found`	Manifest references a missing PVC.	Upgrade the agent; maintenance Jobs no longer mount data volumes.
`CreateContainerConfigError`, `secret ... not found`	Password Secret configured but absent.	Create the Secret, or (mount targets only) use `mount.password_file`.
`repository is already locked`	A killed Job left a repository lock.	Wait for restic to treat the lock as stale (30 minutes by default), or clear the stale lock with your backup tooling in a controlled window.
Retention Job fails while backups succeed	Concurrent maintenance or lock conflict.	Current versions serialize repository maintenance with the repository lease; check the lease and agent logs.
Run stuck in `INIT`	The claiming agent disappeared.	The control plane marks it `ERROR` after `backup.stale_timeout`.

Limitations#

Product-level limits (quiesced-only backups, same-resource restore, deleting a record vs. its snapshots) are covered in Backups. Operator-facing limits are:

Cross-location restore and disaster recovery are not implemented. A shared repository across locations is not serialized by the repository lease.
There is no repository garbage collector. Orphaned snapshots are removed by the resource deletion flow or by manual repository maintenance.
Snapshots written before tenant-qualified tags were introduced cannot be matched by the current filters and are not removed automatically.
Ledger retention is not functional; backup records are not aged out automatically.
Deleted-resource tombstones can appear in listings for up to one reaper interval while snapshots are removed.