This is unreleased documentation for SUSE® Storage 1.13 (Dev).

Create a Recurring Snapshot or Backup Job

From the SUSE Storage UI, the volume can refer to recurring snapshots and backups as independent jobs or as recurring job groups.

To create a recurring job, you can go to the Recurring Jobs page in SUSE Storage and Create Recurring Job or in the volume detail view in SUSE Storage.

You can configure,

Any groups that the job should belong to
The type of schedule, either backup, backup-force-create, snapshot, snapshot-force-create, snapshot-cleanup, snapshot-delete or filesystem-trim
The time that the backup or snapshot will be created, in the form of a CRON expression
How the backups or snapshots are retained, either by number (count-based) or by age (age-based). For more information, see Retention Policies.
The number of jobs to run concurrently
Any labels that should be applied to the backup or snapshot
Parameters that should be applied to the backup
- full-backup-interval: Number of incremental backups that must be completed before SUSE Storage performs a full backup. This integer parameter is applied only to the backup. Notice that if the value is 0, SUSE Storage performs a incremental backup every time. For more information, see Periodic Full Backup and Create a Backup.

Recurring jobs can be set up using the SUSE Storage UI, kubectl, or by using a SUSE Storage RecurringJob.

To add a recurring job to a volume, you will go to the volume detail view in SUSE Storage. Then you can set Recurring Jobs Schedule.

Create a new recurring job
Select from existing recurring jobs
Select from existing recurring job groups

Then SUSE Storage will automatically create snapshots or backups for the volume at the recurring job scheduled time, as long as the volume is attached to a node. To set up recurring snapshots and backups even when the volumes are detached, see the section Allow Recurring Job While Volume Is Detached.

You can set recurring jobs on a Longhorn Volume, Kubernetes Persistent Volume Claim (PVC), or Kubernetes StorageClass.

When the PVC has recurring job labels, they will override all recurring job labels of the associated Volume.

For more information on how snapshots and backups work, refer to the concepts section.

To avoid the problem that recurring jobs may overwrite the old backups/snapshots with identical backups and empty snapshots when the volume does not have new data for a long time, Longhorn does the following:

Recurring backup job only takes a new backup when the volume has new data since the last backup.
Recurring snapshot job only takes a new snapshot when the volume has new data in the volume head (the live data).

Set up Recurring Jobs

Using the SUSE Storage UI

Recurring snapshots and backups can be configured from the Recurring Job page or the volume detail page.

Using the manifest

You can also configure the recurring job by directly interacting with the Longhorn RecurringJob custom resource.

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: snapshot-1
  namespace: longhorn-system
spec:
  cron: "* * * * *"
  task: "snapshot"
  groups:
  - default
  - group1
  retentionPolicy: count-based
  retain: 1
  concurrency: 2
  labels:
    label/1: a
    label/2: b

The following parameters should be specified for each recurring job selector:

name: Name of the recurring job. Do not use duplicate names. And the length of name should be no more than 40 characters.

task: Type of the job. SUSE Storage supports the following:

backup: periodically create snapshots then do backups after cleaning up outdated snapshots
backup-force-create: periodically create snapshots then do backups
snapshot: periodically create snapshots after cleaning up outdated snapshots
snapshot-force-create: periodically create snapshots
snapshot-cleanup: periodically purge removable snapshots and system snapshots

Retain value has no effect for this task, SUSE Storage automatically mutates the retain value to 0 and the retainAge value to 0s

snapshot-delete: periodically remove and purge all kinds of snapshots that exceed the retention count.

The retain value is independent of each recurring job.

Using a volume with 2 recurring jobs as an example:

snapshot with retain value set to 5
snapshot-delete: with retain value set to 2

Eventually, there will be 2 snapshots retained after a complete snapshot-delete task execution.

filesystem-trim: periodically trim filesystem to reclaim disk space

The snapshot-cleanup, snapshot-delete, and filesystem-trim tasks support only the count-based retention policy. Longhorn rejects such a recurring job when retentionPolicy is set to age-based.

cron: Cron expression. It tells the execution time of the job.
retain: How many snapshots/backups Longhorn will retain for each volume job. It should be no less than 1. This value is used only when retentionPolicy is count-based.
concurrency: The number of jobs to run concurrently. It should be no less than 1.

Optional parameters can be specified:

groups: Any groups that the job should belong to. Having default in groups will automatically schedule this recurring job to any volume with no recurring job.
labels: Any labels that should be applied to the backup or snapshot.
retentionPolicy: How Longhorn decides which snapshots/backups the job cleans up, either count-based or age-based. The default value is count-based. For more information, see [Retention Policies](#retention-policies).
retainAge: How long Longhorn will retain the snapshots/backups of the job, specified as a duration string such as 720h. This value is used only when retentionPolicy is age-based.

Retention Policies

The retentionPolicy field determines which snapshots and backups a recurring job cleans up. The two policies work independently, and Longhorn reads only the field that belongs to the policy in effect.

retentionPolicy Field used Behavior

`retentionPolicy`	Field used	Behavior
`count-based` (default)	`retain`	Keeps the newest `retain` snapshots/backups of the job and deletes the older ones, regardless of their age. `retainAge` is ignored.
`age-based`	`retainAge`	Deletes the snapshots/backups of the job that are older than `retainAge` and keeps the rest, regardless of how many there are. `retain` is ignored.

count-based (default)

retain

Keeps the newest retain snapshots/backups of the job and deletes the older ones, regardless of their age. retainAge is ignored.

age-based

retainAge

Deletes the snapshots/backups of the job that are older than retainAge and keeps the rest, regardless of how many there are. retain is ignored.

age-based retention is a rolling window that Longhorn re-evaluates on every run, so the number of retained snapshots/backups varies with how frequently the job runs.

The following example is a recurring backup job that runs daily and retains the backups of the last 30 days:

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: backup-last-30-days
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"
  task: "backup"
  retentionPolicy: age-based
  retainAge: 720h
  concurrency: 2

retainAge accepts the duration units h, m, and s. There is no unit for days or years, so specify one day as 24h and 30 days as 720h.
A recurring job with retentionPolicy set to age-based and retainAge set to 0s never runs. You must specify a positive retainAge before the job can take effect.
Recurring jobs created before Longhorn v1.13.0 use count-based retention with their existing retain value after the upgrade, so their behavior is unchanged.

Add Recurring Jobs to the Default group

Default recurring jobs can be set by tick the checkbox default using UI or adding default to the recurring job groups.

SUSE Storage will automatically add a volume to the default group when the volume has no recurring job.

Delete Recurring Jobs

SUSE Storage automatically removes Volume and PVC recurring job labels when a corresponding RecurringJob custom resource is deleted. However, if a recurring job label is added without an existing RecurringJob custom resource, SUSE Storage does not perform the cleanup process for that label.

Apply Recurring Job to SUSE Storage Volume

Using the SUSE Storage UI

The recurring job can be assigned on the volume detail page. To navigate to the volume detail page, click Volume then click the name of the volume.

Using kubectl

Add recurring job group:

kubectl -n longhorn-system label volume/<VOLUME-NAME> recurring-job-group.longhorn.io/<RECURRING-JOB-GROUP-NAME≥enabled

# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job-group.longhorn.io/default=enabled

Add recurring job:

kubectl -n longhorn-system label volume/<VOLUME-NAME> recurring-job.longhorn.io/<RECURRING-JOB-NAME≥enabled

# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job.longhorn.io/backup=enabled

Remove recurring job:

kubectl -n longhorn-system label volume/<VOLUME-NAME> <RECURRING-JOB-LABEL>-

# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job.longhorn.io/backup-

With a PersistentVolumeClaim Using kubectl

By default, applying a recurring job to a Persistent Volume Claim (PVC) does not have any effect. You can enable or disable this feature using the recurring job source label.

Once the PVC is labeled as the source, any recurring job labels added or removed from the PVC will be periodically synchronized by SUSE Storage to the associated Volume.

kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job.longhorn.io/source=enabled

# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/source=enabled

Add recurring job group:

kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job-group.longhorn.io/<RECURRING-JOB-GROUP-NAME≥enabled

# Example:
# kubectl -n default label pvc/sample recurring-job-group.longhorn.io/default=enabled

Add recurring job:

kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job.longhorn.io/<RECURRING-JOB-NAME≥enabled

# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/backup=enabled

Remove recurring job:

kubectl -n <NAMESPACE> label pvc/<PVC-NAME> <RECURRING-JOB-LABEL>-

# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/backup-

With StorageClass parameters

Recurring job assignment can be configured in the recurringJobSelector parameters in a StorageClass.

Any future volumes created using this StorageClass will have those recurring jobs automatically assigned.

The recurringJobSelector field should follow JSON format:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  recurringJobSelector: '[
    {
      "name":"snap",
      "isGroup":true
    },
    {
      "name":"backup",
      "isGroup":false
    }
  ]'

The following parameters should be specified for each recurring job selector:

name: Name of an existing recurring job or an existing recurring job group.
isGroup: is the name that belongs to a recurring job or recurring job group, either true or false.

Allow Recurring Job While Volume Is Detached

SUSE Storage provides the setting allow-recurring-job-while-volume-detached that allows you to do recurring backup even when a volume is detached. You can find the setting in SUSE Storage UI.

When the setting is enabled, SUSE Storage will automatically attach the volume and take a snapshot or backup when it is time to do a recurring snapshot/backup.

Note that during the time the volume was attached automatically, the volume is not ready for the workload. Workload will have to wait until the recurring job finishes.

Periodic Full Backup

SUSE Storage performs delta backups by default, which means that only data that was changed since the last backup is uploaded. However, when a data block in the backupstore becomes corrupted, SUSE Storage does not replace that data block with a healthy one during subsequent backup operations. Corrupted data blocks in the backupstore may cause restoration operations to fail. When a non-zero full-backup-interval parameter is set, SUSE Storage performs a full backup every full-backup-interval incremental backups. During a full backup, SUSE Storage uploads all data blocks in the volume. Data blocks that exist in the backupstore, including corrupted ones, are overwritten.

Performing a full backup might take longer and generate higher network throughput and costs than the default incremental backup.