This is unreleased documentation for SUSE® Virtualization v1.9 (Dev).

Longhorn V2 Data Engine

The Longhorn V2 Data Engine harnesses the power of the Storage Performance Development Kit (SPDK) to significantly reduce I/O latency while boosting IOPS and throughput. The result is a high-performance storage solution that is capable of meeting diverse workload demands.

The Longhorn V2 Data Engine is a Technical Preview feature. Test the feature extensively before using it in production environments.

Prerequisites

Every node with an active Longhorn V2 Data Engine requires the following dedicated resources:

  • 2 CPU cores for use by the Longhorn instance-manager pod

  • 2 GiB RAM (allocated as 1024 × 2 MiB huge pages)

  • At least one local NVMe disk for data storage

Limitations

The Longhorn V2 Data Engine currently does not support the following operations:

  • Backing image creation and usage

  • Volume encryption

  • SUSE Virtualization does not automatically coordinate Longhorn V2 SPDK CPU assignment with virtual machine CPU pinning. Configure SPDK CPU assignment before running CPU-pinned virtual machines on nodes where the Longhorn V2 Data Engine is enabled. For more information, see CPU core configuration with CPU-pinned virtual machines.

Usage

The Longhorn V2 Data Engine is only available for newly created volumes and images. Existing volumes, virtual machine images and virtual machine root volumes will continue to use the V1 Data Engine.

  1. On the SUSE Virtualization UI, go to Advanced → Settings.

  2. Set longhorn-v2-data-engine-enabled to true.

    SUSE Virtualization automatically loads the kernel modules required by the Longhorn V2 Data Engine, and attempts to allocate 1024 × 2 MiB-sized huge pages (for example, 2 GiB of RAM) on all nodes.

    Changing this setting automatically restarts RKE2 on all nodes but does not affect running virtual machine workloads.

    If you encounter error messages that include the phrase "not enough hugepages-2Mi capacity", allow some time for the error to be resolved. If the error persists, reboot the affected nodes.

    If you do not plan to use hugepages or want to allocate a custom memory size, adjust the longhorn-v2-data-engine-hugepage-enabled and longhorn-v2-data-engine-memory-size settings, preferably before enabling the Longhorn V2 Data Engine. Disabling hugepages is currently supported only for Longhorn V2 disks that use the AIO bdev driver (the default in SUSE Virtualization v1.9.0). Do not disable hugepages when using the NVMe or virtio bdev driver.

    To disable the Longhorn V2 Data Engine on specific nodes (for example, nodes with less processing and memory resources), go to the Hosts screen and add the following label to the target nodes:

    • label: node.longhorn.io/disable-v2-data-engine

    • value: true

  3. Go to the Hosts screen, and then add extra disks to each node as described in Multi-disk Management.

    Set the Provisioner of each extra disk to Longhorn V2 (CSI).

    The default disk driver for newly added Longhorn V2 disks in SUSE Virtualization is aio. This driver uses a Linux block device path and avoids the SPDK NVMe VFIO path.

  4. Go to Advanced → Storage Classes, and then add a new StorageClass.

    Set the Provisioner to Longhorn V2 (CSI).

  5. Use the new StorageClass when creating the following:

    • Volumes (either on the Volumes screen or during virtual machine creation)

    • Images (on the Images screen)

      Volumes and images created using the new StorageClass are backed by the Longhorn V2 Data Engine.

CPU core configuration with CPU-pinned virtual machines

SUSE Storage supports assigning SPDK CPU cores using the Kubernetes CPU Manager. However, the SUSE Virtualization UI does not yet expose these controls. If you run CPU-pinned virtual machines on nodes with the Longhorn V2 Data Engine enabled, you must configure these settings directly on the SUSE Storage UI.

By default, the Longhorn V2 Data Engine uses the data-engine-cpu-mask setting, which binds spdk_tgt to fixed CPU IDs. The default value of this setting is {"v2":"0x3"}, which maps to CPU IDs 0 and 1.

When the SUSE Virtualization CPU Manager is enabled, CPU-pinned virtual machine pods receive exclusive CPU allocations from kubelet. An overlap between these exclusive CPUs and the Longhorn V2 CPU mask can prevent the instance-manager pod from assigning masked CPUs to its cpuset. When this occurs, spdk_tgt fails to start with the following error:

EAL: PANIC in rte_eal_init():
Cannot set affinity

To avoid fixed CPU ID conflicts, configure the SUSE Storage setting data-engine-number-of-cpu-cores before enabling the V2 Data Engine on nodes with CPU-pinned virtual machines. When the value of this global setting is greater than 0, SUSE Storage validates that every node reports the static CPU Manager policy. SUSE Storage then ignores the data-engine-cpu-mask setting, requests the configured number of full CPU cores for each V2 instance-manager pod, and relies on the CPU Manager for dynamic assignment of exclusive CPUs.

SUSE Storage does not support mixed CPU Manager configurations for data-engine-number-of-cpu-cores. If the CPU Manager is enabled on only a subset of nodes, SUSE Storage rejects any V2 core allocation greater than 0. You must enable the CPU Manager on all nodes before configuring this setting.

  1. Enable the CPU Manager on all cluster nodes.

  2. Specify the CPU core count for the V2 instance-manager pod.

    SUSE Storage recommends at least 2 CPU cores for the data-engine-number-of-cpu-cores setting. Ensure each node running the V2 Data Engine has sufficient allocatable CPU capacity for both the selected core count and any CPU-pinned virtual machines.

    The SUSE Virtualization setting instance-manager-resources is _not required for CPU Manager-based SPDK CPU assignment. If SUSE Storage’s CPU reservation for V2 instance-manager pods is higher than the configured SPDK CPU core count, SUSE Storage rounds the reservation up to full CPU cores, and the pod may receive more exclusive CPUs than the data-engine-number-of-cpu-cores value specifies.

  3. If Longhorn V2 volumes are already attached, schedule a maintenance window, stop all virtual machines that use those volumes, and detach the volumes.

    SUSE Storage automatically recreates idle V2 instance-manager pods to apply the new CPU settings.

  4. Configure the data-engine-number-of-cpu-cores setting.

    In the following example, 2 CPU cores are allocated to the SPDK target daemon in each V2 instance-manager pod.

    kubectl -n longhorn-system patch settings.longhorn.io data-engine-number-of-cpu-cores --type=merge -p '{"value":"{\"v2\":\"2\"}"}'

  5. Verify the effective CPU set after the V2 instance-manager pods are recreated.

    kubectl -n longhorn-system exec <v2-instance-manager-pod> -- awk '/^Cpus_allowed_list:/ {print $2}' /proc/self/status

    The allowed CPU list should contain at least the configured number of CPUs. If Longhorn’s guaranteed instance-manager CPU reservation is higher, the list can contain more CPUs.

Migrating Longhorn V2 Disks from NVMe to AIO

Before migrating a disk, verify that all affected volumes are healthy and that the remaining Longhorn V2 disks have sufficient free space to accommodate rebuilt replicas during disk removal.

In SUSE Virtualization v1.9.0 and later, newly created Longhorn V2 disks use the aio disk driver by default. Existing Longhorn V2 disks retain their current driver configuration until you manually remove and add them again.

  1. Remove the affected Longhorn V2 disk.

  2. Add the Longhorn V2 disk again.

  3. Verify that the migrated disk uses aio.

    kubectl -n longhorn-system get blockdevices.harvesterhci.io BLOCKDEVICE_NAME -o yaml

    The matching BlockDevice custom resource should have the following values:

        spec:
      provisioner:
        longhorn:
      state: Active

Known issues

Stalled I/O operations on ARM platforms

On ARM64 platforms, Longhorn V2 disks that use the SPDK NVMe bdev driver can experience stalled I/O operations. For more information, see issue #10710.

For information about the workaround, see Migrating Longhorn V2 Disks from NVMe to AIO.

Upgrading from SUSE Virtualization v1.4.x

SUSE Virtualization v1.4.x, which uses SUSE Storage v1.7.x, is unable to live-migrate virtual machines with V2 volumes attached. Moreover, the V2 Data Engine cannot be used for virtual machine images and boot volumes. These limitations do not exist in SUSE Virtualization v1.5.0 and later versions, which use SUSE Storage v1.8.1 and later versions. However, this applies only to volumes and images that are created after SUSE Virtualization is upgraded.

In V2 StorageClasses created using SUSE Virtualization v1.4.x, the migratable option is set to false. Like all other StorageClass properties, this cannot be changed once set. Similarly, V2 volumes created using SUSE Virtualization v1.4.x remain non-migratable after the upgrade. If you used the V2 Data Engine in SUSE Virtualization v1.4.x and later upgrade to SUSE Virtualization v1.5, you must create a new V2 StorageClass. The migratable option is set to true by default, so volumes and images created using this new V2 StorageClass can be live-migrated.

  • If you are using the SPDK AIO bdev driver (specifically, if disks were added using /dev/sd* device paths), V2 volumes created before the upgrade become unusable and cannot be recovered after the upgrade. For more information, see issue #10461.

  • If you are using the SPDK NVMe bdev driver (specifically, if disks were added using /dev/nvme* device paths), V2 volumes created before the upgrade will still function. However, these volumes will continue to use the SUSE Storage v1.7.x engine and remain non-migratable. You can export the data and create new migratable volumes, if necessary.

  • All virtual machines with attached V2 volumes must be stopped before you start the upgrade. Active V2 volumes will cause the upgrade process to stall during the "upgrading system services" phase. The apply-manifests pod logs will show repeated messages similar to the following:

    instance-manager (aio)(v2) (image=longhornio/longhorn-instance-manager:v1.8.1) state is not running on node harvester-node-0, will retry...

    Stopping all Virtual Machines that are using V2 volumes will allow the upgrade to proceed.

If you are using the SPDK NVMe bdev driver (specifically, if disks were added using /dev/nvme* device paths) and non-migratable V2 volumes are attached to existing virtual machines, you can transition to live-migratable volumes by performing the following steps:

  1. Stop the virtual machines.

  2. Export each attached V2 volume to an image that uses the new V2 StorageClass (with the migratable option is set to true).

  3. Once the volumes are exported to images, edit the virtual machine and perform the following actions on the Volumes tab:

    • Remove the existing V2 volumes.

    • Add the images that were created from the exported volumes.

  4. Start the virtual machines.

    This step may take some time, depending on the amount of data to copy.