This is unreleased documentation for SUSE® Virtualization v1.9 (Dev).

Harvester Cloud Provider

You can provision RKE2 clusters in Rancher using the built-in Harvester Node Driver. SUSE Virtualization provides load balancer and storage passthrough support to the guest Kubernetes cluster.

Backward Compatibility Notice

Please note a known backward compatibility issue if you’re using the Harvester Cloud Provider version v0.2.2 or higher. If your SUSE Virtualization version is below v1.2.0 and you intend to use newer RKE2 versions (i.e., >= v1.26.6+rke2r1, v1.25.11+rke2r1, v1.24.15+rke2r1), it is essential to upgrade your cluster to v1.2.0 or a higher version before proceeding with the upgrade of the guest Kubernetes cluster or Harvester Cloud Provider.

For more informmation, see the Harvester CCM & CSI Driver with RKE2 Releases section of the Support Matrix.

Deploying

Prerequisites

The Kubernetes cluster is built on top of SUSE Virtualization virtual machines.
The SUSE Virtualization virtual machines run as guest Kubernetes nodes are in the same namespace.
The SUSE Virtualization virtual machine guests' hostnames match their corresponding SUSE Virtualization virtual machine names. Guest cluster SUSE Virtualization VMs can’t have different hostnames than their SUSE Virtualization VM names when using the Harvester CSI Driver. We hope to remove this limitation in a future release.

Each SUSE Virtualization VM must have the macvlan kernel module, which is required for the LoadBalancer services of the DHCP IPAM mode.

To check if the kernel module is available, access the VM and run the following commands:

lsmod | grep macvlan
sudo modprobe macvlan

The kernel module is likely to be missing if the following occur:

$ lsmod | grep macvlan does not produce output.
$ sudo modprobe macvlan displays an error message similar to modprobe: FATAL: Module macvlan not found in directory /lib/modules/5.14.21-150400.22-default.

By default, the macvlan kernel module is not included in SUSE Linux Enterprise 15 Service Pack 4/5/6 minimal cloud images (see Issue #6418). Those images contain the kernel-default-base package, which includes only the base modules. However, the macvlan kernel driver becomes available when you install the kernel-default package.

To eliminate the need for manual intervention after the guest cluster is provisioned, build your own cloud images using the openSUSE Build Service (OBS). You must remove the kernel-default-base package and add the kernel-default package in the Minimal.kiwi file to ensure that the resulting cloud image includes the macvlan kernel module. For more information, see Custom SUSE VM Images.

Deploying to the RKE2 Cluster with Harvester Node Driver

When spinning up an RKE2 cluster using the Harvester Node Driver, select the Harvester cloud provider. The node driver will then help deploy both the CSI driver and CCM automatically.

Starting with Rancher v2.9.0, you can configure a specific folder for cloud config data using the Data directory configuration path field.

Manually deploying to the RKE2 cluster

Generate the cloud-config for the Harvester Cloud Provider.
On the RKE2 cluster creation page, go to the Cluster Configuration screen and set the value of Cloud Provider to External.
Copy and paste the cloud-init user data content to Machine Pools > Show Advanced > User Data.

Add the HelmChart CRD for harvester-cloud-provider to Cluster Configuration > Add-On Config > Additional Manifest.

You must replace <cluster-name> with the name of your cluster.

 apiVersion: helm.cattle.io/v1
 kind: HelmChart
 metadata:
   name: harvester-cloud-provider
   namespace: kube-system
 spec:
   targetNamespace: kube-system
   bootstrap: true
   repo: https://raw.githubusercontent.com/rancher/charts/dev-v2.9
   chart: harvester-cloud-provider
   version:  104.0.2+up0.2.6
   helmVersion: v3
   valuesContent: |-
     global:
       cattle:
         clusterName: <cluster-name>

To create the load balancer, add the annotation cloudprovider.harvesterhci.io/ipam: <dhcp|pool>.

Deploying to the RKE2 custom cluster (experimental)

Generate the cloud-config for the SUSE Virtualization Cloud Provider.
Create a VM in the SUSE Virtualization cluster with the following settings:
- Basics tab: The minimum requirements are 2 CPUs and 4 GiB of RAM. The required disk space depends on the VM image.
- Networks tab: Specify a network name with the format nic-<number>.
- Advanced Options tab: Copy and paste the content of the Cloud Config User Data screen.
On the Basics tab of the Cluster Configuration screen, select Harvester as the Cloud Provider and then select Create to spin up the cluster.
On the Registration tab, perform the steps required to run the RKE2 registration command on the VM.

Deploying to the K3s cluster with Harvester Node Driver (experimental)

Generate the cloud-config for the Harvester Cloud Provider.
Copy and paste the cloud-init user data content to Machine Pools > Show Advanced > User Data.

Add the following HelmChart yaml of harvester-cloud-provider to Cluster Configuration > Add-On Config > Additional Manifest.

 apiVersion: helm.cattle.io/v1
 kind: HelmChart
 metadata:
   name: harvester-cloud-provider
   namespace: kube-system
 spec:
   targetNamespace: kube-system
   bootstrap: true
   repo: https://charts.harvesterhci.io/
   chart: harvester-cloud-provider
   version: 0.2.2
   helmVersion: v3

Disable the in-tree cloud provider in the following ways:
- Click the Edit as YAML button.
- Disable servicelb and set disable-cloud-controller: true to disable the default K3s cloud controller.
  machineGlobalConfig: disable: - servicelb disable-cloud-controller: true
- Add cloud-provider=external to use the Harvester Cloud Provider.
  machineSelectorConfig: - config: kubelet-arg: - cloud-provider=external protect-kernel-defaults: false

With these settings in place a K3s cluster should provision successfully while using the external cloud provider.

Generating the cloud-config for the Harvester Cloud Provider

The Harvester Cloud Provider requires a cloud-config file to connect to the remote SUSE Virtualization cluster (for example, to query virtual machine information or allocate load balancers). You can generate this file using either the API endpoint or a bash script.

Support for the bash script method will be deprecated in a future release. Use the API endpoint to ensure long-term compatibility.

API
Bash Script

You can send POST and GET requests to the SUSE Virtualization API endpoint /v1/harvester/kubeconfig using an admin bearer token.

==== Request parameters

Parameter Type Description Example

Parameter	Type	Description	Example
`namespace`	String	Target Kubernetes namespace	`gc-test`
`serviceAccountName`	String	Service account name	`gc4`
`clusterRoleName`	String	ClusterRole to bind to the service account (optional)	`harvesterhci.io:cloudprovider` (only supported value)
`outputFormat`	String	Output format	`yaml` (only supported value)

namespace

String

Target Kubernetes namespace

gc-test

serviceAccountName

String

Service account name

gc4

clusterRoleName

String

ClusterRole to bind to the service account (optional)

harvesterhci.io:cloudprovider (only supported value)

outputFormat

String

Output format

yaml (only supported value)

clusterRoleName: SUSE Virtualization uses harvesterhci.io:cloudprovider by default if this field is left empty.
outputFormat: Setting this to yaml retrieves the cloud-init user data. Specifying any other value or leaving the field empty returns the raw kubeconfig file.

==== POST request

Add -k/--insecure to the curl command only if your SUSE Virtualization endpoint uses a self-signed certificate.

 curl -X POST \
  -H "Authorization: Bearer token-abcde:..." \
  -H "Content-Type: application/json" \
  -d '{"namespace": "gc-test", "serviceAccountName": "gc4", "outputFormat": "yaml"}' \
  "https://<vip>/v1/harvester/kubeconfig"

==== POST response

########## cloud-init user data ############
write_files:
- encoding: b64
  content: <BASE64_CONTENT>
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'
- encoding: b64
  content: <BASE64_CONTENT>
  owner: root:root
  path: /var/lib/rancher/rke2/etc/config-files/cloud-provider-config
  permissions: '0644'

==== GET request

Use a single ampersand (&) to separate query parameters.

curl -X GET \
  -H "Authorization: Bearer token-abcde:..." \
  "https://<vip>/v1/harvester/kubeconfig?namespace=gc-test&serviceAccountName=gc4&outputFormat=yaml"

==== GET response

The API response automatically includes cloud-init configurations for both legacy and new paths. Before applying this configuration, remove the block that does not apply to your environment.

########## cloud-init user data ############
write_files:
- encoding: b64
  content: <BASE64_CONTENT>
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'
- encoding: b64
  content: <BASE64_CONTENT>
  owner: root:root
  path: /var/lib/rancher/rke2/etc/config-files/cloud-provider-config
  permissions: '0644'

The script requires kubectl and jq to interact with the SUSE Virtualization cluster, and functions only when given access to the cluster’s kubeconfig file.

Generate the cloud-config data using the generate_addon.sh script.

curl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s <serviceaccount name> <namespace>

Copy the generated data to every node.
- Legacy path: /etc/kubernetes/cloud-config
- RKE2 default path (v1.9.0 and later): /var/lib/rancher/rke2/etc/config-files/cloud-provider-config

You can find the kubeconfig file on any SUSE Virtualization management node at the following path: /etc/rancher/rke2/rke2.yaml. Before using the kubeconfig file, you must replace the IP address in the server: field with your cluster’s VIP address.

Example of content:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <redacted>
    server: https://127.0.0.1:6443
  name: default
# ...

You must specify the namespace in which the guest cluster will be created.

Example of output:

########## cloud config ############
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <CACERT>
    server: https://HARVESTER-ENDPOINT/k8s/clusters/local
  name: local
contexts:
- context:
    cluster: local
    namespace: default
    user: harvester-cloud-provider-default-local
  name: harvester-cloud-provider-default-local
current-context: harvester-cloud-provider-default-local
kind: Config
preferences: {}
users:
- name: harvester-cloud-provider-default-local
  user:
    token: <TOKEN>

########## cloud-init user data ############
write_files:
- encoding: b64
  content: <CONTENT>
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'

In newer RKE2 versions (such as v1.33.11), the default cloud-config path is /var/lib/rancher/rke2/etc/config-files/cloud-provider-config. You must ensure that the cloudConfigPath value matches the exact file location you write to.

Depending on your setup, choose one of the following approaches:

Use the RKE2 default path: In the write_files entry in your cloud-init configuration, change the value of path to /var/lib/rancher/rke2/etc/config-files/cloud-provider-config.
Keep the legacy path: In the Rancher UI, set .spec.rkeConfig.chartValues.harvester-cloud-provider.cloudConfigPath to /etc/kubernetes/cloud-config.

Upgrade Cloud Provider

Upgrade RKE2

The cloud provider can be upgraded by upgrading the RKE2 version. You can upgrade the RKE2 cluster via the Rancher UI as follows:

Click ☰ > Cluster Management.
Find the guest cluster that you want to upgrade and select ⋮ > Edit Config.
Select Kubernetes Version.
Click Save.

Upgrade K3s

K3s upgrade cloud provider via the Rancher UI, as follows:

Click ☰ > K3s Cluster > Apps > Installed Apps.
Find the cloud provider chart and select ⋮ > Edit/Upgrade.
Select Version.
Click Next > Update.

The upgrade process for a single-node guest cluster may stall when the new harvester-cloud-provider pod is stuck in the Pending state. This issue is caused by a section in the harvester-cloud-provider deployment that describes the rolling update strategy. Specifically, the default value conflicts with the podAntiAffinity configuration in single-node clusters.

For more information, see this GitHub issue comment. To address the issue, manually delete the old harvester-cloud-provider pod. You might need to do this multiple times until the new pod can be successfully scheduled.

Load balancer support

Once you’ve deployed the Harvester Cloud Provider, you can leverage the Kubernetes LoadBalancer service to expose a microservice within the guest cluster to the external world. Creating a Kubernetes LoadBalancer service assigns a dedicated SUSE Virtualization load balancer to the service, and you can make adjustments through the Add-on Config within the Rancher UI.

IPAM

SUSE Virtualization’s built-in load balancer offers both DHCP and Pool modes, and you can configure it by adding the annotation cloudprovider.harvesterhci.io/ipam: $mode to its corresponding service. Starting from Harvester Cloud Provider >= v0.2.0, it also introduces a unique Share IP mode. A service shares its load balancer IP with other services in this mode.

DHCP: A DHCP server is required. The SUSE Virtualization load balancer will request an IP address from the DHCP server.

Starting with Rancher v2.15.1, you can select a VM network when creating a LoadBalancer service using the UI. This enables explicit binding of the virtual IP to the correct network interface. If you do not select a VM network, the load balancer uses the default interface.

In earlier Rancher versions (v2.12.x, v2.13.x, and v2.14.x), you can achieve the same result by adding the following annotations to the Service manifest:
- cloudprovider.harvesterhci.io/ipam: "dhcp"
- cloudprovider.harvesterhci.io/network: "default/mgmt-vlan1"
Pool: A pre-configured IP pool is required. The SUSE Virtualization load balancer controller allocates an IP for the load balancer service according to the IP pool selection policy. You can create IP pools using either the SUSE Virtualization UI or the Rancher UI. For more information, see Best practices.

Starting with Rancher v2.15.1, you can select a VM network when creating a LoadBalancer service using the UI. This enables explicit binding of the load balancer to the correct network interface. If you do not select a VM network (specifically, the cloudprovider.harvesterhci.io/network is empty), the load balancer controller automatically determines the network to be used.

On earlier Rancher versions (v2.12.x, v2.13.x, and v2.14.x), you can achieve the same result by adding the following annotations to the Service manifest:
- cloudprovider.harvesterhci.io/ipam: "ippool"
- cloudprovider.harvesterhci.io/network: "default/mgmt-vlan1"
  
  When a guest cluster uses multiple networks, or when multiple guest clusters with distinct networks share a single namespace, configuring the correct network parameters is critical. For information about how the system automatically determines the network, see Guest cluster load balancer network resolution.
Share IP: When creating a new load balancer service (secondary service), you can reuse the IP address of an existing service (primary service). To specify the primary service, add the annotation cloudprovider.harvesterhci.io/primary-service: $primary-service-name to the secondary service.

This mode has two known limitations:
- Services sharing the same IP address cannot use identical ports.
- Secondary services cannot share their IP with additional services.

Modifying the IPAM mode of an existing service is not supported. To use a different IPAM mode, create a new load balancer service.

See Guest cluster load balancer IP is not reachable.

Asymmetric network topology

The network dropdown list on the UI displays only networks assigned to the exact same interface position across all cluster nodes.

Example:

Network-Interface Mapping UI Behavior Node A Node B Displayed Networks

Network-Interface Mapping	UI Behavior	Node A	Node B	Displayed Networks
Identical mapping across all nodes	All networks are displayed	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-101`	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-101`	`mgmt` and `net-101`
Network in different interface positions across nodes	Network is hidden	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-101`	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-102`<br/>`enp3s0` → `net-101`	`mgmt`
Network absent on some nodes	Network is hidden	`enp1s0` → `mgmt`	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-101`	`mgmt`
Swapped interface mapping order	Only matching networks are displayed	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-101`<br/>`enp3s0` → `net-102`	`enp1s0` → `mgmt`<br/>`enp2s0` → `net-102`<br/>`enp3s0` → `net-101`	`mgmt`

Identical mapping across all nodes

All networks are displayed

enp1s0 → mgmt `enp2s0` → net-101

mgmt and net-101

Network in different interface positions across nodes

Network is hidden

enp1s0 → mgmt `enp2s0` → net-101

enp1s0 → mgmt `enp2s0` → net-102 `enp3s0` → net-101

mgmt

Network absent on some nodes

Network is hidden

enp1s0 → mgmt

enp1s0 → mgmt `enp2s0` → net-101

mgmt

Swapped interface mapping order

Only matching networks are displayed

enp1s0 → mgmt `enp2s0` → net-101 `enp3s0` → net-102

enp1s0 → mgmt `enp2s0` → net-102 `enp3s0` → net-101

mgmt

If VM network interfaces are attached in different orders across nodes, reconfigure the network interface order in the machine pool settings to allow Rancher and RKE2 to reprovision the guest cluster virtual machines.

Limitations

Default load balancer provider: kube-vip is selected by default on the UI. If you disable kube-vip and use an alternative provider, refer to that provider’s documentation for configuration instructions.
Pre-condition for secondary network load balancing: The secondary network interface of each guest cluster node must have a valid IP address and route. Otherwise, the load balancer cannot route traffic. Verifying this interface configuration should be the first step when troubleshooting issues related to secondary network load balancers.
Load balancer network changes: Delete and recreate the load balancer service if you require changes to the load balancer network. Modifying the network annotation on an existing service may cause unexpected behavior and is not supported.
Incorrect network annotation: The load balancer may fail to obtain an IP address or become unreachable if you directly configure the cloudprovider.harvesterhci.io/network annotation and specify a network that is either invalid or exhibits an asymmetric network topology. Because webhook validation is not performed on this annotation, select the target network on the UI instead.
Secondary network load balancer traffic isolation: Incoming traffic arrives on the secondary network interface and undergoes NAT to the pod network. Consequently, only workloads listening on the pod network can receive load balancer traffic. Workloads configured to listen exclusively on the secondary network interface cannot. Full traffic isolation is currently unsupported.

Health checks

Beginning with Harvester Cloud Provider v0.2.0, additional health checks of the LoadBalancer service within the guest Kubernetes cluster are no longer necessary. Instead, you can configure liveness and readiness probes for your workloads. Consequently, any unavailable pods will be automatically removed from the load balancer endpoints to achieve the same desired outcome.

Automatic cleanup

When you delete a guest cluster that has the Harvester Cloud Provider enabled, SUSE Virtualization automatically cleans up all associated LoadBalancer resources. This offers the following key benefits:

Resource management: Automatic cleanup prevents orphaned load balancers from consuming IP addresses after the guest cluster is deleted.
Zero manual intervention: The lifecycle of the load balancer is tied directly to the lifecycle of the guest cluster.

Stale cloud credentials after cluster registration

If you remove a SUSE Virtualization cluster from Rancher and later re-register the same cluster, Rancher may retain stale cloud credentials that reference the previous management cluster ID. This mismatch causes provisioning of guest clusters (such as RKE2 and K3s clusters) to fail.

When this issue occurs, the system logs an error similar to the following:

clusters.management.cattle.io "<old-cluster-id>" not found

The failure occurs because the existing cloud credential still points to the original SUSE Virtualization cluster ID, which no longer exists after the re-registration process.

To mitigate the issue, perform the following workaround:

On the Rancher UI, go to ☰ → Cluster Management → Cloud Credentials.
Delete the stale cloud credential associated with the removed SUSE Virtualization cluster.
Create a new cloud credential using the updated SUSE Virtualization cluster details.
Provision the guest cluster again.

Related issue: #53642