Tuning etcd for Large Installations

When Rancher is used to manage a large infrastructure it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.

The etcd data set is automatically cleaned up on a five-minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see mvcc: database space exceeded errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the quota-backend-bytes setting on the etcd servers.

Example: This Snippet of the RKE2/K3s config.yaml file Increases the Keyspace Size to 5GB

RKE2/K3s config.yaml

etcd-arg:
  - "quota-backend-bytes=5368709120"

Scaling etcd Disk Performance

For RKE2 and K3s-based Rancher management clusters, achieving optimal etcd performance and stability is paramount. It is highly recommended to mount a dedicated, high-performance flash device (SSD or NVMe) at the database path. This volume should be formatted with a robust, high-performance filesystem such as XFS. This configuration isolates critical etcd I/O operations from the host operating system and any other applications, which is a foundational best practice for preventing I/O contention and ensuring low-latency writes.

For RKE2, the path is /var/lib/rancher/rke2/server/db.
For K3s, the path is /var/lib/rancher/k3s/server/db.

Do not attempt to change the default etcd data directory using etcd-arg or other configuration flags. RKE2 and K3s require the etcd database to remain at its default location. Modifying this path will break built-in cluster operations, such as snapshots and restores. To utilize dedicated storage, you must mount the device directly to the default paths listed above.

Additionally, you can follow the recommendations from the etcd docs on how to tune the disk priority on the host.

Advanced WAL Separation

Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.

If you are using dedicated devices to separate the data and wal directories, you must mount your wal disk to the default wal path and explicitly declare the wal-dir in your configuration file using that default path (as shown in the examples below). Without specifying it, the etcd process will try to manipulate the underlying wal mount with insufficient permissions.

RKE2 config.yaml

etcd-arg:
  - "data-dir=/var/lib/rancher/rke2/server/db/etcd"
  - "wal-dir=/var/lib/rancher/rke2/server/db/etcd/wal"

K3s config.yaml

etcd-arg:
  - "data-dir=/var/lib/rancher/k3s/server/db/etcd"
  - "wal-dir=/var/lib/rancher/k3s/server/db/etcd/wal"