GPU 运算符

部署 NVIDIA 运算符

NVIDIA 运算符允许 Kubernetes 集群的管理员像管理处理器一样管理 GPU。它包含了让 Pod 操作 GPU 所需的一切。

主机操作系统要求

为了正确地将 GPU 暴露给 Pod，NVIDIA 内核驱动程序和 libnvidia-ml 库必须在主机操作系统中正确安装。NVIDIA 运算符可以在某些操作系统上自动安装驱动程序和库。有关支持的操作系统版本的信息，请参考 NVIDIA 文档。在您的主机操作系统上安装 NVIDIA 组件超出了本文档的范围，请参考 NVIDIA 文档以获取说明。

如果内核驱动程序正确安装，以下三个命令应返回正确的输出：

lsmod | grep nvidia 返回 nvidia 内核模块的列表。例如：

nvidia_uvm           2129920  0
nvidia_drm            131072  0
nvidia_modeset       1572864  1 nvidia_drm
video                  77824  1 nvidia_modeset
nvidia               9965568  2 nvidia_uvm,nvidia_modeset
ecc                    45056  1 nvidia

cat /proc/driver/nvidia/version 返回驱动程序的 NVRM 和 GCC 版本。例如：

NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  555.42.06  Release Build  (abuild@host)  Thu Jul 11 12:00:00 UTC 2024
GCC version:  gcc version 7.5.0 (SUSE Linux)

find /usr/ -iname libnvidia-ml.so 返回 libnvidia-ml.so 库的路径。例如：
```
/usr/lib64/libnvidia-ml.so
```
此库由 Kubernetes 组件用于与内核驱动程序交互。

运算符安装

一旦操作系统准备就绪并且 RKE2 正在运行，使用以下 yaml 清单安装 GPU 运算符。

v25.3.x
v25.10.x
v26.3.x
v26.3.x with NRI

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v25.3.4
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
      - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
        value: "false"
      - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
        value: "true"
    devicePlugin:
      env:
      - name: DEVICE_LIST_STRATEGY
        value: volume-mounts

环境变量 ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED、ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS 和 DEVICE_LIST_STRATEGY 是正确隔离 GPU 资源所必需的，如本 NVIDIA 文档中所述。

The NVIDIA operator restarts containerd with a hangup call which restarts RKE2

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v25.10.1
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock

NVIDIA GPU 运算符 v25.10.x 使用容器设备接口 (CDI) 规范，这简化了操作：我们不需要传递额外的环境变量来遵守安全要求，工作负载也不再需要传递 runtimeClassName: nvidia。

NVIDIA 运算符通过挂起调用重启 containerd，从而重启 RKE2。

There are two installation options available.

If drivers and libraries are pre-installed or you are using a supported operating system by nvidia, please use the following manifest:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v26.3.2
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock

If your operating system vendor supplies a compatible driver image, you can use the driver value field to point to it. For example, in SLES 16.0, you can use the following manifest:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v26.3.2
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
    driver:
      repository: registry.suse.com/third-party/nvidia
      usePrecompiled: true
      version: 595 # This depends on the nvidia driver that works with your GPU architecture

Node Resource Interface (NRI) specification is a pluggable extension mechanism built into container runtimes like containerd and CRI-O that allows custom plugins to intercept container lifecycle events on a node. It is considered the future integration mechanism for GPUs.

NVIDIA considers NRI as experimental

If you want to try it out, please use the following manifest:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v26.3.1
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    cdi:
      nriPluginEnabled: true

Version Gate

NRI requires containerd 2.1. Containerd 2.1 is available as of September 2025 releases: v1.31.13+rke2r1, v1.32.9+rke2r1, v1.33.5+rke2r1, v1.34.1+rke2r1

大约一分钟后，您可以执行以下检查以验证一切是否按预期工作：

假设驱动程序和 libnvidia-ml.so 库之前已安装，请检查运算符是否正确检测到它们：
```
kubectl get node $NODENAME -o jsonpath='{.metadata.labels}' |  grep "nvidia.com"
```
您应该看到指定驱动程序和 GPU 的标签（例如 nvidia.com/gpu.machine 或 nvidia.com/cuda.driver.major）。
检查 GPU 是否被 nvidia-device-plugin-daemonset 作为节点中的可分配资源添加：
```
kubectl get node $NODENAME -o jsonpath='{.status.allocatable}'
```
您应该看到 "nvidia.com/gpu": 后跟节点中的 GPU 数量。
检查容器运行时二进制文件是否存在（它由 nvidia-container-toolkit-daemonset 安装）：
```
ls /usr/local/nvidia/toolkit/nvidia-container-runtime
```
验证 containerd 配置是否已更新以包含 NVIDIA 容器运行时：
```
grep nvidia /var/lib/rancher/rke2/agent/etc/containerd/config.toml
```

运行一个 Pod 以验证 GPU 资源是否可以成功调度到 Pod 上，并且 Pod 可以检测到它。

apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark
  namespace: default
spec:
  restartPolicy: OnFailure
  # runtimeClassName: nvidia <== Only needed for v25.3.x
  containers:
  - name: cuda-container
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    resources:
      limits:
        nvidia.com/gpu: 1

版本门控

自 2024 年 10 月发布以来可用：v1.28.15+rke2r1，v1.29.10+rke2r1，v1.30.6+rke2r1，v1.31.2+rke2r1。

RKE2 现在将使用 PATH 查找替代容器运行时，此外还会检查容器运行时软件包使用的默认路径。为了使用此功能，您必须修改 RKE2 服务的 PATH 环境变量，以添加包含容器运行时二进制文件的目录。

建议您修改这两个环境文件中的一个：

/etc/default/rke2-server # 或 rke2-agent
/etc/sysconfig/rke2-server # 或 rke2-agent

此示例将在 PATH 中添加 /etc/default/rke2-server：

对 PATH 的更改应谨慎进行，以避免将不受信任的二进制文件放置在以 root 身份运行的服务的路径中。

echo PATH=$PATH >> /etc/default/rke2-server