NVIDIA Container Runtime Not Functioning Correctly in RKE2 (Missing Devices/Libraries) #1089
Comments
|
Just so that it's listed, which version of the NVIDIA Container Toolkit is installed? i.e. What is the output of |
|
Will do first thing Monday morning. |
|
Nvidia-ctk --version: nvidia-container-cli --version: NVIDIA Container Toolkit CLI version 1.17.6 cli-version: 1.17.6 Host machine is running |
|
I think I know what the issue is. The toolkit is configured to REQUIRE volume mounts to request devices for non-privileged containers: but the device plugin is configured to use envvars (the default). Installing / upgrading the device plugin and specifying should address this. Since you mention that v0.17.1 of the device plugin isn't working, do you have the logs for that case? |
|
I will implement these changes shortly. |
|
I used v0,17,1 and it worked ! @nvidia-test:/$ nvidia-smi
Thu May 22 18:16:43 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA TITAN V Off | 00000000:88:00.0 Off | N/A |
| 29% 43C P8 27W / 250W | 0MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+Sincerely thank you so much for your help. Please let me know if I should provide any other details for the benefit of others in this issue. |
Uh oh!
There was an error while loading. Please reload this page.
Desired goal
Spin up a rke2 cluster with the ability to deploy GPU enabled pods via
runtimeClassName: nvidiaIssues I've observed
After I create a pod with a pod spec of
The pod successfully deploys
Describing the pod
kubectl describe pod nvidia-test Name: nvidia-test Namespace: default Priority: 0 Runtime Class Name: nvidia Service Account: default Node: howe/10.202.0.14 Start Time: Fri, 16 May 2025 11:38:29 -0400 Labels: name=nvidia-test Annotations: cni.projectcalico.org/containerID: 7f489e51d15fb9fec8d0497caf9ec4cc276fc1cb5c9599bb6e1e1dcda0fd71ec cni.projectcalico.org/podIP: 10.42.0.65/32 cni.projectcalico.org/podIPs: 10.42.0.65/32 Status: Running IP: 10.42.0.65 IPs: IP: 10.42.0.65 Containers: nvidia-test: Container ID: containerd://2abb2775c07937a96279d255db9ce3d9104d2c8844ff35c350f7f51f5b53c1dc Image: nvidia/cuda:12.4.1-base-ubuntu22.04 Image ID: docker.io/nvidia/cuda@sha256:0f6bfcbf267e65123bcc2287e2153dedfc0f24772fb5ce84afe16ac4b2fada95 Port: <none> Host Port: <none> SeccompProfile: RuntimeDefault Command: tail Args: -f /dev/null State: Running Started: Fri, 16 May 2025 11:38:30 -0400 Ready: True Restart Count: 0 Limits: cpu: 1 memory: 1Gi nvidia.com/gpu: 1 Requests: cpu: 1 memory: 1Gi nvidia.com/gpu: 1 Environment: <none> Mounts: <none> Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: <none> QoS Class: Guaranteed Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 53s default-scheduler Successfully assigned default/nvidia-test to howe Normal Pulling 47s kubelet Pulling image "nvidia/cuda:12.4.1-base-ubuntu22.04" Normal Pulled 47s kubelet Successfully pulled image "nvidia/cuda:12.4.1-base-ubuntu22.04" in 299ms (299ms including waiting). Image size: 91769375 bytes. Normal Created 47s kubelet Created container nvidia-test Normal Started 46s kubelet Started container nvidia-testGrabbing a shell inside the container and trying
nvidia-smiMissing gpu under /dev inside container
I have no name!@nvidia-test:/dev$ ls core fd full mqueue null ptmx pts random shm stderr stdin stdout termination-log tty urandom zeroMissing nvidia-smi and other nvidia executables in /usr/bin
Below I'll post all relevant setup steps and configs.
Node configuration:
sudo apt install nvidia-headless-570-servernvidia-utils-570-servernvidia-smi from host
which nvidia-container-runtimeWorth mentioning that installing nvidia's container tookit detects almost all of the needed configurations in both /etc/nvidia-container-runtime/config.yaml and /var/lib/rancher/rke2/agent/etc/containerd/config.toml
Cluster deployments, resources, etc
kubectl describe node
kubectl get runtimeclasses
apiVersion: node.k8s.io/v1 handler: nvidia kind: RuntimeClass metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"node.k8s.io/v1","handler":"nvidia","kind":"RuntimeClass","metadata":{"annotations":{},"name":"nvidia"}} creationTimestamp: "2025-05-09T17:08:56Z" name: nvidia resourceVersion: "312392" uid: b5906bbc-2461-48a0-8ef0-78deb1f07018nvidia-device-plugin deployed via helm
Had to use 0.15.0 because it seemed 0.17.1(latest) wasn't working
I'm really at a loss of what's going on and I've spent the last few weeks troubleshooting.
Any help is greatly appreciated.
The text was updated successfully, but these errors were encountered: