This guide provides instructions for deploying and managing GPU workloads with RDMA connectivity on Oracle Cloud Infrastructure Kubernetes Engine (OKE). OKE is a fully-managed, scalable, and highly available Kubernetes service that enables you to deploy containerized applications to the cloud.
- Ubuntu 22.04
- Oracle Linux 8 (except for the GPU & RDMA worker pool)
The following policies are required. The OCI Resource Manager stack will create them for you if you have the necessary permissions. If you don't have the permissions, please refer to the policy documentation below.
- Policy Configuration for Cluster Creation and Deployment
- Creating a Dynamic Group and a Policy for Self-Managed Nodes
You will need a CPU pool and a GPU pool. The OCI Resource Manager stack deploys a system worker pool by default, and you can choose to deploy additional CPU/GPU worker pools.
You can use the following images for both CPU and GPU pools.
Note
The GPU image has the GPU drivers pre-installed.
You can use the instructions here for importing the images below to your tenancy.
You can easily deploy the cluster with the Deploy to Oracle Cloud button below, which uses OCI Resource Manager. If you prefer deploying with Terraform locally, you can use the templates in the terraform directory.
You can access the cluster locally by downloading the kubeconfig file.
Alternatively, the template deploys an operator instance with the kubeconfig pre-configured and tools like Helm and k9s pre-installed. You can find the SSH command to access the operator node under the Application information tab in the OCI Resource Manager stack.
Wait until all nodes are ready in the cluster:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.0.103.73 Ready <none> 2d23h v1.31.1
10.0.127.206 Ready node 2d3h v1.31.1
10.0.127.32 Ready node 2d3h v1.31.1
10.0.83.93 Ready <none> 2d23h v1.31.1
10.0.96.82 Ready node 2d23h v1.31.1For more information, see Adding a Service Account Token.
kubectl -n kube-system create serviceaccount kubeconfig-sa
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubeconfig-sa
kubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/service-account/oke-kubeconfig-sa-token.yaml
TOKEN=$(kubectl -n kube-system get secret oke-kubeconfig-sa-token -o jsonpath='{.data.token}' | base64 --decode)
kubectl config set-credentials kubeconfig-sa --token=$TOKEN
kubectl config set-context --current --user=kubeconfig-saTo use the RDMA interfaces on the host in your pods, include the following sections in your manifests:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
volumes:
- { name: devinf, hostPath: { path: /dev/infiniband }}
- { name: shm, emptyDir: { medium: Memory, sizeLimit: 32Gi }}securityContext:
privileged: true
capabilities:
add: [ "IPC_LOCK" ] volumeMounts:
- { mountPath: /dev/infiniband, name: devinf }
- { mountPath: /dev/shm, name: shm }Here's a simple example. You can also look at the NCCL test manifests in the repo here.
apiVersion: v1
kind: Pod
metadata:
name: rdma-test-pod-1
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
volumes:
- { name: devinf, hostPath: { path: /dev/infiniband }}
- { name: shm, emptyDir: { medium: Memory, sizeLimit: 32Gi }}
restartPolicy: OnFailure
containers:
- image: oguzpastirmaci/mofed-perftest:5.4-3.6.8.1-ubuntu20.04-amd64
name: mofed-test-ctr
securityContext:
privileged: true
capabilities:
add: [ "IPC_LOCK" ]
volumeMounts:
- { mountPath: /dev/infiniband, name: devinf }
- { mountPath: /dev/shm, name: shm }
resources:
requests:
cpu: 8
ephemeral-storage: 32Gi
memory: 2Gi
command:
- sh
- -c
- |
ls -l /dev/infiniband /sys/class/net
sleep 1000000Kueue and MPI Operator are required for running the optional NCCL tests.
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.6.0/deploy/v2beta1/mpi-operator.yaml
helm install kueue oci://registry.k8s.io/kueue/charts/kueue --version="0.14.4" --create-namespace --namespace=kueue-systemImportant
The NCCL/RCCL parameters differ between GPU shapes. Ensure that you use the correct manifest for your specific bare metal GPU shape.
kubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.GB200-v2.4.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.GB200.4.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.B200.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.H200.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.H100.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.A100-v2.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU4.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/nccl-tests/kueue/BM.GPU.B4.8.yamlkubectl apply -f https://raw.githubusercontent.com/oracle-quickstart/oci-hpc-oke/main/manifests/rccl-tests/kueue/BM.GPU.MI300X.8.yamlThe initial container image pull may take some time. Once the launcher pod nccl-test-launcher-XXXXX starts running, you can check its logs for the NCCL test results.
Waiting for workers to be ready...
All workers are ready!
Warning: Permanently added '[nccl-test-worker-1.nccl-test.default.svc]:2222' (ED25519) to the list of known hosts.
Warning: Permanently added '[nccl-test-worker-0.nccl-test.default.svc]:2222' (ED25519) to the list of known hosts.
# nThread 1 nGpus 1 minBytes 1073741824 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 88 on inst-fufd1-oke-rdma device 0 [0000:0f:00] NVIDIA A100-SXM4-40GB
# Rank 1 Group 0 Pid 89 on inst-fufd1-oke-rdma device 1 [0000:15:00] NVIDIA A100-SXM4-40GB
# Rank 2 Group 0 Pid 90 on inst-fufd1-oke-rdma device 2 [0000:51:00] NVIDIA A100-SXM4-40GB
# Rank 3 Group 0 Pid 91 on inst-fufd1-oke-rdma device 3 [0000:54:00] NVIDIA A100-SXM4-40GB
# Rank 4 Group 0 Pid 92 on inst-fufd1-oke-rdma device 4 [0000:8d:00] NVIDIA A100-SXM4-40GB
# Rank 5 Group 0 Pid 93 on inst-fufd1-oke-rdma device 5 [0000:92:00] NVIDIA A100-SXM4-40GB
# Rank 6 Group 0 Pid 94 on inst-fufd1-oke-rdma device 6 [0000:d6:00] NVIDIA A100-SXM4-40GB
# Rank 7 Group 0 Pid 95 on inst-fufd1-oke-rdma device 7 [0000:da:00] NVIDIA A100-SXM4-40GB
# Rank 8 Group 0 Pid 88 on inst-aqu5j-oke-rdma device 0 [0000:0f:00] NVIDIA A100-SXM4-40GB
# Rank 9 Group 0 Pid 89 on inst-aqu5j-oke-rdma device 1 [0000:15:00] NVIDIA A100-SXM4-40GB
# Rank 10 Group 0 Pid 90 on inst-aqu5j-oke-rdma device 2 [0000:51:00] NVIDIA A100-SXM4-40GB
# Rank 11 Group 0 Pid 91 on inst-aqu5j-oke-rdma device 3 [0000:54:00] NVIDIA A100-SXM4-40GB
# Rank 12 Group 0 Pid 92 on inst-aqu5j-oke-rdma device 4 [0000:8d:00] NVIDIA A100-SXM4-40GB
# Rank 13 Group 0 Pid 93 on inst-aqu5j-oke-rdma device 5 [0000:92:00] NVIDIA A100-SXM4-40GB
# Rank 14 Group 0 Pid 94 on inst-aqu5j-oke-rdma device 6 [0000:d6:00] NVIDIA A100-SXM4-40GB
# Rank 15 Group 0 Pid 96 on inst-aqu5j-oke-rdma device 7 [0000:da:00] NVIDIA A100-SXM4-40GB
NCCL version 2.25.1+cuda12.8
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
1073741824 268435456 float sum -1 10776 99.64 186.83 0 10781 99.60 186.75 0
2147483648 536870912 float sum -1 21287 100.88 189.15 0 21299 100.82 189.05 0
4294967296 1073741824 float sum -1 42381 101.34 190.02 0 42364 101.38 190.09 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 188.648
#- Adding SSH keys to worker nodes
- Deploying the Monitoring Stack manually
- Running GPU & RDMA active health checks
- Running GPU & RDMA passive health checks
- Using RDMA Network Locality When Running Workloads on OKE
- Running PyTorch Jobs on OKE Using Host Network with RDMA
- Using Cluster Autoscaler with Cluster Networks
- Importing Container Images from OCI File Storage Service Using Skopeo
This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide.
Please consult the security guide for our responsible security vulnerability disclosure process.