Run GPUs on AWS Nitro Enclaves for Confidential AI

You spin up a GPU instance, load your model weights, and start training. The data is encrypted at rest and in transit. But while the model runs, everything sits in plaintext inside VRAM and system memory. Any privileged process, hypervisor tweak, or compromised host admin can snapshot that memory and walk away with your IP. Confidential computing closes that gap, and AWS Nitro Enclaves now let you bring NVIDIA GPUs inside the hardware-isolated boundary.

Nitro Enclaves carve out a dedicated, encrypted memory region from the parent EC2 instance. The enclave has no persistent storage, no external network, and no direct access to the host OS. Communication happens only through a local vsock channel you control. With the recent GPU support on G5g and P4d instances, the same isolation extends to device memory. The GPU driver runs inside the enclave; the host sees only encrypted command buffers.

The enclave image is built from a standard Dockerfile, then converted to an Enclave Image File (EIF) with the nitro-cli tooling. At launch, the parent instance passes the EIF and a set of PCR measurements to the Nitro hypervisor. The hypervisor measures the enclave boot chain into PCR0-PCR3. Your attestation document, signed by the Nitro device certificate chain, proves to downstream services that the exact code you approved is running.

The attestation document includes PCR values, a public key generated inside the enclave, and a nonce you supply. Send that document to AWS KMS with a condition key that matches your expected PCRs. KMS will only decrypt your model weights or data keys if the measurements match. This binds decryption to the exact enclave image, not just the instance type.

1. Parent instance fetches encrypted dataset from S3 using an IAM role scoped to the bucket. 2. Parent streams ciphertext over vsock to the enclave. 3. Enclave requests data key from KMS, presenting attestation document. 4. KMS verifies PCRs, returns plaintext key. 5. Enclave decrypts dataset directly into GPU memory via cuDNN. 6. Training runs; checkpoints are encrypted with the same data key before leaving the enclave. 7. Encrypted checkpoints go back over vsock to parent, then to S3.

Expect a 3-5 percent overhead on GPU kernels due to encrypted command submission. Memory bandwidth inside the enclave matches bare metal because the GPU DMA engines operate on the same physical pages. The vsock path adds latency for control-plane traffic; keep bulk data movement on the GPU side. Pin CPU threads to the same NUMA node as the GPU to avoid cross-socket traffic.

Pin the enclave image digest in your CI pipeline. Rotate the KMS key policy PCR values only after a successful rebuild and test. Monitor the Nitro hypervisor health metrics via CloudWatch; a hypervisor reboot destroys all enclaves on the host. Automate enclave restart with a systemd unit that verifies attestation before resuming training.

Clone the aws-nitro-enclaves-sdk repository. Run the gpu-attestation sample against a G5g instance. Replace the sample model with your own checkpoint, update the KMS policy with your PCR values, and run a full training epoch. Measure the overhead, verify the attestation flow, then promote the same EIF to your P4d cluster. The enclave image is portable across instance families as long as the GPU architecture matches.