Skip to main content

DGX Spark environment

What's on an NVIDIA DGX Spark: DGX OS, CUDA and nvcc, a docker command that runs rootless Podman with GPU access, a vLLM example and the memory limit.

On this page

DGX Spark machines are coming soon. This page describes the environment they'll have, so you can prepare your workflow.

The NVIDIA DGX Spark is a compact computer built around the GB10 Grace Blackwell superchip: a 20-core Arm CPU and a Blackwell GPU sharing 128 GB of LPDDR5x unified memory at 273 GB/s. It runs NVIDIA's AI software stack, so CUDA code and GPU containers built for Arm64 run as they do on NVIDIA's data-centre systems.

You log in over SSH as renter, a standard account with no sudo:

bash
ssh -p 2201 renter@ssh.gputorent.com

Your port is on the reservation page and in the email sent when the reservation goes live (2201 is an example). The machine is yours alone for the reservation.

What's installed

  • DGX OS, NVIDIA's Ubuntu-based Linux, with the usual command-line tools and Python 3.
  • The NVIDIA driver and CUDA, with nvcc and nvidia-smi on the PATH, and libraries such as cuBLAS.
  • A docker command that runs rootless Podman as you, with access to the GPU (see below).

Check the GPU:

bash
nvidia-smi

Python environments go in your home folder: uv, pip in a python3 -m venv, or Miniforge. There's no sudo apt install; if you need a system package, ask us and we can add it between rentals.

It's an Arm machine

The GB10's CPU is Arm64 (aarch64), not x86. Python wheels, container images and binaries must be built for linux/arm64. Most major projects publish them — NVIDIA's NGC containers, PyTorch, vLLM — but check before relying on an older or niche package.

Containers: docker is rootless Podman

The docker command on the machine runs rootless Podman as you. Containers never have root on the host, and there's no docker group. In practice:

  • Most docker run commands work unchanged, and so do docker ps, docker logs, docker stop and docker rm.
  • Short image names like vllm/vllm-openai resolve to Docker Hub.
  • Publish ports above 1024 with -p, for example -p 8000:8000.
  • Images and containers are stored in your home folder, so they're erased at the end of the reservation like everything else.

Using the GPU in a container

Give a container the GPU with --device nvidia.com/gpu=all. The Docker flags --gpus all and -e NVIDIA_VISIBLE_DEVICES=all don't work here.

bash
docker run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

NVIDIA's NGC images, such as nvcr.io/nvidia/pytorch:25.09-py3, work the same way.

Serving a model with vLLM

Our Download a model template fetches Qwen/Qwen3-8B into ~/.cache/huggingface before you log in. To serve it from that cache with vLLM's image, as an OpenAI-compatible API on port 8000:

bash
docker run -d --name model-serve --shm-size=16g -p 8000:8000 \
  --device nvidia.com/gpu=all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  docker.io/vllm/vllm-openai:latest \
  Qwen/Qwen3-8B --gpu-memory-utilization 0.85 --max-num-seqs 4

The image starts vllm serve by itself, so the model name follows the image name directly. Follow the start-up with docker logs -f model-serve, and stop it with docker stop model-serve && docker rm model-serve.

Reaching a server on the machine

Only the SSH port reaches the machine from the internet, so port 8000 isn't open to the world. Reach it through SSH port forwarding. On your own computer, run:

bash
ssh -p 2201 -N -L 8000:localhost:8000 renter@ssh.gputorent.com

Leave it running, and the API is at http://localhost:8000 on your computer:

bash
curl http://localhost:8000/v1/models

The same works for Jupyter or any other server; see VS Code and Jupyter.

Memory

The GPU and the operating system share the 128 GB, and all of your processes together are limited to about 90% of it (roughly 115 GB), so the machine stays responsive. Containers count towards the limit too, because they run as you.

Leave headroom inside that limit too: with vLLM, keep --gpu-memory-utilization around 0.85 rather than close to 1.0, and keep --max-num-seqs small (about 4). The same applies to other frameworks that pre-allocate GPU memory.

Unified memory also means a CUDA program can allocate with cudaMallocManaged and share data between CPU and GPU without explicit copies.

Performance notes

  • The GB10 is rated at up to 1 PFLOP of FP4 compute, so prompt processing and batch work are fast.
  • Token generation is bound by the 273 GB/s memory bandwidth, like any large-model inference. Quantised models (FP8, FP4 or 4-bit) generate faster because each token reads fewer bytes.

DGX Spark and the GB10, explained goes into what fits and how fast it runs. What's allowed on the machine lists the rules that apply to every machine.