Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

Running a frontier-scale example locally method keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected complete ConnectX-8 RDMA assist one example through a sole endpoint, alongside tensor parallelism splitting the example weights and computation throughout the two GPUs. 

The NVIDIA GLM-5.3 NVFP4 model, a quantized type of Z.ai’s mixture-of-experts (MoE) model, has 753 milliard parameters, alongside 40 milliard activated per token. Sparse activation reduces computation, but all example weights motionless need to remain accessible since all token can be routed to distinct experts. At concerning 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a sole GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, alongside CPU importance offload disabled.

Before starting, nexus the two Stations alongside the Connect Two DGX Stations playbook.

This article shows how to assist GLM-5.3 locally throughout two NVIDIA DGX Stations alongside vLLM, dispatch a request, and measure behavior under concurrent load. Tensor parallelism (TP=2) splits the model’s tensors throughout the two GB300 GPUs. Both Stations compute all request, and Station A returns the answer through one OpenAI-compatible API. By the end, you’ll have a operating endpoint and a baseline for how it behaves under load.

Prerequisites

• Two GB300 DGX Stations operating Linux, connected and validated alongside the Connect Two DGX Stations playbook, including its GPU-memory RDMA test on the two rails.
• SSH and sudo admission on the two Stations alongside the identical Linux username. Docker and the NVIDIA Container Toolkit arrive preinstalled alongside the DGX Station application stack. Allow outbound admission to the receptacle registry, Hugging Face, and bundle repositories.
• At smallest 464 GB of liberated disk area on all Station for its copy of the checkpoint, affirmative area for the vLLM receptacle depiction and impermanent files.
• Acceptance of the NVIDIA Open Model Agreement and the first GLM-5.3 licence terms, listed on the model card. SPEED-Bench is governed by the NVIDIA Evaluation Dataset License Agreement.
• A Hugging Face document and peruse admission token, alongside admission to the gated cais/hle dataset: sign in, open its page, and choose Agree and admission repository. Step 5 downloads SPEED-Bench prompts from this dataset.

The companion video follows the identical workflow.

Step 1. Prepare the two Stations

This tutorial uses a distinct computer, specified as your laptop, to oversee the two Stations. Start by beginning an SSH meeting to all Station from that computer. Alternatively, activity from a local terminal on all Station. Run all command in this stage on the two Stations unless it starts alongside “On A.”

The two Stations run the identical receptacle and checkpoint in distinct roles: Station A (rank 0) hosts the API, and Station B (rank 1) runs lone a worker, alongside no API server.

First, inspect Docker admission and collect the network values you’ll use later:

 docker version ibdev2netdev ip -br addr 

The ibdev2netdev output lists the two straightforward rails, cx8r0 and cx8r1. Pick one for serving traffic, for example cx8r1, and note two things for it on all Station:

• Its RDMA device, on the ibdev2netdev row that ends alongside the rail’s name. In mlx5_1 harbor 1 ==> cx8r1 (Up), the equipment is mlx5_1.
• Its IPv4 address, on the ip -br addr row that starts alongside the rail’s name. In cx8r1 UP 192.168.101.1/24, the location is 192.168.101.1.

The barrier name and equipment are the identical on the two Stations; lone the IPv4 location differs.

Next, create ~/glm53.env on all Station. This document stores the values that afterward commands use:

nano ~/glm53.env 

Paste the following obstacle and substitute all <...> placeholder, keeping the quotes, alongside the values you noted.

 RANK="<NODE_RANK>" # RANK="0" on Station A, RANK="1" on Station B HEAD_IP="<STATION_A_IPV4_ADDRESS>" # IPv4 location you noted on Station A; identical in the two files WORKER_IP="<STATION_B_IPV4_ADDRESS>" # IPv4 location you noted on Station B; identical in the two files RAIL_IFACE="<RAIL_NAME>" # Rail you picked, for example cx8r1; identical in the two files RAIL_HCA="<RDMA_DEVICE>" # RDMA equipment you noted, for example mlx5_1; identical in the two files IMG=vllm/vllm-openai:v0.30.0-aarch64 IMG+=@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56 MODEL_REPO=nvidia/GLM-5.3-NVFP4 MODEL_REV=e3b8f9f2fad29727dfa95e766c5cac18ea1414bf MODEL_HOST="$HOME/models/GLM-5.3-NVFP4" CONTAINER=glm53-full PORT=8000 if [ "$RANK" = 0 ]; then NODE_IP=$HEAD_IP; PEER_IP=$WORKER_IP else NODE_IP=$WORKER_IP; PEER_IP=$HEAD_IP fi GPU_UUID=$(nvidia-smi --query-gpu=uuid,name --format=csv,noheader | awk -F, '$2 ~ /GB300/ {gsub(/ /,"",$1); imprint $1; exit}') 

For example, alongside the default addresses on cx8r1, Station A’s document starts akin this. Station B’s document is identical apart from for RANK="1":

RANK="0" HEAD_IP="192.168.101.1" WORKER_IP="192.168.101.2" RAIL_IFACE="cx8r1" RAIL_HCA="mlx5_1" 

MODEL_HOST is built from your residence directory, and Step 2 copies the checkpoint complete SSH to the identical way on Station B.

On the two Stations, burden the values and inspect storage, connectivity, and baseline memory:

source ~/glm53.env mkdir -p "$MODEL_HOST" && df -h "$MODEL_HOST" ethtool "$RAIL_IFACE" ip nexus display dev "$RAIL_IFACE" ping -I "$RAIL_IFACE" -M do -s 8972 -c 3 "$PEER_IP" timedatectl display -p NTPSynchronized --value nvidia-smi 

These checks capture cabling and addressing mistakes before they exterior as NCCL timeouts. Expect a 400000Mb/s link, MTU 9000, prosperous jumbo-frame pings, and synchronized time. nvidia-smi should display no compute processes on the GB300. On a Station alongside a display GPU, Xorg can appear on the two GPUs alongside a few MiB.

Step 2 copies the checkpoint from A to B complete SSH, so A needs passwordless admission to B. On A, create a key without a passphrase (see SSH keys for details), copy its community key to B following checking B’s presenter fingerprint, and verify the login:

ssh-keygen -t ed25519 ssh-copy-id "$WORKER_IP" ssh -o BatchMode=yes "$WORKER_IP" hostname 

Step 2. Download and copy the model

Tensor parallelism requires the two ranks to run the identical application against the identical weights. Start by pulling the pinned depiction on the two Stations:

docker drag "$IMG" && docker depiction inspect "$IMG" --format '{{.Architecture}} {{.RepoDigests}}' 

The drag can obtain multiple minutes. Matching arm64 digests average the two ranks run identical software.

On A, download the pinned checkpoint. At concerning 464 GB, it takes approximately an hr at 1 Gb/s, or concerning six minutes at 10 Gb/s:

mkdir -p "$MODEL_HOST" docker run --rm --network presenter -v "$MODEL_HOST:/model" \ --entrypoint attack "$IMG" -lc \ "exec hf download '$MODEL_REPO' --revision '$MODEL_REV' --local-dir /model" 

When hf download prints Downloaded, copy the records to B. On A:

ssh "$WORKER_IP" "mkdir -p '$MODEL_HOST'" && rsync -aH --info=progress2 --exclude '.cache/' \ "$MODEL_HOST/" "$WORKER_IP:$MODEL_HOST/" 

Each disk keeps the complete checkpoint; vLLM partitions the weights throughout the GPUs at burden time, alongside no additional quantization.

On A, difference the two copies (the inspect says B’s document catalog complete SSH), afterward validate the checkpoint:

( set -euo pipefail L=$(find "$MODEL_HOST" -path '*/.cache' -prune -o \ -type f -printf '%P %s\n' | LC_ALL=C kind | sha256sum) R=$(ssh "$WORKER_IP" "set -o pipefail; discover '$MODEL_HOST' \ -path '*/.cache' -prune -o -type f -printf '%P %s\\n' | \ LC_ALL=C kind | sha256sum") test "$L" = "$R" test "$(find "$MODEL_HOST" -maxdepth 1 \ -name 'model-*-of-00050.safetensors' | wc -l)" = 50 python3 -c "import json; c=json.load(open('$MODEL_HOST/config.json')); \ assert c['architectures']==['GlmMoeDsaForCausalLM']; \ assert c['quantization_config']['quant_algo']=='MIXED_PRECISION'" echo MODEL_READY ) 

MODEL_READY method the two Stations clasp the identical document names and sizes in the expected layout. MIXED_PRECISION is expected: NVFP4 quantization targets MoE-block linear weights and activations, not all tensor.

Step 3. Launch the distributed service

This stage turns two machines into one service: the identical initiate manuscript runs on the two Stations, says ~/glm53.env, and starts one vLLM position on each. Two settings matter most if you modify it: --max-model-len 98304, the environment ceiling in tokens, which covers input affirmative output, and --gpu-memory-utilization 0.93, which gives vLLM 93% of GPU memory. If you alter another flags, alter one at a period and test again.

For the latest recommended configurations, see NVIDIA’s Local AI formula repository.

On the two Stations, create a document named ~/launch-glm53.sh:

nano ~/launch-glm53.sh 

The manuscript enables multi-token prediction (MTP), a form of speculative decoding: the model’s MTP tier drafts four tokens per step, and vLLM verifies them alongside the complete example before accepting them, so all stage can create multiple tokens. The acquire depends on how frequently drafts are accepted, which varies alongside the prompt.

FP8 KV-cache retention halves cache recollection compared alongside 16-bit storage, leaving area for additional tokens. Prefix caching is enabled to evade reprocessing immediate prefixes the server has already seen. Paste the manuscript below; it checks your configuration, finds the RoCE GID indicator for the selected rail, and starts this Station’s vLLM position in a container:

 set -euo pipefail source "$HOME/glm53.env" case "$RANK" in 0|1) ;; *) echo "Invalid RANK" >&2; exit 1;; esac test -n "$GPU_UUID" test -f "$MODEL_HOST/config.json" GID_INDEX=$(/usr/sbin/show_gids 2>/dev/null | awk -v d="$RAIL_HCA" -v ip="$NODE_IP" \ '$1==d && $5==ip && $6=="v2" {g=$3} END {if(g!="") imprint g}') test -n "$GID_INDEX" HEADLESS= if [ "$RANK" = 1 ]; afterward HEADLESS=--headless; fi SPEC_CONFIG='{"method":"mtp","num_speculative_tokens":4}' KERNEL_CONFIG='{"enable_flashinfer_autotune":false,"moe_backend":"flashinfer_cutlass"}' COMPILE_CONFIG='{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[5,10,20,40,80],"pass_config":{"fuse_allreduce_rms":false}}' docker run -d --name "$CONTAINER" \ --gpus device="$GPU_UUID" --network presenter --ipc presenter --shm-size 32g \ --device /dev/infiniband --cap-add IPC_LOCK \ --ulimit memlock=-1 --ulimit stack=67108864 \ -v "$MODEL_HOST:/model:ro" \ -e RANK="$RANK" -e HEADLESS="$HEADLESS" \ -e MASTER_ADDR="$HEAD_IP" -e MODEL_REPO="$MODEL_REPO" -e PORT="$PORT" \ -e SPEC_CONFIG="$SPEC_CONFIG" -e KERNEL_CONFIG="$KERNEL_CONFIG" \ -e COMPILE_CONFIG="$COMPILE_CONFIG" \ -e NCCL_CUMEM_ENABLE=1 -e NCCL_IB_HCA="$RAIL_HCA" \ -e NCCL_IB_GID_INDEX="$GID_INDEX" \ -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 \ -e VLLM_ALLREDUCE_USE_FLASHINFER=0 -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 \ -e NCCL_SOCKET_IFNAME="$RAIL_IFACE" -e GLOO_SOCKET_IFNAME="$RAIL_IFACE" \ -e NCCL_NET_GDR_LEVEL=5 -e NCCL_DEBUG=INFO -e NCCL_DEBUG_SUBSYS=INIT,NET \ -e NCCL_DEBUG_FILE='/tmp/nccl_%h_%p.log' -e VLLM_HOST_IP="$NODE_IP" \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e FLASHINFER_DISABLE_VERSION_CHECK=1 \ --entrypoint attack "$IMG" -lc ' set -euo pipefail # Use NCCL 2.30.7 from the image: older versions can deadlock # cross-node CUDA chart replays complete RoCE (vLLM matter #52504). export VLLM_NCCL_SO_PATH=$(python3 -c "import nvidia.nccl,os; print(os.path.join(list(nvidia.nccl.__path__)[0], \"lib\", \"libnccl.so.2\"))") test -f "$VLLM_NCCL_SO_PATH" EXTRA=() if [ -n "$HEADLESS" ]; afterward EXTRA+=("$HEADLESS"); fi exec vllm assist /model --served-model-name "$MODEL_REPO" \ --host 0.0.0.0 --port "$PORT" \ --nnodes 2 --node-rank "$RANK" --master-addr "$MASTER_ADDR" \ --master-port 29501 --tensor-parallel-size 2 \ --speculative-config "$SPEC_CONFIG" \ --max-model-len 98304 --max-num-batched-tokens 8192 \ --block-size 128 --max-num-seqs 16 \ --trust-remote-code --disable-custom-all-reduce \ --kv-cache-dtype fp8 --enable-prefix-caching \ --gpu-memory-utilization 0.93 \ --enable-prompt-tokens-details \ --kernel-config "$KERNEL_CONFIG" --compilation-config "$COMPILE_CONFIG" \ --distributed-timeout-seconds 1800 \ --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \ "${EXTRA[@]}" ' 

Run the manuscript on B first, afterward on A:

bash ~/launch-glm53.sh 

Each command prints a receptacle ID and returns immediately; the assistance is not prepared yet. B’s employee waits for A to join, afterward all Station loads its portion of the weights from local storage, prepares kernels, and captures CUDA graphs. This takes multiple minutes.

On A, prosecute the log to see whenever the assistance is prepared (--since hides output from before runs):

docker logs --follow --since "$(docker inspect \ --format '{{.State.StartedAt}}' "$CONTAINER")" "$CONTAINER" 

When the log shows Application startup complete, media Ctrl+C to halt following it; the assistance keeps running.

At this item one example spans two machines. Neither GPU holds a complete copy, and neither can answer a petition alone: all token needs the two ranks to toggle partial results complete the RoCE rail.

The remaining GPU recollection holds the KV cache, which stores the environment of in-flight requests. The startup log reports its capacity (GPU KV cache size). All requests in escape portion it, so longer prompts or additional users need additional of it. If you run out, lesser --max-model-len or dispatch small requests at once.

Step 4. Verify serving and dispatch a request

On A, inspect the API (127.0.0.1 is A itself):

docker exec "$CONTAINER" curl -fsS --max-time 10 \ "http://127.0.0.1:$PORT/health" >/dev/null && echo HEALTH_OK 

HEALTH_OK proves that Station A answers. Next, on the two Stations, inspect that the ranks communicate complete GPU RDMA fairly than falling rear to TCP sockets. The inspect says the NCCL logs that the launcher writes through NCCL_DEBUG=INFO, and the identical obstacle shows the loaded memory:

docker exec "$CONTAINER" attack -lc ' set -euo pipefail files=(/tmp/nccl_*.log) test -e "${files[0]}" grep -hE "NET/IB.*GDRDMA|GPU Direct RDMA Enabled" "${files[@]}" >/dev/null grep -hF "$NCCL_IB_HCA" "${files[@]}" >/dev/null if grep -hE "NET/Socket|NCCL WARN|NCCL ERROR" "${files[@]}"; then exit 1 fi echo RDMA_CHECK_OK ' nvidia-smi 

RDMA_CHECK_OK method the inspect passed alongside no warnings. In nvidia-smi, A shows the VLLM::Worker_TP0 procedure and B shows VLLM::Worker_TP1, one position per GPU. The startup log reports concerning 218 GiB of weights per GPU, approximately fractional the checkpoint, which confirms that the example is divided throughout the two GPUs.

On A, dispatch a conversation petition through the OpenAI-compatible API:

docker exec "$CONTAINER" curl -fsS --max-time 120 \ "http://127.0.0.1:$PORT/v1/chat/completions" \ -H 'Content-Type: application/json' \ -d '{"model":"nvidia/GLM-5.3-NVFP4", "messages":[{"role":"user","content":"Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one declaration and compute the percent alter from Q1 to Q2."}], "temperature":0,"max_tokens":4096, "chat_template_kwargs":{"reasoning_effort":"low"}}' | python3 -c 'import json,sys; c=json.load(sys.stdin)["choices"][0]; print(c["message"]["content"]); print("finish_reason:",c["finish_reason"])' 

The reply should appearance akin this:

**Trend:** Units sold rosy sharply in Q2 before partially declining in Q3, remaining complete the Q1 level. **Q1 → Q2 percent change:** (180 - 120) / 120 × 100 = **50% increase** finish_reason: stop 

finish_reason: halt method the example completed its answer alternatively of hitting the token limit. The petition sets reasoning_effort to low in chat_template_kwargs. GLM-5.3 additionally accepts elevated and max, which let it logic longer before answering, at the disbursal of latency. Because the launcher enables tool calling (--enable-auto-tool-choice), delegate frameworks that use the OpenAI tools API can call the identical endpoint.

Step 5. Concurrency and publish resources

AIPerf, the NVIDIA benchmarking tool for conclusion endpoints, measures throughput and latency. Installing it inner the serving receptacle can alter vLLM’s dependencies, so on A, instal it in its own virtual environment:

sudo apt update && sudo apt instal -y python3.12-venv python3.12-dev build-essential && python3.12 -m venv "$HOME/.venvs/aiperf-0.12.0" && "$HOME/.venvs/aiperf-0.12.0/bin/pip" instal 'aiperf==0.12.0' && "$HOME/.venvs/aiperf-0.12.0/bin/pip" check 

The instal should end alongside No damaged requirements found.

AIPerf can dispatch synthetic prompts or replay an existing dataset. This tutorial uses the 80 Coding conversations from SPEED-Bench, a set of genuine prompts built to measure speculative decoding, so MTP is measured on realistic text. SPEED-Bench’s preparedness manuscript fetches these prompts from multiple Hugging Face datasets, including the gated cais/hle: if you haven’t already, open its leaf and choose Agree and admission repository before you continue. Then, on A, log in to Hugging Face:

"$HOME/.venvs/aiperf-0.12.0/bin/hf" auth login 

Next, download and prepared the dataset. See the official AIPerf guide for details:

SPEED_BENCH_DIR="$HOME/datasets/speed-bench" mkdir -p "$SPEED_BENCH_DIR" curl -fLsS https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | \ "$HOME/.venvs/aiperf-0.12.0/bin/python" - --config qualitative --output_dir "$SPEED_BENCH_DIR" 

On A, inspect the dataset and service. Continue whenever the inspect prints BENCHMARK_READY:

 ( set -euo pipefail origin ~/glm53.env curl -fsS "http://$HEAD_IP:$PORT/health" >/dev/null "$HOME/.venvs/aiperf-0.12.0/bin/python" - <<'PY_CHECK' import json from pathlib import Path root = Path.home() path = base / "datasets/speed-bench/qualitative.jsonl" assert path.exists(), "Dataset missing: rerun the SPEED-Bench preparation" rows = [json.loads(line) for row in path.read_text().splitlines() if line.strip()] coding = [row for row in rows if row["category"] == "coding"] assert len(coding) == 80, "Expected 80 Coding conversations" assert sum(len(row["messages"]) for row in coding) == 89 assert all("FULL BENCHMARK DATA SHOULD BE FETCHED" not in message["content"] for row in coding for communication in row["messages"]) print("BENCHMARK_READY: dataset has 80 Coding conversations, 89 turns in total.") PY_CHECK ) 

The complete clear covers concurrency 1, 8, 16, and 32. At all level, AIPerf sends 3×C warmup requests, afterward measures 10×C requests, anywhere C is the concurrency. Some conversations are sent additional than formerly at higher concurrency.

Benchmark requests use the model’s default reasoning attempt (max) and an 8,192-token output cap; responses can complete before reaching it. Full sweeps can obtain multiple hours.

 A two-hour petition timeout leaves area for lengthy responses, --server-metrics collects server counters used to measure MTP acceptance, and --export-level raw keeps all reply for inspection. 

For a quick test, run the obstacle below on A alongside the two containers operating and no another API traffic. It tests C1 and C8 alongside prefix caching enabled, as in the citation measurements in Table 2, below.

 ( set -euo pipefail origin ~/glm53.env CODING_DATASET="$HOME/datasets/speed-bench/qualitative.jsonl" test -f "$CODING_DATASET" export CONCURRENCIES=${CONCURRENCIES:-"1 8"} # Quick run: C1 and C8. # Optional complete benchmark: prosecute the reproduction instructions below. # Step 3: --max-num-seqs 16 -> export CONCURRENCIES="1 8 16" # Step 3: --max-num-seqs 32 -> export CONCURRENCIES=32 RUN_DIR="$HOME/aiperf-results/blog-coding-$(date +%Y%m%d-%H%M%S)" for C in $CONCURRENCIES; do ART="$RUN_DIR/c$C" env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE \ "$HOME/.venvs/aiperf-0.12.0/bin/aiperf" overview \ -m "$MODEL_REPO" --url "http://$HEAD_IP:$PORT" \ --endpoint-type conversation --streaming --tokenizer "$MODEL_HOST" \ --custom-dataset-type speed_bench_coding --input-file "$CODING_DATASET" \ --request-count "$((10 * C))" --warmup-request-count "$((3 * C))" \ --osl 8192 \ --use-server-token-count --concurrency "$C" --random-seed 42 \ --request-timeout-seconds 7200 \ --server-metrics "http://$HEAD_IP:$PORT/metrics" \ --export-level raw --artifact-dir "$ART" printf "Results: %s\n" "$ART" done printf "Results directory: %s\n" "$RUN_DIR" ) 

AIPerf saves metrics, raw responses, and logs in the results directory. Warmup requests are excluded from the reported metrics.

Optional: Reproduce Table 2. In ~/launch-glm53.sh from Step 3 on the two Stations, keep --enable-prefix-caching and set --max-num-seqs to 16 or 32 for the corresponding collection shown in the comments above. Before all group, halt and eliminate the current containers on A, afterward B (docker halt --timeout 60 "$CONTAINER", followed by docker rm "$CONTAINER"). Relaunch the updated manuscript on B, afterward A alongside attack ~/launch-glm53.sh and delay for Application startup complete. On A, run the matching export CONCURRENCIES command from the comments, afterward rerun the AIPerf block.

To test without prefix caching, substitute --enable-prefix-caching alongside --no-enable-prefix-caching in the Step 3 manuscript on the two Stations, afterward recreate the containers as above. Performance may differ from Table 2.

Figure 2 compares general output throughput throughout measured requests alongside average per-user throughput. Token counts contain reasoning.

Table 2, below, shows how throughput and latency alter as additional requests portion the service.

Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, anywhere average petition latency is 453.5 s. Higher concurrency additionally reduces average per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32. 

After the requests finish, halt the vLLM receptacle on A, afterward on B:

docker halt --timeout 60 "$CONTAINER" docker inspect --format \ 'Status={{.State.Status}} Running={{.State.Running}}' "$CONTAINER" nvidia-smi 

The position should peruse Status=exited Running=false, alongside no vLLM processes in nvidia-smi; the checkpoint, containers, and results remain on disk. To resume, run docker commencement "$CONTAINER" on B, afterward A, and reiterate the Step 4 checks.

Troubleshoot average blockers

• Missing variables or SSH failure: origin ~/glm53.env following reconnecting; verify A-to-B key authentication and B’s fingerprint.
• SPEED-Bench preparedness fails alongside a gated-dataset error: sign in to Hugging Face alongside the identical account, open the cais/hle page, choose Agree and admission repository, afterward rerun the preparation.
• Startup or RDMA inspect failure: inspect docker logs "$CONTAINER" on the two nodes; verify the checkpoint and the GID, and rerun the playbook’s validation (script 07).
• CUDA out of memory: lesser --max-model-len or concurrency, afterward retest.

Put the endpoint to work

You now have a 753-billion-parameter example operating rearward one local API on two NVIDIA DGX Stations, alongside all importance in GPU memory. That opens up activity that’s difficult to do on a hosted endpoint: operating conclusion on delicate or proprietary data, prototyping agents locally, and iterating on prompts without per-token costs.

Connect your own use using the vLLM serving documentation, afterward rerun AIPerf alongside your prompts to discover the concurrency that fits your workload. A frontier-scale example is now prepared to build on, correct from your desk.