How to deploy an LLM with vLLM on a GPU server

To deploy an LLM pinch vLLM, prepare your GPU server and instal vLLM there, past load your preferred vLLM-compatible exemplary and service it done an OpenAI-compatible API.

vLLM itself is simply a instrumentality for moving and serving LLMs connected GPUs. It’s designed for businesslike exemplary serving, particularly erstwhile respective requests request to stock the aforesaid GPU.

Here’s really to deploy an LLM pinch vLLM connected your GPU server:

1. Choose a GPU pinch capable VRAM for your exemplary and expected workload.
2. Connect to the GPU server complete SSH and verify the NVIDIA hardware.
3. Set up an isolated Python situation and instal vLLM.
4. Load the exemplary and commencement its OpenAI-compatible API.
5. Keep the vLLM server moving pinch a systemd service.
6. Protect distant API entree pinch a backstage larboard and SSH tunnel.
7. Send trial requests and corroborate the API cardinal and web restrictions work.
8. Compare conclusion throughput pinch azygous and concurrent requests.

This tutorial uses Qwen2.5-Coder-7B-Instruct arsenic the vLLM exemplary and Hostinger GPU arsenic the provider. The wide process applies to different vLLM-compatible models and GPU providers, pinch flimsy differences successful GPU sizing, server setup, and control-panel menus.

1. Choose a Hostinger GPU for the model

Choose the L40S GPU pinch 48 GB of VRAM for the Qwen2.5-Coder-7B-Instruct exemplary erstwhile you rent a GPU from Hostinger.

Your GPU for vLLM needs capable VRAM for the exemplary weights, which dangle connected the parameter count and precision, positive the KV cache, runtime overhead, discourse length, and concurrent requests.

Start pinch the exemplary weights because they group the baseline VRAM requirement. Use this elemental estimate for a exemplary pinch BF16 weights, specified arsenic Qwen2.5-Coder-7B-Instruct:

VRAM for exemplary weights ≈ parameter count × 2 bytes

The exemplary has 7.61 cardinal parameters, truthful the calculation is:

7.61 cardinal × 2 bytes ≈ 15.2 GB

Don’t dainty 15.2 GB arsenic the model’s full VRAM requirement, though.

You besides request GPU representation for the CUDA runtime, activations that temporarily clasp information while the exemplary processes tokens, which stores antecedently processed token information for reuse during generation.

Hostinger offers the RTX 4090 pinch 24 GB of VRAM, starting astatine $0.38/hour. That capacity exceeds the model’s estimated 15.2 GB weight footprint.

However, we urge the L40S, starting astatine $0.92/hour, because its 48 GB of VRAM provides substantially much room for the KV cache, runtime overhead, longer discourse windows, and concurrent requests.

Use this comparison array to choose the correct Hostinger GPU for your workload:

GPU	VRAM	When to take it
RTX 4090	24 GB	Development, experimentation, and small- to medium-model inference
L40S	48 GB	AI conclusion and generative AI workloads (recommended for this deployment)
A100 80GB PCIe	80 GB	LLM conclusion pinch larger representation requirements, exemplary training, and investigation workloads
RTX PRO 6000 (Server)	96 GB	Large-model conclusion and fine-tuning that require much VRAM than the L40S provides
B200	192 GB	Large-model fine-tuning, conclusion astatine scale, and different highly memory-intensive AI workloads
B200 (Dedicated)	192 GB	The aforesaid workloads arsenic the B200 erstwhile you specifically request dedicated, non-shared GPU resources

After choosing your GPU, set up your instance successful hPanel to take your operating system, apical up your relationship credits, and deploy it.

2. Connect to the GPU server and verify the hardware

Connect to your Hostinger GPU server complete SSH and verify the L40S pinch nvidia-smi earlier installing vLLM.

In hPanel, spell to Dev Tools → GPU → Manage and transcript the SSH bid from the Overview page:

Then unfastened a terminal connected your computer, paste the command, and participate the guidelines password to connect. Note the username and larboard successful the SSH bid – you’ll request some later to unfastened an SSH tunnel.

You tin besides set up passwordless SSH for much unafraid key-based entree without entering the SSH password each clip you log in.

Once connected, update the package list:

sudo apt update

Next, verify the GPU:

nvidia-smi

The output should database the NVIDIA L40S, its driver version, the CUDA type supported by the driver, and astir 48 GB of GPU VRAM.

Record the existent VRAM usage truthful you tin comparison it pinch the usage aft vLLM loads the Qwen2.5-Coder-7B-Instruct model.

3. Install vLLM successful a Python environment

To instal vLLM successful a Python environment, first group up the required tools, past usage uv to create the situation and instal the package.

uv downloads and manages the required Python version, truthful you don’t request to instal Python separately.

First, instal curl and ninja-build, past usage the erstwhile to instal uv:

sudo apt instal -y curl ninja-build curl -LsSf https://astral.sh/uv/install.sh | sh source "$HOME/.local/bin/env"

Next, create a directory for vLLM and move into it:

sudo mkdir -p /opt/vllm sudo chown "$USER":"$USER" /opt/vllm cd /opt/vllm

Create and activate the Python virtual environment:

uv venv --python 3.12 --seed --managed-python source .venv/bin/activate

Now instal the pinned vLLM version. Pinning the type keeps your setup reproducible and matches the commands successful this tutorial.

uv pip instal "vllm==0.28.0" --torch-backend=auto

The –torch-backend=auto action selects a PyTorch build that matches your NVIDIA driver, truthful you don’t request to take a CUDA-specific package yourself.

Finally, verify the installation and corroborate that PyTorch detects the L40S:

python --version vllm --version python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('GPU:', torch.cuda.get_device_name(0) if torch.cuda.is_available() other 'none')"

The output should show Python 3.12, vLLM 0.28.0, CUDA available: True, and NVIDIA L40S.

4. Serve the exemplary pinch the vLLM OpenAI-compatible API

Serve Qwen2.5-Coder-7B-Instruct pinch vllm serve to commencement an OpenAI-compatible API connected the GPU server.

First, make an API key:

export VLLM_API_KEY="$(python -c 'import secrets; print(secrets.token_hex(32))')" echo "$VLLM_API_KEY"

The first bid generates the cardinal and stores it arsenic VLLM_API_KEY. The 2nd prints the generated key, which looks akin to this:

6f1c82e0b9d64cda829f5d707f5571590e98c42dd75a5204349fd9029e498865

Save the printed cardinal location unafraid because you’ll usage the aforesaid worth to link to the API later.

Start the exemplary and walk the API cardinal to vLLM:

vllm service Qwen/Qwen2.5-Coder-7B-Instruct \ --host 127.0.0.1 \ --port 8000 \ --api-key "$VLLM_API_KEY"

The –host 127.0.0.1 action keeps vLLM accessible only from the GPU server, while –port 8000 sets the section API port. The –api-key action requires clients to supply the cardinal erstwhile sending requests to the OpenAI-compatible API endpoints.

The first motorboat takes a fewer minutes to an hr because vLLM needs to download and cache astir 15 GB of exemplary files. The nonstop clip depends connected your net speed.

Wait until the terminal shows that the server has started successfully, past time off vLLM moving successful the existent terminal.

Next, unfastened a caller terminal window, link to the server the aforesaid measurement arsenic before, and cheque the GPU again:

nvidia-smi

Compare the existent VRAM usage pinch the worth you noted earlier starting vLLM. The summation is expected because the loaded model, KV cache, and vLLM runtime each usage GPU memory.

You tin now safely adjacent this terminal.

5. Keep the vLLM server moving pinch systemd

Create a systemd work for vLLM truthful the server keeps moving aft you disconnect from your SSH convention and restarts automatically aft a failure.

Go to the terminal wherever you started vLLM and property Ctrl+C to extremity it.

Save the API cardinal successful a record truthful systemd tin usage it aft you adjacent the terminal:

printf 'VLLM_API_KEY=%s\n' "$VLLM_API_KEY" | sudo tee /etc/vllm.env > /dev/null sudo chmod 600 /etc/vllm.env

You don’t request to switch $VLLM_API_KEY pinch the existent key. The bid takes the API cardinal already saved successful your terminal convention and writes its worth to /etc/vllm.env.

Next, create the systemd service:

sudo tee /etc/systemd/system/vllm.service > /dev/null <<'EOF' [Unit] Description=vLLM conclusion server After=network-online.target Wants=network-online.target [Service] Type=simple EnvironmentFile=/etc/vllm.env WorkingDirectory=/opt/vllm ExecStart=/opt/vllm/.venv/bin/vllm service Qwen/Qwen2.5-Coder-7B-Instruct --host 127.0.0.1 --port 8000 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF

Restart=on-failure starts vLLM again 5 seconds aft the process exits pinch an error.

Reload systemd, commencement the service, and alteration it to commencement automatically aft a reboot:

sudo systemctl daemon-reload sudo systemctl alteration --now vllm sudo systemctl position vllm --no-pager

The position should show active (running).

vLLM still needs clip to load the exemplary and initialize its GPU components. Wait for a fewer minutes earlier checking anything.

Next, corroborate that vLLM is serving requests aft it finishes loading the cached model:

source <(sudo feline /etc/vllm.env) curl -sS \ -H "Authorization: Bearer $VLLM_API_KEY" \ http://127.0.0.1:8000/v1/models

The consequence should see Qwen/Qwen2.5-Coder-7B-Instruct. You tin now safely disconnect from SSH without stopping vLLM.

Check the work logs pinch this journalctl command if systemctl status shows failed aliases vLLM doesn’t respond aft loading the model:

journalctl -u vllm -n 100 --no-pager

6. Secure entree to the vLLM API

Secure entree to the vLLM API by keeping larboard 8000 backstage and connecting done an SSH tunnel.

Open a caller terminal connected your machine and run:

ssh -N -L 8000:127.0.0.1:8000 -p SSH_PORT ubuntu@GPU_IP_ADDRESS

Replace SSH_PORT and GPU_IP_ADDRESS pinch the values shown for your GPU lawsuit successful hPanel.

After you authenticate, the terminal stays unfastened without showing a ammunition prompt. This is expected because -N creates the SSH relationship only for larboard forwarding.

Leave this terminal unfastened while you usage the vLLM API. The passageway forwards http://127.0.0.1:8000 connected your machine to 127.0.0.1:8000 connected the GPU server.

Note that Hostinger doesn’t make non-SSH services publically accessible by default, though you tin expose services connected your GPU instance done hPanel.

However, debar exposing immoderate work utilizing larboard 8000 because doing truthful makes the vLLM API reachable from the internet.

Also, secure exposed services pinch app authentication, firewall rules, and HTTPS certificates erstwhile you intentionally make a work public.

7. Test the vLLM API connection

To trial the vLLM API connection, nonstop a chat petition to your deployed exemplary done the SSH tunnel, past verify that the API cardinal is required and larboard 8000 isn’t publically accessible.

Open a caller terminal connected your computer. Then, load the API cardinal you saved earlier:

printf "Paste the vLLM API key: " read -s VLLM_API_KEY echo export VLLM_API_KEY

Paste the cardinal astatine the prompt. The terminal won’t show it.

Next, nonstop a petition to the /v1/chat/completions endpoint:

curl -sS -w '\nHTTP %{http_code}\n' \ http://127.0.0.1:8000/v1/chat/completions \ -H "Authorization: Bearer $VLLM_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-Coder-7B-Instruct", "messages": [ { "role": "user", "content": "Write a Python usability named add(a, b) that returns a + b. Return codification only." } ], "temperature": 0, "max_tokens": 64 }'

A successful petition returns HTTP 200. The JSON consequence should show Qwen/Qwen2.5-Coder-7B-Instruct successful the model field, generated codification successful the response, and token counts nether usage.

After that, nonstop a petition without the Authorization header:

curl -sS -o /dev/null -w 'HTTP %{http_code}\n' \    http://127.0.0.1:8000/v1/models

vLLM should return HTTP 401, confirming that it rejects requests without the API key.

Finally, effort connecting straight to larboard 8000 connected the GPU lawsuit alternatively of utilizing the SSH tunnel:

curl --connect-timeout 5 http://GPU_IP_ADDRESS:8000/v1/models

The relationship should clip retired aliases neglect because vLLM listens connected 127.0.0.1 and you haven’t exposed larboard 8000.

Also cheque your exposed services successful hPanel and make judge nary usage soul larboard 8000.

8. Benchmark conclusion throughput

Benchmark your vLLM throughput by comparing 1 petition astatine a clip pinch up to 8 simultaneous requests.

Close the terminal you utilized for the API tests and extremity the SSH passageway pinch Ctrl+C.

Then commencement a caller terminal, link to your GPU server complete SSH, activate the vLLM environment, and load the API key::

cd /opt/vllm source .venv/bin/activate export VLLM_API_KEY="$(sudo sed -n 's/^VLLM_API_KEY=//p' /etc/vllm.env)"

Next, tally the benchmark pinch 1 petition astatine a time:

vllm chair service \ --backend openai-chat \ --base-url http://127.0.0.1:8000 \ --endpoint /v1/chat/completions \ --model Qwen/Qwen2.5-Coder-7B-Instruct \ --dataset-name random \ --num-prompts 32 \ --input-len 512 \ --output-len 128 \ --max-concurrency 1 \ --header "Authorization=Bearer ${VLLM_API_KEY}" \ --ignore-eos

Run the aforesaid bid again aft the first benchmark finishes, but alteration –max-concurrency 1 to –max-concurrency 8.

Keep the different settings unchanged. Both runs usage 32 requests, 512 input tokens, and 128 output tokens, truthful concurrency is the only variable.

The –ignore-eos action makes each petition make the afloat 128 output tokens, which keeps the 2 runs comparable.

While moving the benchmarks, cheque the Metrics conception for your GPU lawsuit successful hPanel. Use the 1h position to spot the GPU compute throughput and VRAM usage during the tests.

Record the Request throughput (req/s) and Output token throughput (tok/s) values shown astatine the extremity of each benchmark. A completed comparison could look for illustration this:

Max concurrency	Request throughput (req/s)	Output throughput (tok/s)
1	0.38	48.86
8	2.89	370.41

Request throughput shows really galore requests vLLM completes per second, while output throughput shows really galore output tokens it generates per second.

Increasing maximum concurrency from 1 to 8 raised petition throughput from 0.38 to 2.89 req/s and output-token throughput from 48.86 to 370.41 tok/s.

How vLLM handles concurrent conclusion efficiently

vLLM handles concurrent conclusion efficiently pinch continuous batching, which keeps adding waiting requests arsenic processing capacity becomes available, and PagedAttention, which reduces wasted KV-cache memory.

Unlike a fixed batch that waits for the full group to finish, continuous batching lets vLLM commencement waiting requests while different requests are still generating tokens.

PagedAttention stores each request’s KV cache, the attraction information vLLM saves for antecedently processed tokens, successful mini blocks wherever VRAM is available. This reduces wasted gaps successful GPU representation and leaves much room for concurrent requests.

When to usage vLLM alternatively of Ollama

Use vLLM alternatively of Ollama erstwhile you expect respective users aliases apps to stock the aforesaid GPU and serving much requests efficiently matters much than simplifying exemplary setup.

The main quality is what each instrumentality prioritizes. vLLM focuses connected conclusion serving, pinch features specified arsenic continuous batching and PagedAttention, while Ollama simplifies downloading, running, and switching betwixt models.

In practice, vLLM lets the GPU process concurrent requests much efficiently, but you configure much of the serving setup yourself. Ollama handles much of that setup for you, making it easier to get a exemplary moving quickly.

Choose vLLM for squad coding assistants, exertion backends, shared soul tools, and different workloads that person requests from respective clients.

Set up Ollama for section development, trying different models, aliases individual devices wherever easiness of usage matters much than maximizing throughput.

For a elemental AI-powered app sending only a fewer API requests astatine a time, either action useful well.

How to link vLLM to Continue successful VS Code

To link vLLM to Continue successful Visual Studio (VS) Code, configure Continue to nonstop requests to http://127.0.0.1:8000/v1 done your SSH tunnel.

Continue is an open-source AI coding adjunct disposable arsenic a VS Code extension. It supports OpenAI-compatible APIs, truthful you tin usage the Qwen exemplary and API cardinal you already configured.

Set up Continue pinch the vLLM endpoint

Set up Continue by installing the VS Code extension, adding your vLLM API cardinal arsenic a section secret, and utilizing http://127.0.0.1:8000/v1 arsenic the API guidelines URL.

1. Open the hold marketplace successful VS Code, hunt for Continue, and prime Install.

2. Start the SSH passageway and time off its terminal open:

ssh -N -L 8000:127.0.0.1:8000 -p SSH_PORT ubuntu@GPU_IP_ADDRESS

Replace SSH_PORT and GPU_IP_ADDRESS accordingly.

3. Open the Continue sidebar and spell to Settings → Configs → Main Config. Click the cogwheel icon to unfastened config.yaml, which Continue creates automatically.
4. Add the Qwen2.5-Coder-7B-Instruct exemplary to config.yaml:

name: Self-hosted vLLM version: 1.0.0 schema: v1 models: - name: Qwen2.5 Coder 7B provider: openai model: Qwen/Qwen2.5-Coder-7B-Instruct apiBase: http://127.0.0.1:8000/v1 apiKey: ${{ secrets.VLLM_API_KEY }} roles: - chat - edit - apply

The provider: openai mounting tells Continue to usage the OpenAI-compatible API format. apiBase sends those requests to vLLM done the SSH tunnel.

Save config.yaml erstwhile you’re done.

5. In the aforesaid .continue files arsenic config.yaml, create a record named .env and adhd the vLLM API cardinal you saved earlier:

VLLM_API_KEY=your-vllm-api-key

Replace your-vllm-api-key pinch the existent key, past prevention .env.

6. Restart VS Code truthful Continue loads the API cardinal from .env. Open Continue and prime Qwen2.5 Coder 7B arsenic the model.

Test the deployed exemplary pinch a coding request

To trial the vLLM Continue integration, nonstop a coding petition from Continue while the SSH passageway remains open.

For example, ask:

Write a Python usability that checks whether a drawstring is simply a palindrome and adhd 3 pytest tests.

The nonstop consequence will vary, but a successful consequence should incorporate codification akin to this:

This confirms that Continue reached your vLLM API done the SSH passageway and received output from the configured model.

Repeat the curl API trial you ran antecedently if Continue doesn’t respond. Double-check the model, apiBase, and API cardinal successful your Continue configuration if curl useful but Continue still can’t connect.

Next steps for your LLM deployment pinch vLLM

Extend your LLM deployment pinch quantization to trim VRAM usage, LoRA adapters to service fine-tuned variants, aliases tensor parallelism to usage aggregate GPUs.

• Quantization. It stores exemplary weights astatine little precision to trim VRAM usage. Use it adjacent erstwhile you want to fresh a larger exemplary connected the aforesaid GPU aliases time off much representation disposable for inference. Serve a supported quantized model, specified arsenic an AWQ aliases GPTQ checkpoint.
• LoRA adapters. They adhd mini sets of fine-tuned weights to a guidelines exemplary without loading a abstracted transcript of the afloat model. Use them adjacent erstwhile you want to service different specialized versions of the aforesaid guidelines model. In vLLM, alteration LoRA support pinch –enable-lora and load adapters pinch –lora-modules adapter-name=adapter-path.
• Tensor parallelism. It splits a exemplary crossed aggregate GPUs. Use it adjacent erstwhile the exemplary nary longer fits connected 1 GPU aliases you want to dispersed its representation requirements crossed respective GPUs. Set –tensor-parallel-size to the number of GPUs you want vLLM to use, for example, –tensor-parallel-size 2 for 2 GPUs.

All of the tutorial contented connected this website is taxable to Hostinger's rigorous editorial standards and values.