Self-Hosting Open-Source LLMs with vLLM on KServe

I’d been serving MLflow models on KServe for a while. That worked for tabular and classical ML workloads, but it doesn’t help when you want to run a small open-source language model in-cluster. Sending everything to Anthropic or OpenAI gets expensive fast, and some workloads shouldn’t leave the cluster at all.

So I spent time deploying and testing self-hosted open-source LLMs — specifically vLLM behind KServe on dedicated GPU nodes. The model serving part turned out to be straightforward. Getting GPUs to actually show up in Kubernetes was not.

The stack

Client (curl / app)
      |
      v
  Istio gateway
      |
      v
  KServe InferenceService (RawDeployment)
      |
      v
  vLLM OpenAI API server  (port 8080)
      |
      v
  g5.2xlarge GPU node  (A10G, 24GB VRAM)

vLLM exposes an OpenAI-compatible API — /v1/chat/completions, /v1/completions, /v1/models. KServe handles the Kubernetes plumbing. Istio routes external traffic to the right service. The interesting work is everything below the InferenceService.

Getting GPUs to work

GPU support on my cluster depends on two layers.

kops base config

In the kops cluster spec:

containerd:
  nvidiaGPU:
    enabled: true

This is the foundation (kops GPU docs). It bakes the NVIDIA driver into the AMI, configures containerd with the NVIDIA runtime, and exposes /dev/nvidia* on the host. Without this, nothing else matters.

NVIDIA GPU Operator

On top of kops, the GPU Operator runs in a dedicated namespace and wires up the rest:

Startup order matters: driver (AMI) → container toolkit → validator → device plugin → nvidia.com/gpu allocatable. If any step fails, inference pods sit in Pending with Insufficient nvidia.com/gpu and you’ll waste an hour staring at resource requests that look correct.

Dedicated GPU nodes

I run GPU workloads on a separate kops InstanceGroup:

spec:
  machineType: g5.2xlarge
  minSize: 0
  maxSize: 2
  nodeLabels:
    purpose: llm-gpu
  taints:
    - dedicated=llm-gpu:NoSchedule

minSize: 0 lets the group scale down when nothing needs a GPU. The taint keeps random pods off these nodes — but it also means both the GPU Operator daemonsets and your inference pods need matching tolerations. I missed this on the ClusterPolicy the first time; the device plugin never scheduled onto GPU nodes, and every inference pod stayed pending.

The fix was patching the operator’s daemonset tolerations to include the dedicated=llm-gpu taint alongside the standard nvidia.com/gpu one.

Deploying vLLM with KServe

I was already running KServe for MLflow models, so the path of least resistance was a classic InferenceService with a custom vLLM container. That works for a basic single-GPU deployment. KServe has since added LLMInferenceService (v0.16+) — a CRD built specifically for generative AI, with model URI handling, storage initialization, and routing baked in. If I were starting fresh on LLM serving today, I’d use that instead:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: qwen25-7b-vllm
  namespace: inference
spec:
  model:
    uri: hf://Qwen/Qwen2.5-7B-Instruct
    name: qwen25-7b-instruct
  replicas: 1
  template:
    containers:
      - name: main
        image: vllm/vllm-openai:v0.8.5
        resources:
          limits:
            nvidia.com/gpu: "1"
    nodeSelector:
      purpose: llm-gpu
    tolerations:
      - key: dedicated
        operator: Equal
        value: llm-gpu
        effect: NoSchedule
  router:
    gateway: {}
    route: {}

What I actually deployed was the older InferenceService in RawDeployment mode — mostly because the cluster already had KServe, Istio routing, and the MLflow serving patterns in place. Knative serverless scaling is a bad fit for GPU models anyway — cold-starting even a small model means re-downloading weights and reloading into VRAM. That takes minutes, not seconds.

Here’s the manifest I used for Qwen 2.5 7B Instruct — a small model that fits comfortably on a single A10G:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen25-7b-vllm
  namespace: inference
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 2
    runtimeClassName: nvidia
    nodeSelector:
      purpose: llm-gpu
    tolerations:
      - key: dedicated
        operator: Equal
        value: llm-gpu
        effect: NoSchedule
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    containers:
      - name: kserve-container
        image: vllm/vllm-openai:v0.8.5
        command:
          - python3
          - -m
          - vllm.entrypoints.openai.api_server
        args:
          - --host=0.0.0.0
          - --port=8080
          - --model=Qwen/Qwen2.5-7B-Instruct
          - --served-model-name=qwen25-7b-instruct
          - --tensor-parallel-size=1
          - --max-model-len=12288
          - --gpu-memory-utilization=0.9
          - --enable-prefix-caching
          - --max-num-seqs=8
          - --disable-log-requests
        startupProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 10
          failureThreshold: 60    # up to 10 min for download + load
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 10
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 20
          failureThreshold: 5
        resources:
          requests:
            cpu: "4"
            memory: 24Gi
            nvidia.com/gpu: "1"
          limits:
            nvidia.com/gpu: "1"

A few things that aren’t obvious from the YAML:

• startupProbe is critical. vLLM downloads model weights from HuggingFace on first boot. Without a generous startup probe, the liveness probe kills the container mid-download. I learned this the hard way — pod restart loops that looked like OOM errors were just the health check firing too early.
• runtimeClassName: nvidia selects the NVIDIA container runtime. Needs the kops + operator stack above to be healthy.
• One GPU per replica. Each pod that requests nvidia.com/gpu: "1" needs its own GPU. With maxSize: 2 on the node group, I’m capped at two concurrent replicas.

Model weight caching

A model in this size class is ~14GB of weights. That download should happen once, not on every pod restart or scale-up event. This isn’t optional — if you skip it, you’ll sit through a 10-minute HuggingFace pull every time Kubernetes reschedules your pod.

With LLMInferenceService, KServe’s storage initializer handles this — it pulls the model into a persistent volume before the vLLM container starts. That’s one reason the newer CRD is worth using.

On my InferenceService setup, the same principle applies: mount a PVC (or use node-local disk with enough headroom) so weights survive restarts. The GPU node’s root volume needs to be sized for it too — a default 20GB disk won’t fit the container image, model weights, and logs. I used 200GB on the instance group for that reason. If I add a second replica, I’d want a ReadWriteMany PVC so new pods don’t each download their own copy.

Istio routing sits in front — a VirtualService matches a path prefix and rewrites to the predictor service. After that, testing is a normal OpenAI API call:

curl -s https://<your-gateway>/inference/qwen25-7b/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen25-7b-instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }' | python3 -m json.tool

What the numbers look like

Baseline measurements for Qwen 2.5 7B (small-model territory) on a single g5.2xlarge (A10G, 24GB):

These shift with prompt length, output length, and concurrent load. But they’re useful for back-of-envelope capacity planning — at ~26 tok/s per GPU and two replicas max, I’m looking at roughly 52 tok/s ceiling before I need more nodes.

The hard limits are mostly VRAM. The A10G has 24GB; model weights take ~14 GiB, leaving ~7.4 GiB for the KV cache. Cranking --max-model-len up (Qwen supports 128k) doesn’t magically work — longer contexts mean fewer concurrent requests fit in cache.

vLLM exposes Prometheus metrics at /metrics. The ones I actually watch:

Tuning for the actual workload

The default config above is reasonable for interactive chat. I also tested a batch profile — document parsing, entity extraction, summarization — where requests have long inputs and short outputs. Same model, different vLLM args:

args:
  - --max-model-len=16384
  - --gpu-memory-utilization=0.85
  - --no-enable-prefix-caching       # independent requests, no shared prefixes
  - --max-num-seqs=4                 # fewer concurrent = more KV cache per request
  - --max-num-batched-tokens=16384   # bigger prefill batches
  - --enable-chunked-prefill         # overlap prefill with decode

The tradeoff is deliberate: lower concurrency, more context headroom, better per-request latency on long documents. The chat config optimizes for many short requests; the batch config optimizes for a few heavy ones.

For tool-calling in chat workloads, add:

  - --enable-auto-tool-choice
  - --tool-call-parser=hermes          # Qwen 2.5 uses hermes format

Sizing cheat sheet

Rough starting points — your mileage will vary:

What this costs

GPU nodes are the dominant line item. A g5.2xlarge (one A10G, 4 vCPU, 32GB RAM) runs about $1.20/hour on-demand in us-east-2. With minReplicas: 1, that’s roughly $870/month if the node stays up continuously.

The node group’s minSize: 0 helps — when no GPU pods are scheduled, the instance group can scale down entirely. In practice I kept minReplicas: 1 on the InferenceService, so the GPU node was always running during testing.

Compared to API pricing: a small self-hosted model has no per-token charge, but you’re paying for the GPU whether anyone is asking questions or not. For steady, predictable workloads — batch extraction, internal tooling, high-volume agent loops — the crossover against Claude or GPT-4 API costs can happen quickly. For bursty, low-volume chat, an API is probably cheaper and definitely less ops.

There’s no separate “model license” cost for Qwen 2.5 — it’s open source. The variable cost beyond the GPU is electricity and whatever you spend on HuggingFace bandwidth for the initial download.

What broke along the way

The GPU Operator + taint interaction was the sneaky one. Everything in the manifest looked right; the device plugin just wasn’t running on the nodes that had the GPUs.

Takeaways

Self-hosting smaller open-source models on Kubernetes is very doable once the GPU stack is healthy. The serving config (vLLM + KServe + Istio) was maybe a third of the effort. The rest was kops, the GPU Operator, taints, tolerations, and probes — the usual Kubernetes GPU tax.

If you’re already on KServe like I was, adding vLLM is mostly a new manifest and a route. For new LLM deployments, LLMInferenceService is the better starting point — it handles model caching and routing without rolling your own container spec. Either way: get the startup probe right, cache your model weights, and pick vLLM args based on whether you’re serving chat or batch — the defaults lean toward chat.

I’d love to try deploying something larger — 13B or 70B with tensor parallelism — but that means more GPUs, and getting capacity from AWS can be its own project. Even g5.2xlarge instances aren’t always available when you want them; during peak hours I’ve seen smaller CPU-based nodes come back InsufficientInstanceCapacity, and GPU instance types are worse. Requesting a quota increase helps, but it doesn’t guarantee you can actually launch the thing on a Tuesday afternoon.

References

• vLLM
• KServe
• LLMInferenceService overview
• kops GPU support
• NVIDIA GPU Operator