Sonda Red – Sonda Red

Lab Notes: XPU Manager 2.0 finally lets me delete my custom image

Wed, 24 Jun 2026 00:00:00 +0000

This note documents the cleanup path from a patched custom XPU Manager deployment to the upstream XPUMD 2.0 Helm chart for Intel GPU telemetry in my Kubernetes homelab.

Disclaimer: This is lab documentation, not production guidance. It describes what changed in my cluster and why I am happy to delete a workaround.

00 Finally, a boring XPU Manager install
01 What I used to do
02 The DRA monitor shape I was patching toward
03 What changed upstream
04 The new shape in my cluster
05 The metric name situation
06 What this removes from my life

00 Finally, a boring XPU Manager install

This is a small update, but it removes one of the uglier pieces of my GPU observability setup.

Intel XPU Manager has been one of those pieces in my homelab that I needed, but did not particularly enjoy maintaining.

I wanted GPU telemetry from the Intel Arc cards in my Kubernetes cluster. For a while, the implementation was very ugly.

Now XPU Manager 2.0 ships an upstream OCI Helm chart:

oci://ghcr.io/intel/xpumanager/charts/xpumd

The chart pulls the upstream image:

ghcr.io/intel/xpumanager/xpumd:v2.0.0

01 What I used to do

The old setup in my GitOps repo was not something I was proud of.

I had a Flux GitRepository pointed at intel/xpumanager, then a Flux Kustomization pointed into the upstream Kubernetes DaemonSet base.

On top of that I patched the DaemonSet hard enough to make it fit my cluster:

replace both container images
add a GPU-node toleration
force scheduling to sonda-core
add a DRA ResourceClaimTemplate
attach the DRA claim to the container resources
remove the original GPU resource limits
set ZES_ENABLE_SYSMAN=1
maintain a separate Service on port 29999
maintain a separate ServiceMonitor

The image was the worst part.

I had a custom image under sonda-red/custom-images because the public container image I had been using had gone stale for my needs.

There was also an upstream issue about regularly updating the DockerHub image, which was a good enough signal that I should stop treating that path as dependable for this setup.

That was not some clever platform abstraction. It was just a workaround that became infrastructure.

The Dockerfile did more than I want a lab image to do:

build on Ubuntu 24.04
install a Python virtual environment
pull the XPUM 1.3.7 deb from the GitHub release
install Intel GPU userspace packages
create an entrypoint that started xpumd
expose the old REST exporter path through Gunicorn

In rough terms, it was this shape:

ubuntu:24.04
 -> Python venv
 -> Flask / prometheus-client / grpcio / protobuf / gunicorn
 -> Intel graphics PPA packages
 -> xpumanager_1.3.7_...u24.04_amd64.deb
 -> custom entrypoint
 -> xpumd + REST exporter on port 29999

My old active image looked like this in the manifests:

ghcr.io/sonda-red/xpumanager@sha256:...

That is useful when you need to unblock yourself. It is not where I want the lab to stay.

This is the kind of thing that works, becomes normal, and then becomes suspicious because you cannot quite remember why all of it exists.

02 The DRA monitor shape I was patching toward

The other reason my old setup became awkward was DRA.

I have written about DRA a few times in this series already, so I will not re-explain the whole model here. The specific part that matters for this note is monitor access.

Intel’s GPU DRA monitor deployment documentation includes a monitor pod example using a ResourceClaimTemplate with monitor-style access.

The YAML is verbose, but the important part is the claim shape:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: monitor-claim
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 deviceClassName: gpu.intel.com
 adminAccess: true
 allocationMode: "All"
 tolerations:
 - effect: NoExecute
 - effect: NoSchedule
---
apiVersion: v1
kind: Pod
metadata:
 name: monitor-pod
spec:
 restartPolicy: Never
 containers:
 - name: monitor
 image: registry.k8s.io/e2e-test-images/busybox:1.29-2
 command: ["sh", "-c", "ls -la /dev/dri/ && sleep 60"]
 resources:
 claims:
 - name: resource
 resourceClaims:
 - name: resource
 resourceClaimTemplateName: monitor-claim

The important distinction is:

a normal inference pod should request the GPU resources it needs
a monitor needs visibility into devices on the node
the monitor should not make those devices unavailable to actual workloads

So the shape I wanted was clear:

DaemonSet
 -> runs on the GPU node
 -> gets DRA monitor access
 -> exposes Prometheus metrics
 -> does not consume a normal GPU allocation

The problem was that I was assembling that shape myself by patching upstream manifests after the fact.

03 What changed upstream

XPU Manager 2.0 changes the shape quite a bit. The upstream XPUM 1.x vs 2.x changes cover the broader redesign; the part that mattered for my cluster was the exporter and deployment shape.

The old 1.x world had a C++ daemon and a separate Python exporter path. Metrics came out with XPUM-specific names like:

xpum_power_watts
xpum_temperature_celsius
xpum_memory_used_bytes

XPUMD 2.x is a Go daemon built around OpenTelemetry components and Level Zero Go bindings. The Prometheus endpoint is still there, but the exported metric names now follow OpenTelemetry-style conventions, for example:

hw_gpu_utilization_ratio
hw_temperature_celsius
hw_memory_usage_bytes

The chart also understands GPU monitor access directly:

gpuAccess: dra

That single value is the part I had been patching toward. It tells the chart to use the monitor shape above instead of making me splice that behavior into the DaemonSet after rendering.

So instead of patching the upstream DaemonSet after the fact, I can tell the chart what I mean.

04 The new shape in my cluster

The new local module is much simpler.

First, the OCI chart source:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: xpumanager
 namespace: intel
spec:
 interval: 1m
 type: oci
 url: oci://ghcr.io/intel/xpumanager/charts

Then the HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: xpumanager
 namespace: intel
spec:
 interval: 5m
 chart:
 spec:
 chart: xpumd
 version: 2.0.0
 sourceRef:
 kind: HelmRepository
 name: xpumanager
 namespace: intel

And the values that matter for my cluster:

fullnameOverride: intel-xpumanager

image:
 repository: ghcr.io/intel/xpumanager/xpumd
 tag: "v2.0.0"

gpuAccess: dra

nodeSelector:
 kubernetes.io/hostname: sonda-core

tolerations:
 - key: dedicated
 operator: Equal
 value: gpu
 effect: PreferNoSchedule

extraEnv:
 - name: ZES_ENABLE_SYSMAN
 value: "1"

prometheus:
 monitor: true
 release: kube-prometheus-stack

config:
 service:
 pipelines:
 metrics:
 receivers: [intelxpu]
 processors: [intelxpustatus]
 exporters: [intelxpuinfo, prometheus]

In my lab, I hardcode the hostname because sonda-core is the GPU node.

In a larger cluster, I would use GPU node labels, for example labels from Node Feature Discovery, instead of binding the monitor DaemonSet to a specific node name.

With these values enabled, the chart now creates the pieces I used to maintain locally:

ServiceAccount
ConfigMap
DaemonSet
Service
ResourceClaimTemplate
ServiceMonitor

05 The metric name situation

My old local dashboard queried xpum_* metrics. That dashboard had to go because XPUMD 2.x exposes different metric names.

I switched Grafana provisioning to the tagged upstream 2.0.0 dashboard:

https://raw.githubusercontent.com/intel/xpumanager/refs/tags/v2.0.0/xpumd/charts/xpumd/json/dashboard.json

After switching the dashboard provisioning, the new panels came back like this:

Examples of the new metric shape:

avg by (pci_bdf) (hw_gpu_utilization_ratio{node="$Node"})

hw_temperature_celsius{statistic="max", node="$Node", hw_sensor_location="gpu"}

hw_memory_usage_bytes{node="$Node"} / hw_memory_size_bytes{node="$Node"}

I like this direction. The names are less tied to XPUM internals and more aligned with the broader telemetry ecosystem.

The practical migration note is simple: the pods can be healthy while the dashboard is blank. Check the metric names before blaming Prometheus.

06 What this removes from my life

This is what I got to delete or stop caring about:

my custom ghcr.io/sonda-red/xpumanager image
the custom Dockerfile that rebuilt XPUM 1.x around Ubuntu, Python, Gunicorn, and Intel userspace packages
the patched upstream DaemonSet source
the separate XPU Manager Service
the separate ServiceMonitor Kustomization
the hand-written monitor ResourceClaimTemplate
the old xpum_* dashboard
the Renovate custom manager that tracked XPU Manager as a Git tag

I do not regret the custom image. It got me telemetry when I needed telemetry.

In this case, the final result is boring, but that is a good thing. I can delete the workaround and move on.

Flux HelmRelease
 -> upstream xpumd chart 2.0.0
 -> upstream xpumd image v2.0.0
 -> chart-managed DRA monitor claim
 -> chart-managed ServiceMonitor
 -> upstream XPUMD 2 dashboard

Kleym: Intro to creating SPIFFE workload identity from inference intent

Fri, 05 Jun 2026 00:00:00 +0000

This note introduces Kleym, my experimental Kubernetes operator for turning inference intent into deterministic SPIFFE identity registration through SPIRE Controller Manager.

Disclaimer: This is lab documentation, not production guidance. It explains why I built Kleym, what boundary it covers today, and what I still need to test.

00 Personal notes
01 What am I doing here?
02 The problem I am testing
03 What Kleym does today
04 The two identity modes
05 Why not just write ClusterSPIFFEID YAML?
06 Reaching some limitations
07 What is ahead

Some personal notes follow below. If you want the technical part, skip to 01 What am I doing here? or use the table of contents.

00 Personal notes

I started this project because I needed something tangible that connected my work direction to the personal AI-on-Kubernetes work I’ve been describing here for a while.

My career has been following the pattern of finding a niche, then a niche within the niche. Two very important milestones:

Kubernetes - I knew I wanted to focus on it. I quickly figured out the ecosystem is already too broad, so it’s best to actually choose what aspects of it intrigue you the most.
Cluster API - I knew I wanted to deal with Kubernetes problems in a Kubernetes-native way, extending the API and building controllers instead of just running pipelines or different kinds of scripts.

Professionally, I feel super lucky. I moved closer to controller work and got introduced to SPIFFE/SPIRE. Kleym became a way to study the operator author’s side more deliberately: reconciliation, CRDs, status, and controller-runtime. I had already used Go for a few small CLI tools instead of scripts, plus a larger web scraper project some years ago, but controller work was the missing layer I wanted to understand.

At the same time, my homelab pulled me into inference infrastructure. I just didn’t want this AI train to pass me by, and I wasn’t going to stop at a local ollama run; I wanted to run model serving on Kubernetes and understand the lower layers. It’s too interesting to pass up: models are not just files, GPUs are not just devices, and vLLM is not just another HTTP deployment.

The project I’m describing here became the overlap: small enough to learn controller mechanics and identity on my own time, under my rules, but connected enough to inference infrastructure that I don’t have to choose which one to compromise.

Time, doubt, and AI help

Figuring out what to work on when you have so little time between work and family is super hard. I spent two on-and-off months around November to December 2025 writing the first spec draft. Every choice feels wrong, the topic feels interesting only to you, and there is constant doubt about whether it’s worth it.

Luckily, I’ve felt these feelings before, and I know I just need to start somewhere and deal with the fact that almost every aspect of this exercise has tradeoffs. If I told myself it would take me more than 6 months to have a minimal controller project with a narrow first boundary, I’d honestly just not start. It’s disheartening projecting these thoughts into the future, but looking back, I’m super glad I started. I have this pet project, and nobody can take away what I’ve learned through it. It’s more valuable than any tech reading list you could possibly give me.

Deciding to use AI as an assistant has cut the time spent making this by orders of magnitude, and it’s an interesting challenge to learn how to manage surrendering versus keeping control over a concept and implementation. I’ve been using mostly OpenAI Codex as a helper, as well as various types, flavours, and configurations of local models.

Giving it a name

First, I called it Terence. Get it? Trusted Inference? No? Anyway… as soon as it started to take shape and was no longer just a joke for me, I started moulding the repo more seriously into what I understand as a well-structured Kubernetes controller project. I renamed it to Kleym, created a drill logo to go with my story, used my Hugo blog experience to create a docs page to log the design and learning process, and turned sonda.red into a mix of notes, projects, and technical work coming from me.

You can explore the Kleym codebase at github.com/sonda-red/kleym, and the docs are published at kleym.sonda.red. I’m creating releases, and the project is still settling as I document the design and test the boundaries.

01 What am I doing here?

Workload identity sits in the infrastructure/security part of the stack, and you’ve probably seen it mentioned in that context and/or maybe under the guise of zero trust. Very briefly, it’s the idea that a running workload should be able to prove “this is who I am” without relying on a long-lived secret. SPIFFE is a standard way to name that workload with a SPIFFE ID, for example:

spiffe://example.org/ns/llm-d/pool/qwen3

SPIRE is one implementation of SPIFFE. It can issue short-lived credentials, called SVIDs, to workloads after matching them through selectors such as namespace, service account, pod labels, and container name.

Kleym sits one step before that issuance path. It’s a Kubernetes operator that watches inference-aware resources from the Gateway API Inference extension and registers the expected logical identity of the model-serving workload in SPIRE.

02 The problem I am testing

The earlier notes followed the stack from running a model locally, to containerizing it, to building it for Kubernetes.

The Intel AI inference MVP got the first stable layer working: Intel Arc GPUs, DRA, ModelKits, Harbor, vLLM, Open WebUI, and enough observability to keep the stack understandable.

The llm-d note changed the routing model. InferencePool became an LLM-aware backend instead of a Service, because a model-serving pool is not just another group of interchangeable HTTP pods.

After that, the Gateway API / agentgateway work made the routing boundary explicit because Ingress could not follow the shape of the inference stack.

Securing the flow was a natural next issue to tackle because it just didn’t feel serious to run vLLM with --api-key. As with everything else in this inference homelab journey, the solution just wasn’t there initially.

I now had resources that describe inference intent, but default SPIRE configuration just produced a normal workload identity based on pod labels and service accounts. I could fine-tune a ClusterSPIFFEID to be more specific, but I’m changing models, scaling, scheduling, and routing too often to write and maintain those by hand. An obvious automation solution was a controller.

To get into a bit more detail

InferencePool describes where a serving pool lives
InferenceObjective can describe a more specific model-serving target
the workload service account is still a required identity boundary
SPIRE Controller Manager already has ClusterSPIFFEID as the registration API
SPIFFE IDs should stay logical and stable even when pods, nodes, and devices change

The difference is easier to see in YAML.

A normal Service selects pods and exposes a port. It does not say that the selected pods form a model-serving pool:

apiVersion: v1
kind: Service
metadata:
 name: qwen3
 namespace: llm-d
spec:
 selector:
 app: qwen3
 ports:
 - port: 8000
 targetPort: 8000

An InferencePool still needs selectors, but the object is part of an inference-aware API. Trimmed to the field Kleym currently consumes, it can become the stable source of “these pods are the serving pool I mean”:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
 name: qwen3
 namespace: llm-d
spec:
 selector:
 matchLabels:
 app: qwen3

Kleym needed to get the same selectors from the resource that describes the serving pool, instead of copying those labels into a separate identity manifest by hand.

03 What Kleym does today

Kleym has two pieces today.

The in-cluster kleym-operator watches an InferenceIdentityBinding custom resource, resolves Gateway API Inference Extension resources such as InferencePool and InferenceObjective, and renders a SPIRE Controller Manager ClusterSPIFFEID.

I’ve also made a kleym CLI to inspect what the operator is doing on a live cluster. The status command summarizes the visible install, required CRDs, operator readiness, binding health, and findings.

kleym status

Kleym
 CLI: v0.7.4
 Operator: Available
 Deployment: kleym-system/kleym-operator
 Ready: 1/1
 Version: v0.7.4
 Config:
 trustDomain: sonda.red.intra
 clusterSPIFFEIDClass: kleym
 API:
 InferenceIdentityBinding: v1alpha1

Bindings
 Total: 1
 Conditions:
 Ready: 1
 Conflict: 0
 InvalidRef: 0
 UnsafeSelector: 0
 RenderFailure: 0

Dependencies
 GAIE: Available
 InferencePool: v1
 InferenceObjective: v1alpha2
 SPIRE: Available
 ClusterSPIFFEID: v1alpha1

The inspect command resolves one binding and shows the rendered identity, expected ClusterSPIFFEID, current conditions, Kubernetes-visible matched pods, and findings.

Shortened, an inspection is meant to answer questions like this:

kleym inspect binding qwen3 -n llm-d

Identity
 Mode: PoolOnly
 SPIFFE ID: spiffe://kleym.sonda.red/ns/llm-d/pool/qwen3

ClusterSPIFFEID
 Name: kleym-operator-llm-d-qwen3-...
 Selectors:
 k8s:ns:llm-d
 k8s:sa:qwen3
 k8s:pod-label:app:qwen3

Matched pods
 llm-d/qwen3-decode-0
 llm-d/qwen3-decode-1

Findings
 none

Detailed info on the InferenceIdentityBinding can be found in the docs, but overall:

poolRef anchors the identity to an InferencePool
objectiveRef optionally narrows the identity to an InferenceObjective
serviceAccountName gives the service account boundary
mode chooses PoolOnly or PerObjective
containerName gives PerObjective an extra container boundary

The rendered SPIFFE IDs stay logical to that boundary:

spiffe://<trust-domain>/ns/<namespace>/pool/<pool-name>
spiffe://<trust-domain>/ns/<namespace>/objective/<objective-name>

Under the hood, the operator does a few things to complement the identity registration:

resolve the referenced inference resources
derive selectors from the pool
add namespace and service-account safety selectors
add the container selector for PerObjective
refuse unsafe selectors
refuse overlapping per-objective identities
remove stale managed output when the binding becomes invalid
report the reason through status conditions

04 The two identity modes

The modes are where the inference-specific part becomes more concrete.

PoolOnly: one logical model pool, multiple replicas

In the llm-d note I wrote about what changes when a model-serving workload scales beyond one pod. The replicas are still part of one logical model-serving pool, but routing and scheduling become more interesting because of GPU placement, request cost, and cache locality.

From an identity point of view, the first simple rule is:

If two pods are serving the same logical model pool, they usually need the same workload identity.

For example:

qwen3-decode-0 app=qwen3 serviceAccount=qwen3 -> spiffe://kleym.sonda.red/ns/llm-d/pool/qwen3
qwen3-decode-1 app=qwen3 serviceAccount=qwen3 -> spiffe://kleym.sonda.red/ns/llm-d/pool/qwen3

That is the PoolOnly case:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
 name: qwen3
 namespace: llm-d
spec:
 selector:
 matchLabels:
 app: qwen3
---
apiVersion: kleym.sonda.red/v1alpha1
kind: InferenceIdentityBinding
metadata:
 name: qwen3
 namespace: llm-d
spec:
 poolRef:
 name: qwen3
 serviceAccountName: qwen3
 mode: PoolOnly

Expected selector shape:

k8s:ns:llm-d
k8s:sa:qwen3
k8s:pod-label:app:qwen3

PerObjective: one pool, separate model objectives

The second case is when the pod or serving pool boundary is too broad.

Imagine one pool where different containers, or different objective-level serving paths, should not share the same identity. A chat model and an embedding model might be deployed together for operational reasons, but downstream systems may need to tell them apart.

In that case, the identity needs one more boundary:

pod: multi-model-0
 container: qwen3-chat -> spiffe://kleym.sonda.red/ns/llm-d/objective/qwen3-chat
 container: bge-embed -> spiffe://kleym.sonda.red/ns/llm-d/objective/bge-embed

That is the PerObjective case. Kleym requires containerName here because the pool selector alone would match the whole pod set:

apiVersion: inference.networking.k8s.io/v1
kind: InferenceObjective
metadata:
 name: qwen3-chat
 namespace: llm-d
spec:
 poolRef:
 name: qwen3
---
apiVersion: kleym.sonda.red/v1alpha1
kind: InferenceIdentityBinding
metadata:
 name: qwen3-chat
 namespace: llm-d
spec:
 poolRef:
 name: qwen3
 objectiveRef:
 name: qwen3-chat
 serviceAccountName: qwen3
 mode: PerObjective
 containerName: qwen3-chat

Expected selector shape:

k8s:ns:llm-d
k8s:sa:qwen3
k8s:pod-label:app:qwen3
k8s:container-name:qwen3-chat

If two PerObjective bindings resolve to the same pool selector and the same container name, Kleym treats that as a collision. Different logical objective identities would land on the same workload slice, so the operator refuses to create ambiguous registrations.

05 Why not just write ClusterSPIFFEID YAML?

For a simple setup, writing the ClusterSPIFFEID by hand is fine.

If there is one namespace, one model server, one service account, one stable label set, and one person reviewing the manifests, direct ClusterSPIFFEID YAML is often enough.

The direct version looks something like this:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
 name: qwen3
spec:
 spiffeIDTemplate: spiffe://kleym.sonda.red/ns/llm-d/pool/qwen3
 podSelector:
 matchLabels:
 app: qwen3
 workloadSelectorTemplates:
 - k8s:ns:llm-d
 - k8s:sa:qwen3
 - k8s:pod-label:app:qwen3

Kleym’s value is not YAML generation. The value is that it makes the identity boundary explicit and repeatable:

the binding namespace is always part of the selector set
the service account is always part of the selector set
pool-derived selectors come from the referenced inference resource
PerObjective identities require a container boundary
overlapping objective identities are treated as a collision instead of quietly producing ambiguous registrations
status conditions make invalid references, unsafe selectors, render failures, and collisions visible

I am still testing whether this boundary deserves a standalone controller or should stay as a pattern documented in the lab.

06 Reaching some limitations

Currently, Kleym registers expected identity and stops. It does not prove that identity has been issued, consumed, or enforced.

When a binding reaches Ready=True:

InferenceIdentityBinding is reconciled
ClusterSPIFFEID is rendered
SPIRE registration intent exists

It does not mean:

SVID issued
workload consumed the SVID
gateway enforced mTLS
request authorization happened
model artifact was verified
runtime state was attested

All of that still belongs to the cluster administrator and the surrounding platform. SPIRE Server and Agent must be installed correctly. The workload, sidecar, proxy, gateway, or application must actually consume the SVID. Downstream policy still has to validate the SPIFFE ID and decide what it allows. Envoy SDS, mTLS, external authorization, OPA, JWT-SVID exchange, route policy, and audit are downstream integration work, not things Kleym currently manages.

That may sound like a small scope, but I think it is the right one for the project today. Identity registration is already useful if it is deterministic, inspectable, and tied to inference intent. It becomes dangerous if the project starts implying more than that.

The future runtime question is still interesting:

Does the workload that Kleym manages identity-wise appear to be using the expected model artifact and compute placement?

Host and runtime signals can help with inspection and drift detection: process, cgroup, device placement, file path, DRA allocation, GPU usage, or what vLLM reports. Reaching this point has mostly taught me that complex systems have many edges, and that there are many ways to be wrong.

07 What is ahead

The roadmap is mostly about making the current boundary stronger before adding anything larger.

First, the identity compiler itself needs to become more stable, and the proper identity format still needs more exploration.

Second, the project should be easier to try and operate without becoming a platform installer. Better install paths, clearer configuration, observability examples, other demos, etc. Installing SPIRE, model servers, gateways, Envoy, OPA, or policy resources is already quite a bit of work by itself.

Third, runtime evidence can stay on the research side until the static contract is stable. The line I want to preserve is:

Kleym core:
 compile expected logical identity

Future runtime evidence:
 inspect and classify runtime state through an attestor-style extension

SPIRE:
 remain the authority that issues identity

That keeps the current project honest while leaving room for the more interesting runtime work.

Maybe Kleym becomes useful to someone else. Maybe it mostly teaches me where this abstraction breaks. Both are better than waiting for the perfect project.

Back to the code.

Lab Notes: Why llm-d pushed me out of Ingress (and into agentgateway)

Tue, 14 Apr 2026 00:00:00 +0000

This note documents why my inference path moved away from Ingress after llm-d changed the routing problem from “which Service” to “which model-serving backend should handle this request right now.”

Link to Part 4: Intel AI Inference Platform MVP 2 with llm-d

Disclaimer: This is not production guidance and it is not sponsored. It documents what actually ran in my homelab. Please double-check before you roll it into your own setup.

Table of Contents
00 I came back from vacation to a broken routing layer
01 Why Ingress couldn’t follow
02 The migration I didn’t plan for
03 When timeouts become product behavior
04 Where the stack landed
05 Three lessons from three migrations
References

00 I came back from vacation to a broken routing layer

So I went on vacation for a week. Before I left I had queued a minor version bump on the gateway provider. Came back, checked Flux, and nothing was routing. All inference traffic was dead. Turns out the provider had deprecated the AI Gateway path I was using and I just hadn’t noticed the release notes.

That was fun to debug on a Monday morning. But it also made me realize this whole thing was not one migration but three, and I should probably write down how I got here.

In Part 4, I introduced llm-d. It was a solution after my original thought that inference on Kubernetes can follow a “standard” path of:

Get a model running locally
Run it in a container
Run it in Kubernetes
Expose it with Ingress

Step 4 was the original plan. I had Ingress in place for all my other apps, so it felt natural to just add another route.

With one model pod, Ingress looks fine. With two replicas serving the same model, the question changes from “which Service” to “which replica should take this request right now.” Ingress only sees HTTP endpoints. It does not know anything about decode pods, cache locality, or inference-specific backends, while llm-d’s scheduler does.

 ┌───────┐
 │Client │
 │ │
 └───────┘
 │
 HTTPS
 │
 ▼
 ┌──────────────┐
 │ingress-nginx │
 │ │
 └──────────────┘
 │ │ │ │
 ┌─────────────┘ │ │ └─────────┐
 │ │ │ │
 │ ┌───┘ └────┐ │
 │ │ │ │
 ▼ ▼ ▼ │
 ┌────────────┐ ┌────────┐ ┌────────┐ │
 │ OpenWebUI │ │ MinIO │ │Grafana │ Service
 │ │ │ │ │ │ │
 └────────────┘ └────────┘ └────────┘ │
 │
 ▼
 ┌─────────────┐
 │vLLM Service │
 │ │
 └─────────────┘
 │ │
 ┌────┘ └────┐
 │ │
 ▼ ▼
 ┌───────────┐ ┌───────────┐
 │vLLM pod A │ │vLLM pod B │
 │ │ │ │
 └───────────┘ └───────────┘

Stage 1: Ingress NGINX

Once llm-d became the center of inference, Ingress stopped fitting naturally.

The routing question was no longer “which Service” but “which inference pool backend.” Inference pools are a new kind of backend that encode model-serving semantics, and they are only supported in Gateway API with the Inference Extension.

Kubernetes docs now explicitly recommend Gateway API over Ingress, and the Ingress API is marked as frozen (stable, but no new feature development). Around the same time, ingress-nginx retirement was announced on November 11, 2025, with best-effort maintenance through March 2026.

So the path became clear: keep existing Ingress where needed, but invest new routing work in Gateway API.

One dependency bump later, I learned this was not one migration but three:

Ingress -> Gateway API because of llm-d
kgateway -> agentgateway because the provider path changed
Default timeouts -> explicit timeout policies because LLM traffic is long-lived

I originally chose kgateway for its early Gateway API support, but the provider ecosystem is still evolving. When agentgateway emerged with a more focused vision on AI workloads, it made sense to follow that path.

01 Why Ingress couldn’t follow

llm-d depends on Gateway API Inference Extension CRDs, so Gateway API became a hard dependency in this repo.

# infrastructure/gateway-api/gateway-api-inference-extension.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: gateway-api-inference-extension
 namespace: gateway-system
spec:
 url: https://github.com/kubernetes-sigs/gateway-api-inference-extension
 ref:
 tag: v1.4.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: gateway-api-inference-extension
 namespace: gateway-system
spec:
 path: ./config/crd
 dependsOn:
 - name: gateway-api

Then the route itself stopped targeting a plain Service and started targeting an InferencePool.

# deployments/llm-d/inference-scheduling/httproute.yaml
rules:
 - backendRefs:
 - name: llm-d-inferencepool
 kind: InferencePool
 group: inference.networking.k8s.io
 port: 8200

And the pool encodes model-backend semantics (v1 API, decode label selection, target port):

# deployments/llm-d/inference-scheduling/inferencepool/inferencepool-values.yaml
inferencePool:
 apiVersion: inference.networking.k8s.io/v1
 targetPortNumber: 8200
 modelServerType: vllm
 modelServers:
 matchLabels:
 llm-d.ai/role: "decode"

At that point I could have just left Ingress in place for the non-inference apps and only used Gateway API for llm-d. I thought about it for maybe a day. Running two routing stacks in parallel sounded like the kind of decision I’d regret every time something broke and I had to check both. Since I already had to learn Gateway API anyway, I just moved everything over.

The part I didn’t plan for came next.

02 The migration I didn’t plan for

A concrete timeline from my infra repo:

f01d2cf (2026-02-07): migrate app ingress routes to Gateway API
0a1415d (2026-02-07): migrate infra ingress routes (Flux, MinIO, Grafana, Prometheus, VictoriaLogs)
829d7a4 (2026-02-13): split OpenWebUI route behavior and add long-stream policy
7b9a137 -> 4fde0b6 (2026-04-07): Gateway API dependency bump to v1.5.1, then revert to v1.4.1
1904628 + 8397a51 (2026-04-07): kgateway -> agentgateway refactor plus listener/route cleanup

The revert in step 4 was where I learned about the provider migration. I had been following kgateway releases, but I missed the deprecation notice for AI Gateway support. When I bumped to v1.5.1, all my inference routes stopped working and I had to dig into release notes and code to understand why.

The key nuance: kgateway was not dead. The AI/inference path moved. In kgateway 2.1 release notes, AI Gateway and Gateway API Inference Extension support on Envoy-based proxies was marked deprecated in favor of agentgateway proxy support, with removal planned in 2.2. Then 2.2 introduced dedicated agentgateway.dev APIs and a separate chart/controller split (release notes, 2.2 breaking changes).

The obvious diff looked small:

# commit 1904628 (llm-d infra values)
- provider: kgateway
+ provider: agentgateway

- gatewayClassName: kgateway
+ gatewayClassName: agentgateway

But there was more surface area:

# commit 1904628 (route parent refs)
- namespace: kgateway-system
+ namespace: agentgateway-system

# commit 8397a51 (listener + route cleanup)
- - name: infer-https
+ - name: inference-gateway-https

- - name: agtw-https
+ - name: agentgateway-admin-ui-https

- # HTTP to HTTPS redirect routes
+ # removed and consolidated around HTTPS listeners

After I stabilized these bindings, the next bottleneck was connection behavior.

03 When timeouts become product behavior

The long-connection behavior of LLM inference and UI sessions forced me to treat timeout config as architecture, not optional tuning.

For external inference traffic:

# deployments/llm-d/inference-scheduling/httproute-external.yaml
rules:
 - matches:
 - path:
 type: PathPrefix
 value: /
 timeouts:
 backendRequest: "3600s"
 request: "0s"

For OpenWebUI:

# deployments/openwebui/httproute.yaml
rules:
 - matches:
 - path:
 type: PathPrefix
 value: /
 timeouts:
 backendRequest: "3600s"
 request: "0s"

If you serve short request/response APIs, defaults are often fine. If you serve token streams and slower model turns, default timeout assumptions can quietly wreck UX. I started receiving strange “connection reset” errors in OpenWebUI and llm-d, and it took a while to connect the dots that these were not random network issues but timeout policies kicking in.

04 Where the stack landed

The current gateway shape is one shared main-gateway in agentgateway-system, with explicit HTTPS listeners per hostname. But to appreciate how it got here, here’s what the kgateway stage looked like before it broke:

 ┌───────┐
 │Client │
 │ │
 └───────┘
 │
 HTTPS
 │
 ▼
 ┌─────────────┐
 │main-gateway │
 │ │
 └─────────────┘
 │ │
 ┌─────┘ │
 │ │
 │ infer FAILED
 chat │
 │ │
 ▼ ▼
 ┌────────────┐ ┌──────────────────┐
 │ OpenWebUI │ │inference-gateway │
 │ │ │ │
 └────────────┘ └──────────────────┘
 │
 ▼
 ┌──────────────┐
 │InferencePool │
 │ │
 └──────────────┘
 │
 ▼
 ┌──────┐
 │ EPP │
 │ │
 └──────┘
 │
 ▼
 ┌────────────┐
 │decode pods │
 │ │
 └────────────┘

Stage 2: kgateway — the inference path (❌) broke when AI Gateway support was deprecated

The red link is the one that broke. kgateway 2.1 deprecated AI Gateway and Inference Extension support on Envoy proxies, and 2.2 removed it. The inference path moved to agentgateway, and everything had to follow.

The clearest way to think about the current layout is to separate public endpoints from inference-internal routing. chat.sonda.red.intra is the browser UI. infer.sonda.red.intra is the single OpenAI-compatible API surface (/v1/...) used by OpenWebUI and direct clients. The hostname tells you which API surface you are hitting; the request body still carries model=..., and the llm-d layer still has to pick a concrete backend pod.

 ┌──────────┐┌───────────┐
 │ Browser ││API client │
 │ ││ │
 └──────────┘└───────────┘
 │ │
 │ infer
 chat │
 │ │
 │ ┌───────┘
 │ │
 ▼ ▼
 ┌─────────────┐
 │main-gateway │
 │ │
 └─────────────┘
 │ ▲ │
 │ │ └─────────┐
 │ │ │
 UI infer │
 │ │ API
 │ │ │
 ▼ │ ▼
 ┌────────────┐ ┌──────────────┐
 │ OpenWebUI │ │llm-d gateway │
 │ │ │ │
 └────────────┘ └──────────────┘
 │
 choose
 │
 ▼
 ┌──────┐
 │ EPP │
 │ │
 └──────┘
 │
 selected
 │
 ▼
 ┌───────────┐
 │decode pod │
 │ │
 └───────────┘

Stage 3a: runtime request path through agentgateway and llm-d.

 ┌───────────────────┐┌───────────────────┐
 │External HTTPRoute ││Internal HTTPRoute │
 │ ││ │
 └───────────────────┘└───────────────────┘
 │ │ │ │
 host fwd attach ref
 │ │ │ │
 ▼ ▼ ▼ ▼
 ┌─────────────┐ ┌──────────────┐ ┌──────────────┐
 │main-gateway │ │llm-d gateway │ │InferencePool │
 │ │ │ │ │ │
 └─────────────┘ └──────────────┘ └──────────────┘
 │
 labels
 │
 ▼
 ┌────────────┐
 │decode pods │
 │ │
 └────────────┘

Stage 3b: Gateway API and InferencePool objects bind hostnames, routes, and backend selection.

This was the mental model I was missing at first. HTTPRoute and InferencePool are not extra network hops in the same sense as Gateway -> pod. They are the objects that tell the llm-d inference gateway how to resolve a request. The actual runtime path is simpler: request hits main-gateway, inference traffic gets handed to the llm-d gateway, EPP picks one backend from the pool, and only then does a specific decode pod answer.

# infrastructure/agentgateway/gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: main-gateway
 namespace: agentgateway-system
spec:
 gatewayClassName: agentgateway
 listeners:
 - name: openwebui-https
 hostname: chat.sonda.red.intra
 port: 443
 protocol: HTTPS
 - name: inference-gateway-https
 hostname: infer.sonda.red.intra
 port: 443
 protocol: HTTPS

This ended up cleaner than what I had before:

Gateway API is now a stable dependency of the inference stack
llm-d routing semantics are explicit in manifests
The provider migration is complete and aligned with the current control plane
Long-lived connection behavior is handled in route policy, not left to defaults

05 Three lessons from three migrations

What I expected to be one migration taught me three separate lessons:

Inference routing is not generic web routing. I think I’m spending the bulk of my time on these issues because of the unique semantics of LLM workloads, not just because of the newness of Gateway API.
Gateway provider lifecycle matters as much as application lifecycle. I missed one release, went on vacation, came back to a dead routing layer. Things are moving fast in this space and nobody is going to wait for you to catch up.
Timeout policy is part of product behavior when LLMs are in the loop. A connection is more akin to a session than a request, and the “request” can be arbitrarily long.

References

Lab Notes: Intel AI Inference Platform MVP 2 with llm-d

Mon, 08 Dec 2025 00:00:00 +0000

This note extends the Intel Arc inference MVP with llm-d, Gateway API Inference Extension concepts, and the routing layer needed once vLLM replicas stop behaving like interchangeable HTTP pods.

Link to Part 3: Intel AI Inference Platform MVP

Disclaimer: This is not production guidance and it is not sponsored. It documents what actually ran in my homelab with Arc GPUs and k3s. Please double-check before you roll it into your own setup.

Table of Contents
00 Scale a vLLM replica to 2 and see what happens
01 What made llm-d click for me
02 LLMs as CPUs, and what Kubernetes does not know yet
03 How llm-d slots into the Intel MVP
04 The three llm-d components in this cluster
05 Lighting the path
06 Conclusion: why llm-d matters to me

00 Scale a vLLM replica to 2 and see what happens

In Part 3 I ended with a working Intel only inference MVP:

Models packaged as ModelKits via KitOps and stored in Harbor
A shared PVC populated by kitops-init for fast local model loading
Intel Arc A770 GPUs allocated through Dynamic Resource Allocation
vLLM with IPEX as the serving engine behind an OpenAI compatible API
Open WebUI on top, with xpumanager and Prometheus watching everything

That stack did the job. DeepSeek R1 ran on my GPUs, the configuration was reproducible and stable.

Something was bugging me however. In Kubernetes we’re used to 12 factor apps, or at least close enough to that philosophy so that you treat scale, replication and stateless behavior as default and somewhat straightforward to manage.

The vLLM deployments I made previously are one pod, with one model, loaded in one GPU. Tensor parallelism enables this same one pod to house its model on two or more GPUs but the single replica point of failure remains. You don’t just scale a vLLM or similar deployment to 2 replicas. You’ll get another instance on another GPU you can put a load balancer in front of two separate inference workloads but it’s a mistake to consider them basic http services:

Requests have wildly different prompt and response lengths
Each replica holds its own KV and prefix cache
Long prompts and agent loops can pin a GPU for quite a while

I feel lucky to be honest. I feel lucky I chose vLLM as my main inference engine in the lab by gut feeling, didn’t really dig deeper until later. I’m also lucky I decided to take a listen to the Kubernetes Podcast while I was loading the dishwasher one night and stumbled upon the llm-d episode mid October: https://kubernetespodcast.com/episode/258-llmd/

You can try to imagine me holding a dirty dish and just listening, forgetting about the dishwasher at all. Says lots about the talk but yeah, in short:

This post is an extension of my at-home inference MVP: the same hardware and the same vLLM engine, but with llm-d and the Gateway API Inference Extension sitting in the middle as a real routing and scheduling layer.

01 What made llm-d click for me

I loaded the dishwasher and instantly set to search what the hell is llm-d and can I install it. The docs at llm-d.ai were somewhat alright, but I also found an interesting, though rather short Hacker News thread where I also learned about Nvidia’s Dynamo inference solution but that’s for another time.

In that thread, one of the maintainers described llm-d as three clean layers.

Balance and schedule incoming requests to the right backend
Run model server replicas on different hardware topologies
Provide a prefix caching hierarchy with tuned variants for different use cases

So llm-d is not “yet another AI platform.” It is a very opinionated three tier architecture:

A routing and scheduling plane
A pool of model servers (vLLM in my case)
A cache hierarchy for tokens and prefixes

It uses the Gateway API Inference Extension to define routing, request priorities and flow control in Kubernetes owned APIs.

02 LLMs as CPUs, and what Kubernetes does not know yet

The Podcast episode I mentioned in the beginning made me see the stack with a different analogy: an LLM is a new kind of CPU.

vLLM is the microarchitecture
The GPU plus context window plus KV cache is the “core”
Each request is a process whose “cost” depends on token shapes
The prefix cache is a special memory that can dramatically change cost

Kubernetes today is very good at:

CPU and memory allocation
Container scheduling and bin packing
Generic load balancing through Services and Ingress

It has no built in concept of “LLM core capacity” or cache locality.

That missing layer is exactly where llm-d and the Gateway API Inference Extension fit. They sit between “L4 networking” and “model server,” and they give Kubernetes some language for this new CPU.

03 How llm-d slots into the Intel MVP

The nice part: I did not have to throw away anything from Part 3.

vLLM stays
Intel GPUs and DRA stay

Interestingly I implemented llm-d before they had DRA support and still used GPU plugins as default. I had to Frankenstein a solution between their helmfile(which I love with my whole heart) approach and my new love of FluxCD. Thankfully, the PR enabling DRA support got merged quite quickly https://github.com/llm-d-incubation/llm-d-modelservice/pull/144 and now I can run vLLM with DRA natively through llm-d’s modelservice chart.

ModelKit and kitops-init stay

However with a small but important change: I now use kitops-init to populate a PVC that is then mounted into the vLLM pods declared by llm-d’s modelservice chart. This way I can keep my existing model packaging and loading flow, however this time not as sidecar but as a volume mount.

GitOps with Flux and Kustomize stays

The updated flow looks like this:

A client or Open WebUI sends a request to an OpenAI compatible endpoint
kgateway receives it on port 80 and matches an HTTPRoute
The HTTPRoute forwards to an InferencePool instead of a Service
Envoy inside the gateway calls the Endpoint Picker (EPP) via External Processing
The EPP looks at pod load and cache hints and returns a single backend
Envoy forwards the request to the chosen vLLM pod

From the vLLM pod’s perspective, nothing changed. It is still just serving tokens over HTTP. From the cluster’s perspective, a whole new set of decisions became explicit and observable.

A full diagram of the added components:

04 The three llm-d components in this cluster

In my repo, the integration lives under llm-d/inference-scheduling (link). It breaks into three pieces.

04.1 modelservice: vLLM on Intel GPUs with DRA and ModelKit

The llm-d modelservice chart is where I declare “how many vLLM replicas, which image, which GPUs.”

The important bits from modelservice-values.yaml (full file):

modelArtifacts:
 name: "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"
 uri: "hf://deepseek-ai/DeepSeek-R1-Distill-Llama-8B" # placeholder; ModelKit PVC overrides

dra:
 enabled: true
 type: "intel-a770-single"
 claimTemplates:
 - name: "intel-a770-single"
 class: "gpu.intel.com"
 match: "exactly"
 count: 1
 selectors:
 - cel:
 expression: 'device.attributes["gpu.intel.com"].model == "A770"'

decode:
 replicas: 2
 containers:
 - name: "vllm"
 image: "intelanalytics/ipex-llm-serving-xpu:latest"
 command: ["/bin/bash", "-lc"]
 args:
 - |
 source /opt/intel/1ccl-wks/setvars.sh && \
 python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
 --served-model-name $(SERVED_MODEL_NAME) \
 --port 8000 \
 --model /data/ds-r1-llama-8 \
 --device xpu \
 --dtype float16 \
 --tensor-parallel-size 1 \
 --enable-prefix-caching
 volumeMounts:
 - name: modelkit
 mountPath: /data  # PVC populated by kitops-init

This is basically Part 3, wrapped in a chart:

vLLM with Intel IPEX
one A770 per decode pod from DRA
prefix cache enabled
models pulled from Harbor into /data by kitops-init

llm-d does not replace this. It assumes you know how you want to serve the model.

04.2 InferencePool: an LLM aware backend instead of a Service

The Gateway API Inference Extension adds InferencePool as a new kind of backend. It is like a Service that knows its pods are LLM servers.

Conceptually, my pool looks like this (inferencepool-values.yaml, full file):

inferenceExtension:
 image:
 name: epp
 hub: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension
 tag: v1.1.0
 extProcPort: 9002

inferencePool:
 apiVersion: inference.networking.k8s.io/v1
 targetPortNumber: 8000 # vLLM HTTP port
 modelServerType: vllm
 modelServers:
 matchLabels:
 llm-d.ai/role: "decode"

Three key ideas:

modelServers.matchLabels ties the pool to vLLM decode pods that serve the same model
targetPortNumber indicates which port speaks the model protocol
inferenceExtension packages the Endpoint Picker (EPP) that must be consulted before routing

This is the Kubernetes owned description of “these pods form a single logical LLM backend, and this is the brain that decides which one gets each request.” ([Hacker News][1])

04.3 HTTPRoute: Gateway to InferencePool

The last piece of the path is a regular HTTPRoute that targets the InferencePool instead of a Service (httproute.yaml, full file):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: llm-d-route
 namespace: llm-d
spec:
 parentRefs:
 - name: llm-d-infra-inference-gateway
 namespace: llm-d
 kind: Gateway
 rules:
 - backendRefs:
 - name: llm-d-inferencepool
 kind: InferencePool
 group: inference.networking.k8s.io
 port: 8000
 matches:
 - path:
 type: PathPrefix
 value: /
 timeouts:
 backendRequest: 3600s
 request: 3600s

Clients do not see any of this. They still send POST /v1/chat/completions with an OpenAI style payload. The only difference is that the backend reference is an InferencePool that has an Endpoint Picker attached.

04.4 Routing and DRA

The Endpoint Picker:

Tracks per pod load and queue depth
Uses cache hints and conversation metadata
Keeps multi turn flows on the same backend when that makes sense
vLLM exposes metrics about cache hits and token latency
EPP uses hints and past routing decisions to keep similar prompts together

In Part 3, DRA was mostly a cleaner way to say “give this pod one Arc A770”.

With llm-d in place:

DRA sets which pods have which GPUs
InferencePool defines which pods form a logical backend
EPP knows which backends exist and how busy they are

05 Lighting the path

The llm-d maintainer called out somewhere that their main users are large inference deployers who already run Kubernetes across providers, often mixing serving, batch and training on the same fleet.

I’m not a H100 datacenter, for which this tooling can generally solve a lot more problems, but even on my two A770s, you can see benefit and learning value from having this layer in place. Examples I have here are basically the Intelligent Inference Scheduling well lit path from their docs, however llm-d is more than that and capabilities have been steadily growing, since I first tried it in October.

When the llm-d docs talk about “well lit paths”, they mean concrete, tested recipes, not another “hello world” Helm chart. Each path is a configuration that’s been documented, benchmarked and wired the way real clusters run in production, so you start from something proven instead of hacking in the dark. Right now the main paths are:

Intelligent Inference Scheduling – vLLM running behind the Inference Gateway, with prefix cache aware routing and scheduling policies that actually move the needle on latency and throughput.

Prefill/Decode Disaggregation – splitting prompt handling and token generation across different servers to cut time to first token and stabilize time per output token on larger models and long prompts.

Wide Expert Parallelism – a setup for very large MoE models like DeepSeek R1, using data and expert parallelism across fast accelerator networks.

Tiered Prefix Cache – an extra path that layers a multi-tier prefix cache on top (for example offloading to CPU RAM) and can be combined with any of the above to squeeze more reuse out of long or concurrent workloads.

More in the official Well-Lit Path Guides.

06 Conclusion: why llm-d matters to me

Since I started this homelab project, the surface area has grown, but the core activity has not changed much:

Run DeepSeek R1 locally
Containerise it
Schedule it on Intel GPUs in Kubernetes
Scale it across more pods or devices

The outcome from the outside is trivial: a prompt goes in, tokens come out, some GPUs are busy. Hardware is the same.

The interesting part has moved from “can I make this run at all” to how the cluster thinks this workload is, tinkering with the idea how a production grade LLM inference setup could behave, dissecting the pieces and seeing how they fit together.

It’s just really fun.

Lab Notes: Intel AI Inference Platform MVP

Thu, 18 Sep 2025 00:00:00 +0000

This note documents the first stable inference layer in my Intel Arc Kubernetes homelab: model packaging, GPU allocation, OpenAI-compatible serving, OpenWebUI, and the observability needed to keep the stack understandable.

Link to Part 2: Laying the Cluster Foundation

Disclaimer: This is not production guidance and it is not sponsored. It documents what actually ran in my homelab with Arc GPUs and k3s. Please double-check before you roll it into your own setup.

Table of Contents
00 Reflecting on a year of learning
01 The Stack at a Glance
02 Packaging and Storage
03 GPU Allocation
04 Application Layer
05 Monitoring and Observability
- Aim
- Stack
- Actions I took
- Challenges
  - xpumanager
Conclusion

Some personal notes follow below. If you want to skip them, scroll to the next section or use the table of contents.

00 Reflecting on a year of learning

I ordered the first Intel Arc A770 almost exactly a year ago on September 17, 2024, and told myself I’ll learn how AI works on Kubernetes, because I’ve spent the money on the hardware anyway. I have to confess the build did gather dust for a few months due to life/work, but I did manage in roughly a year to achieve 3 key goals:

Stable inference MVP. Intel Arc GPUs run reliably inside Kubernetes.
My own OpenAI-compatible API on a budget. Two GPUs cost about €600 in total. I don’t pay tokens or cloud credits. It’s slower and less convenient than cloud.
Transferable knowledge. I stayed with open standards and open source so I can apply the same ideas elsewhere.

In this regard, I’m most grateful for the experience I gained with Kubernetes in general, FluxCD, the new concept of Dynamic Resource Allocation and packaging models with KitOps.

Not a lot, is it?

Well, everything seemed intimidating at first as it was very unfamiliar. I especially remember the first times reading docs or articles on the topic of LLMs and I didn’t understand more than 80% of the content. Terms like FP8, BF16, KV cache, context length, 8B/13B/70B param, VRAM, quantization 4-bit/8-bit, LoRA/QLoRA. Getting familiar with Python environments, dealing with Intel’s extension for PyTorch before XPU support got merged upstream. A lot of hours reading GitHub issues, because documentation was often and still is very much lagging behind or missing on the issues I was facing. Intel’s CEO switches and company restructuring don’t help the ecosystem either, and it was felt while trying to make all this work. But I digress…

I’m happy I now have a platform I can upgrade and develop on top of, while somewhat understanding it on a more fundamental level.

01 The Stack at a Glance

Users hit OpenWebUI. OpenWebUI talks to vLLM which serves an OpenAI-compatible API. Models are packaged with KitOps and stored in Harbor with blobs in MinIO. vLLM starts with KitOps as a sidecar and pulls weights from Harbor. GPU access uses Dynamic Resource Allocation (DRA) and Intel resource drivers. Prometheus and VictoriaLogs leverage the services and xpumanager to feed Grafana so I can see metrics and logs in one place.

I’ve named this article “MVP” because this is not a production-ready stack. But it’s stable, working, and I can iterate on it. You’ll see below in the notes where I describe the challenges how a lot of times a small part of the stack was not working as expected and broke balance.

The three main pillars of the stack are:

Packaging and Storage
GPU Allocation
Monitoring and Observability

These 3 allow the application layer to run reliably. I will cover each in turn below.

02 Packaging and Storage

This really is the backbone of the stack. If you can’t get models to the workloads reliably, the rest falls apart. When we’re dealing with small Docker containers, pulling a few hundred MBs from Docker Hub is no big deal.

Imagine baking in the 122GB DeepSeek-R1-Distill-Qwen-32B model into a container image and you had a typo in the Dockerfile. Are you going to rebuild it? How long will it take? How much space will it take on your registry? What if you want to try a different model? Do you bake it in again?

Let’s download it from HuggingFace into a volume then? That’s the solution, right? You’d have to architect a solution not only for reliable pulling but for versioning that handles exact dataset, code, hyperparameters, etc. You need to roll back or run on small differences between models. What’s the best practice for that?

Aim

Version models and move them between workloads without surprises.
Keep I/O paths short so large pulls and unpacks don’t stall pods.

Stack

MinIO (S3) - as a Helm release on the GPU node
Harbor (OCI registry) - as a Helm release on the GPU node
KitOps (ModelKits) - as a sidecar in vLLM workloads
Postgres and Redis for app state - as Helm releases on NUC nodes

Actions I took

Gave MinIO a large PV on the GPU node.
Set Harbor to use S3 as backend.

Both of these steps are described in more detail in the previous post.

Pinned Harbor to the GPU node with node affinity to avoid cross-node traffic.
Pushed models as KitOps ModelKits to Harbor and referenced them from vLLM.

Supplying models to workloads

I already had some models downloaded locally from my local tests:

DeepSeek-R1-Distill-Llama-8B
DeepSeek-R1-Distill-Qwen-14B
DeepSeek-R1-Distill-Qwen-1.5B
DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1-Distill-Qwen-7B
Llama-2-13b-chat-hf
Llama-2-13b-hf
Llama-2-7b-chat-hf
Llama-3.1-8B-Instruct
Llama-3.2-3B-Instruct

In a local Docker environment, I would just mount a volume and point the model loader to it.

docker-compose.yaml

services:
 llm-serving:
 image: intelanalytics/ipex-llm-inference-cpp-xpu:latest
 container_name: ilx
 network_mode: host
 deploy:
 resources:
 limits:
 memory: 48G
 shm_size: 32G
 environment:
 - MODEL_PATH=/llm/models
 - USE_XETLA=OFF
 - SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
 - SYCL_CACHE_PERSISTENT=1
 - ONEAPI_DEVICE_SELECTOR=level_zero:*
 volumes:
 - /data/llm/models:/llm/models
 - /data/llm/models/ollama:/root/.ollama/models
 devices:
 - "/dev/dri/renderD129:/dev/dri/renderD129"
 - "/dev/dri/renderD130:/dev/dri/renderD130"
 stdin_open: true
 tty: true
 webui:
 image: ghcr.io/open-webui/open-webui:main
 container_name: webui
 network_mode: host
 volumes:
 - /data/llm/webui:/app/backend/data

In Kubernetes, I could always mount a host path to a volume and call it a day. However, this is a dirty, wasteful, and unsustainable solution, not at all applicable to workloads at scale. I found KitOps which packages models as OCI images (ModelKits) and can push/pull them from any OCI registry (Harbor in my case). It’s not the only thing it does; be sure to explore it in their docs. In short, it helps if:

You have multiple models / datasets being developed in parallel.
Experimentation to production hand-offs are a pain: “but which dataset?”, “which model version?”, “what code was used?"
You care about audit/compliance or want tighter security.
You already use container registries, CI/CD pipelines, Kubernetes etc., so you can integrate without too much new infra.

The initContainer kitops-init pulls the ModelKit image, unpacks it to a volume, and vLLM will serve the model from there later. We’ll see that in action here. Now let’s see how to package our model:

Before creating a fully automated pipeline, I manually packaged a model with KitOps CLI:

kit login harbor.sonda.red.local
# If we didn't already download the model, we can import it from HuggingFace
kit import https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-14B
# Otherwise, init a kitfile in the model directory and you'll be prompted for metadata
cd DeepSeek-R1-Distill-Llama-14B
kit init .
cat Kitfile

manifestVersion: 1.0.0
package:
 name: ds-r1-qwen-14b
 description: DeepSeek-R1-Distill-Qwen-14B
 authors: [Kalin Daskalov]
model:
 path: model-00001-of-000004.safetensors
 license: MIT
 parts:
 - path: model-00002-of-000004.safetensors
 - path: model-00003-of-000004.safetensors
 - path: model-00004-of-000004.safetensors
 - path: config.json
 - path: generation_config.json
 - path: model.safetensors.index.json
 - path: tokenizer.json
 - path: tokenizer_config.json
code:
 - path: .
docs:
 - path: LICENSE
 description: License file
 - path: README.md
 description: Readme file

# Pack a modelkit with a specific kitfile and tag
kit pack . -t harbor.sonda.red.local/sonda-red/ds-r1-qwen-14b:0.0.1
kit push harbor.sonda.red.local/sonda-red/ds-r1-qwen-14b:0.0.1

That’s it. The model is now in Harbor and can be referenced from workloads. I repeated this for all models I wanted to use. And if I wanted to change something, I just updated the Kitfile and repack/push. You’d push only the diffs, not the whole blob again.

kit list

REPOSITORY TAG MAINTAINER NAME SIZE DIGEST
harbor.sonda.red.local/sonda-red/ds-r1-llama-8 0.0.1 Kalin Daskalov ds-r1-llama-8 29.9 GiB sha256:1aed2ade6511630754d79589b10475cb5b92e9f96cf241c182fc2c1c7f88ba7f
harbor.sonda.red.local/sonda-red/ds-r1-qwen-1.5b 0.0.1 Kalin Daskalov ds-r1-qwen-1.5b 6.6 GiB sha256:6ee43ee48bb1e72032a89dc0c876cf04c2438756e4495e1f9ab4ca498763bc36
harbor.sonda.red.local/sonda-red/ds-r1-qwen-14b 0.0.1 Kalin Daskalov ds-r1-qwen-14b 55.0 GiB sha256:7a75e2eaef06858408a4572af7cf31ee9464c39bd92781fe27afcf8de2ac3d96
harbor.sonda.red.local/sonda-red/ds-r1-qwen-32b 0.0.1 Kalin Daskalov ds-r1-qwen-32b 122.0 GiB sha256:0147c9694a513a4c38e7ec000a50aeb9cf16a59568a1d2e75f1201dd28b7e953
harbor.sonda.red.local/sonda-red/llama-3.1-8b 0.0.1 Kalin Daskalov llama-3.1-8b 59.8 GiB sha256:6dbbab498985a21f95728219a1dbbec7ff76e7190107badffc9ae8766ce71552

Challenges

Network considerations

In my case, the ingress controller load balancer service has a MetalLB VIP. If pods or hosts resolve harbor.sonda.red.local to the MetalLB VIP, traffic leaves the node and comes back. Big blobs transfer at very low speeds because my GPU node is far from the router and NUC master and worker node. I connect it to the same network via WiFi, but it’s still a bottleneck.
If you’re hosting your own object storage, registry, and GPU workloads, try to keep them on the same node or ensure nodes have a fast path between them.
Leveraging Cilium’s topology-aware routing helps keep traffic local.
Your ingress controller (NGINX in my case) may have body size limits and timeouts that choke big pushes/pulls.

metadata:
 annotations:
 nginx.ingress.kubernetes.io/proxy-body-size: "4g"
 nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
 nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

Runners executing pull/push pipelines also need to be positioned well to maximize throughput.
Of course, storage hardware and filesystem matter too. I use a fast NVMe SSD on the GPU node.

03 GPU Allocation

I thought I had it all figured out with the classic Intel device operator and GPU plugin that I described in my previous post, but I ran into issues when trying to run tensor parallel workloads.

When researching for an alternative, I found the rather new Kubernetes Dynamic Resource Allocation (DRA) API that acts similarly to dynamic volume provisioning, in which you use PersistentVolumeClaims to claim storage capacity from storage classes and request the claimed capacity in your Pods.

I decided to give DRA a try as it seemed to fix what I kept struggling with.

Aim

Make Intel GPUs discoverable and claimable by workloads.
Claim multiple GPUs for one workload.
Make multiple workloads share the same GPU claim.

Stack

k3s 1.33 / 1.34
Dynamic Resource Allocation (References as DRA below)
Intel resource drivers (gpu.intel.com)
containerd with CDI enabled

Actions I took

Enabled DRA and CDI in k3s and containerd.
Installed the Intel resource driver DaemonSet.
Created ResourceClass and ResourceClaimTemplate objects.
Requested claims from vLLM workloads.

What DRA is

DRA lets pods request vendor devices through first-class APIs instead of opaque resourceName counters. It introduces DeviceClass, ResourceClaim or ResourceClaimTemplate, and ResourceSlice. A DRA driver advertises devices and uses CDI to inject them into pods.

In regards to vanilla Kubernetes, DRA reached v1 and was enabled by default in version 1.34. However, I was running k3s v1.33.4+k3s1, which was the latest stable release at the time. A few issues I hit:

k3s 1.33.4+k3s1 did not have DRA enabled by default
Intel resource drivers templates are all v1beta1 and the API group was not served by default in k3s 1.33.4+k3s1
Playing around with feature gates and runtime config flags was needed to get DRA working, but mixing up API versions and feature gates caused confusion, because in 1.33 the API version is v1beta2.
Took the risk and upgraded to k3s 1.34.1-rc+k3s1 pre-release, which didn’t enable the feature gate by default either, but at least the API group was served by default and I could use v1 resources.

Feature gates in Kubernetes are a way to enable or disable experimental or optional features. In this case, the feature gate DynamicResourceAllocation needs to be enabled for DRA to work. In k3s, you can pass feature gates to the kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet components via the k3s config file.

/etc/rancher/k3s/config.yaml

kube-apiserver-arg:
 - feature-gates=DynamicResourceAllocation=true
kube-controller-manager-arg:
 - feature-gates=DynamicResourceAllocation=true
kube-scheduler-arg:
 - feature-gates=DynamicResourceAllocation=true
kubelet-arg:
 - feature-gates=DynamicResourceAllocation=true

To enable the resource.k8s.io/v1beta1 API group in k3s 1.33, you can use the --runtime-config flag for the kube-apiserver. This flag allows you to enable or disable specific API groups and versions. /etc/rancher/k3s/config.yaml

...
kube-apiserver-arg:
 - runtime-config=resource.k8s.io/v1beta1=true

Enable CDI in containerd

k3s uses containerd as its only built-in container runtime, running it as an integrated lightweight service with simplified defaults and patches, whereas vanilla Kubernetes leaves containerd (or another CRI runtime) as a separately installed and managed component. Configuring containerd in k3s is done via the k3s config file, but you need to create a custom config.toml file for containerd itself, especially on the node where the Intel resource driver DaemonSet will run (the GPU node). In contrast, the feature gate for DRA is set in the k3s config file on the master node.

Intel resource driver docs have a section in their docs describing on how to enable CDI in containerd. The difference in k3s is that instead of editing /etc/containerd/config.toml directly, you edit the config file used by k3s to generate containerd’s config and then restart k3s:

/var/lib/rancher/k3s/agent/etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri"]
 enable_cdi = true
 cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

Intel resource driver DaemonSet

From here on, installing the Intel resource driver is rather straightforward and their docs are clear. You have options to use NFD or not; I chose not to. The DaemonSet will run only on the GPU node with a nodeSelector. You can install the raw manifests or use Helm. I chose Helm and discarded all traces of the Intel device operator and GPU plugin.

Verification that the DaemonSet is running and devices are advertised:

kubectl api-resources --api-group=resource.k8s.io
NAME SHORTNAMES APIVERSION NAMESPACED KIND
deviceclasses resource.k8s.io/v1 false DeviceClass
resourceclaims resource.k8s.io/v1 true ResourceClaim
resourceclaimtemplates resource.k8s.io/v1 true ResourceClaimTemplate
resourceslices resource.k8s.io/v1 false ResourceSlice

kubectl get deviceclasses
NAME AGE
gpu.intel.com 6d20h

kubectl get resourceslices -A
NAME NODE DRIVER POOL AGE
sonda-core-gpu.intel.com-z2z9z sonda-core gpu.intel.com sonda-core 5d17h

From the output above, you can see that the gpu.intel.com DeviceClass is registered and the sonda-core node (my GPU node) has a ResourceSlice with Intel GPUs. Two of the resources are namespaced (ResourceClaim and ResourceClaimTemplate) because they are requested by pods, while DeviceClass and ResourceSlice are cluster-wide.

Since the workloads will be concentrated mostly in the vllm namespace, I created the ResourceClaimTemplates there (deployments/vllm/resource-claim-templates.yaml).

deployments/vllm/resource-claim-templates.yaml

# Dual GPU claim template for tensor parallel workloads (TP=2)
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: dual-gpu-claim
 namespace: vllm
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 allocationMode: ExactCount
 count: 2
 selectors:
 - cel:
 expression: 'device.attributes["gpu.intel.com"].model == "A770"'
 deviceClassName: gpu.intel.com
---
# Single GPU claim template for single GPU workloads (TP=1)
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: single-gpu-claim
 namespace: vllm
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 allocationMode: ExactCount
 count: 1
 selectors:
 - cel:
 expression: 'device.attributes["gpu.intel.com"].model == "A770"'
 deviceClassName: gpu.intel.com

Due to my CPU being the Intel i9-12900K with integrated Intel UHD 770 graphics, it also appears as an available GPU device. To avoid workloads being scheduled on the iGPU, I used a CEL expression to filter only discrete GPUs (A770 in my case). CEL is a simple expression language that can be used in Kubernetes for various purposes, including filtering devices based on their attributes.

Exploring your ResourceSlice with kubectl describe can help you identify the exact model names of your GPUs, as well as attributes like memory size, vendor, etc. You can then use these attributes in your CEL expressions to filter devices according to your requirements.

Challenges

If you’re still on the classic Intel device plugin (shares)

With the Intel device GPU plugin we had this particular setup:

# Number of containers that can share the same GPU device (default: 1)
# Higher values allow more workloads to share GPU resources
- "-shared-dev-num"
- "10"
# Enable '*_monitoring' resource that provides access to all Intel GPU devices on the node
# Useful for metrics collection with tools like XPU Manager or Prometheus
- "-enable-monitoring"
# GPU allocation policy for shared-dev-num > 1 (default: none)
# Options: balanced (spreads workloads), packed (fills one GPU first), none (first available)
- "-allocation-policy"
- "balanced"

More info on the flags: [https://intel.github.io/intel-device-plugins-for-kubernetes/cmd/gpu_plugin/README.html#modes-and-configuration-options]

With this setup, the cluster has two A770 GPUs and each GPU is split into 10 shares, so the total capacity is gpu.intel.com/i915: 20 and gpu.intel.com/i915_monitoring is 2 (one per physical GPU). When you create an example pod with 1 share requested:

apiVersion: v1
kind: Pod
metadata:
 name: gpu-test
spec:
 containers:
 - name: gpu-test
 image: intel/xe-llm-inference-cpp-xpu:latest
 resources:
 limits:
 gpu.intel.com/i915: 1

Everything works fine. You see the GPU from within the pod and your inference app works. However, if we change the limits to gpu.intel.com/i915: 2, things break.

Why?

The GPU plugin doesn’t guarantee two shares will land on two different physical GPUs. They can land on the same physical GPU and the inference app (vLLM in my case) will try to initialize two devices, but only one device file is mounted in the pod.

Official docs say that you should set --shared-dev-num=1 in the plugin if you want workloads to use the whole GPU:

Intel GPU-plugin supports a few different operation modes. Depending on the workloads the cluster is running, some modes make more sense than others. Below is a table that explains the differences between the modes and suggests workload types for each mode. Mode selection applies to the whole GPU plugin deployment, so it is a cluster wide decision.

Mode	Sharing	Intended workloads	Suitable for time critical workloads
shared-dev-num == 1	No, 1 container per GPU	Workloads using all GPU capacity, e.g. AI training	Yes
shared-dev-num > 1	Yes, >1 containers per GPU	(Batch) workloads using only part of GPU resources, e.g. inference, media transcode/analytics, or CPU bound GPU workloads	No

This way, you can request gpu.intel.com/i915: 2 and be sure that two physical GPUs will be allocated. However this scenario depletes the available GPU resources and now only one pod has access to them.

DRA solves this problem by allowing you to claim multiple physical GPUs for one workload and also share them between workloads. You can have two pods sharing the same GPU, but each pod can request two physical GPUs for tensor parallel workloads.

On a node with 2 GPUs, here’s what each approach can (and cannot) do:

Scenario	Intel Device Plugin	DRA (Dynamic Resource Allocation)
1. One pod → one GPU	✅ Works (`shared-dev-num=1` or >1)	✅ Use a single-GPU `ResourceClaim`
2. One pod → multiple GPUs	⚠️ Only reliable with `shared-dev-num=1`; with shares, `-allocation-policy` (`balanced/packed/none`) influences placement but doesn’t guarantee distinct GPUs	✅ Either attach two single-GPU claims or one claim with `count: 2`
3. Multiple pods → one GPU	✅ With `shared-dev-num > 1` (share-level, weak isolation)	✅ Reuse the same claim across pods (intentional sharing); separate claims = isolation
4. Multiple pods → multiple GPUs	⚠️ Spread depends on share placement; still opaque	✅ Explicit: per-pod claims (isolated) or shared claims (intentional sharing)

The plugin’s -allocation-policy only steers share placement; DRA gives claim-level guarantees.

Conclusion:

The Intel plugin forces you into a cluster-wide choice between exclusivity or sharing, while DRA lets you mix and match all four scenarios safely within the same cluster.

04 Application Layer

Where it all comes together.

This part of the stack is relatively straightforward as it basically provides the user-facing API and web UI. The challenges faced in the previous steps trickled down from here, as vLLM and OpenWebUI work as intended and all issues were tied to the underlying infrastructure layers.

Aim

Serve an OpenAI-compatible API.
Host a web UI to interact with the models.

Stack

vLLM for the OpenAI-compatible API
OpenWebUI for the web interface

Linked manifests:
- deployments/vllm
- deployments/openwebui

Actions I took

Deployed OpenWebUI with Helm, Postgres, and Redis.
Deployed vLLM with KitOps and DRA claims.

OpenWebUI

OpenWebUI is a web interface for LLMs that supports multiple backends, including vLLM. It has user management, chat history, and other features. Basically your own ChatGPT interface. I deployed it with Helm and used Postgres for sessions/history and Redis for WebSockets/cache.

vLLM with KitOps ModelKit and DRA claims

Why vLLM?

Not much to say here. Locally, I’ve tried several options like ollama, llama.cpp and vLLM. I got them all to work with Intel GPUs, but vLLM was configurable and had sane explanations for using multiple GPUs with tensor parallelism. It also has a built-in OpenAI-compatible API server, which is a big plus if you plan on developing services that use the OpenAI API.

Let’s dissect the key features of the deployment manifest. It’s been redacted for showcase purposes, but you can find the full file in the repo link below: ➡️ Exact file in repo: deployments/vllm/vllm-14b-tp2/vllm-14b-tp2.yaml

# ==============================================================================
# vLLM 14B Model Deployment - Blog Showcase Version
# ==============================================================================
# This deployment demonstrates four key Kubernetes patterns:
# 1. KitOps sidecar for model artifact management
# 2. Dynamic Resource Allocation for GPU claims 
# 3. Persistent Volume Claims for model storage
# 4. vLLM configuration for tensor parallel inference
# ==============================================================================

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-14b-tp2
 namespace: vllm
spec:
 replicas: 1
 selector:
 matchLabels: { app.kubernetes.io/name: vllm-14b-tp2 }
 template:
 metadata:
 labels: { app.kubernetes.io/name: vllm-14b-tp2 }
 spec:
 nodeSelector:
 kubernetes.io/hostname: sonda-core

 # =======================================================================
 # =======================================================================
 # 🎯 KEY FEATURE #1: Dynamic Resource Allocation for GPU Claims
 # =======================================================================
 # Uses Kubernetes 1.30+ DRA to claim multiple GPUs for tensor parallelism
 resourceClaims:
 - name: intel-gpu-resource
 resourceClaimTemplateName: dual-gpu-claim  # Claims 2x Intel GPUs

 # =======================================================================
 # 🎯 KEY FEATURE #2: Persistent Volume Claims for Model Storage
 # =======================================================================
 volumes:
 - name: modelkit
 persistentVolumeClaim:
 claimName: vllm-modelkits # Shared storage across vLLM instances (see PVC manifest: https://github.com/sonda-red/cluster-management/blob/main/deployments/vllm/pvc.yaml)
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }  # Shared memory for tensor parallel

 # =======================================================================
 # 🎯 KEY FEATURE #3: KitOps Sidecar for Model Artifact Management
 # =======================================================================
 initContainers:
 - name: kitops-init
 image: ghcr.io/kitops-ml/kitops-init:v1.7.0
 env:
 - name: MODELKIT_REF
 value: "harbor.harbor/sonda-red/ds-r1-qwen-14b:0.0.1" # Model artifact in registry
 - name: UNPACK_PATH
 value: /data/ds-r1-qwen-14b  # Where to extract model files
 - name: UNPACK_FILTER
 value: model  # Only extract model weights, not datasets
 - name: EXTRA_FLAGS
 value: "--tls-verify=false --plain-http -vv" # Registry connection flags
 volumeMounts:
 - { name: modelkit, mountPath: /data/ds-r1-qwen-14b }

 # =======================================================================
 # 🎯 KEY FEATURE #4: vLLM Configuration for Tensor Parallel Inference
 # =======================================================================
 containers:
 - name: vllm
 image: intelanalytics/ipex-llm-serving-xpu:0.8.3-b20
 command: ["/bin/bash","-lc"]
 args:
 - |
 python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
 --served-model-name ds-r1-qwen-14b \
 --port 8014 \
 --model /data/ds-r1-qwen-14b \
 --device xpu \
 --tensor-parallel-size 2 \ # Split across 2 GPUs
 --gpu-memory-utilization 0.85 \ # Use 85% of GPU memory
 --load-in-low-bit sym_int8 \ # 8-bit quantization
 --max-model-len 2048 \ # Context window
 --max-num-seqs 12 # Concurrent sequences

 # Essential environment variables for Intel XPU multi-GPU setup
 env:
 - { name: ZE_AFFINITY_MASK, value: "0,1" } # Use GPU 0 and 1
 - { name: ONEAPI_DEVICE_SELECTOR, value: "level_zero:gpu" }
 - { name: CCL_ATL_TRANSPORT, value: "ofi" } # Multi-GPU communication

 ports: [{ containerPort: 8014 }]

 volumeMounts:
 - { name: modelkit, mountPath: /data/ds-r1-qwen-14b }  # Model files from PVC
 - { name: dshm, mountPath: /dev/shm }  # Shared memory

 resources:
 requests: { cpu: "8", memory: "28Gi" }
 limits: { cpu: "16", memory: "36Gi" }
 claims:
 - name: intel-gpu-resource  # Claim the GPUs from DRA

 # Health checks optimized for LLM startup time
 startupProbe:
 tcpSocket: { port: 8014 }
 periodSeconds: 10
 failureThreshold: 120 # Allow 20 minutes for model loading

---
# ==============================================================================
# Service: Expose vLLM API with OpenAI-compatible endpoints
# ==============================================================================
apiVersion: v1
kind: Service
metadata:
 name: vllm-14b-tp2
 namespace: vllm
spec:
 selector: { app.kubernetes.io/name: vllm-14b-tp2 }
 ports: [{ name: http, port: 8014, targetPort: 8014 }]
 type: ClusterIP

Challenges

Graceful shutdowns

The most impactful issue I hit was related to vLLM’s handling of shutdown signals. When Kubernetes decides to terminate a pod (for example, during scaling down or rolling updates), it sends a SIGTERM signal to the main container process. vLLM needs to handle this signal gracefully to ensure that ongoing requests are completed and resources are freed properly.

In the future, I’ll need to explore this further and write a custom lifecycle handler if needed. For now, I set a long terminationGracePeriodSeconds to give vLLM enough time to shut down properly. I see this topic is still actively discussed in the vLLM community, so hopefully it will improve in future releases.

… The goal is that when vLLM shuts down - whether intentionally or due to an internal failure - the cause of shutdown should be logged with a useful level of detail, and the server’s resources (especially GPU memory) should be freed. …

05 Monitoring and Observability

In my LinkedIn promo about the previous post I shared a screenshot of how I used to monitor my two A770 GPUs with intel_gpu_top, docker stats and bashtop in terminal windows side by side. It was clunky and not a solution for Kubernetes.

Aim

See real GPU load and memory.
See inference throughput and latency next to logs.

Stack

Prometheus and Grafana with kube-prometheus-stack
xpumanager

Actions I took

Deployed kube-prometheus-stack with Helm.
Set up a monitoring DRA ResourceClaim for xpumanager that doesn’t make GPUs unavailable to workloads.
Deployed xpumanager as a DaemonSet with nodeSelector.
Configured service and ServiceMonitor for xpumanager and vLLM.
Created Grafana dashboards for the xpum_ metrics

Intel’s solution for monitoring Arc GPUs and Gaudi accelerators is called xpumanager. It exposes GPU metrics over HTTP in a Prometheus-compatible format. I deployed it as a DaemonSet, but it’s only on the GPU node. It’s ServiceMonitor scrapes the /metrics endpoint and Prometheus stores the data. I then built a Grafana dashboard to visualize GPU metrics alongside inference throughput and latency from vLLM logs (dashboard JSON).

XPU metrics Dashboard	vLLM metrics Dashboard

Challenges

xpumanager

I’m really trying to keep a neutral tone here, but the way xpumanager is handled in the repos is atrocious. When you go to the official repo of the Intel resource driver and follow the deployment instructions for deploying xpumanager, you hit a few issues:

The monitoring ResourceClaimTemplate uses v1beta1 in the latest release of the Intel resource drivers. So I need to use my own manifests instead of using the ones from the repo.
xpumanager manifest uses outdated images that need dependency fixes and prevent startup of the monitor service. The issue is still not closed and I’m using the workaround found in the issue comments.
The /metrics endpoint sometimes returns HTML instead of Prometheus text.

time=2025-08-31T19:02:04.889Z level=ERROR source=scrape.go:1631 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=intel-xpumanager target=http://intel-xpumanager.intel.svc:29999/metrics content_type="text/html; charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html; charset=utf-8\" and no fallback_scrape_protocol specified for target"

That error means Prometheus is hitting the xpumanager service but instead of getting plain text metrics it’s getting an HTML page.

Prometheus only accepts text/plain; version=0.0.4 (the Prometheus exposition format) or OpenMetrics (application/openmetrics-text). If the response is text/html, Prometheus assumes it’s a web page, not metrics. I needed to reconfigure Prometheus to handle this case.

The Grafana dashboard available in the xpumanager repo is not adapted to Arc GPUs and shows no data for most of its panels. I had to build my own dashboard from scratch (dashboard JSON).
Intel Arc A770 doesn’t expose all the metrics that xpumanager can collect. For example, temperature is not available on Arc GPUs, so any panel showing temp data will be empty.

Conclusion

This milestone has made the effort of building an Intel Arc GPU based LLM inference stack worthwhile. There were moments when I nearly gave up. I even put the cards up for sale out of frustration, but there were no buyers. That left me more or less forced to make them work. Every time I considered switching to NVIDIA though, the high prices and the reminder that this was meant to be a learning exercise, not a production grade investment, kept me back.

Now it feels like the fun is just beginning. The plumbing is largely in place: model packaging, GPU allocation, and observability are solved enough to let me shift my focus upward. I can finally spend more time at the application layer and start building useful services on top of the stack, instead of wrestling with infrastructure basics.

The timing also feels right. With Nvidia buying a $5 billion stake in Intel and the two companies announcing a strategic chip design partnership, my bet on Intel hardware looks a little less lonely. I am also keeping an eye on the upcoming Intel Arc Pro B60 series. If the single or dual chip models land well, I might add one to the mix for more GPU power.

The next step is to see what meaningful applications can be built on top of this foundation.

Lab Notes: Laying the Cluster Foundation

Mon, 11 Aug 2025 00:00:00 +0000

Link to Part 1: Assembling an Intel-Powered Lab

Disclaimer: This is not production guidance and it is not sponsored. It documents what actually ran in my homelab with Arc GPUs and k3s, so please double check before you roll it into your own setup.

Table of Contents
00 Philosophy: Get to Kubernetes fast and dirty, because I want to run some damned AI finally!
- Flux and GitOps
01 Day 0: k3s no bs
02 Software updates
03 Storage layout
04 Intel GPU support: the bare minimum
05 TLS: a local CA for now
06 Secrets: SOPS, encrypted in Git
07 Architecture diagram
08 Component overview
09 To-do and next steps

In the first part of this series, I walked through the hardware choices and the initial setup of my Intel-powered homelab. With the machines successfully doing their best to keep optimal temps this summer, it was time to document the first big step - building up the cluster.

Some personal notes follow below, if you want to skip them, just scroll down to the next section or up to the table of contents to get to wherever you want to go.

00 Philosophy: Get to Kubernetes fast and dirty, because I want to run some damned AI finally!

Not too dirty though…

In previous homelab builds I spent weeks on setting up complex home networks, PXE booting, HA experiments with external datastores, etc. An important lesson is that your project needs to have a priority and learning goal, otherwise you can get lost in making the most balanced, perfect, and future-proof setup. So forgive the ignored best practices of some aspects of this setup. The focus is of course running AI workloads on Intel Arc GPUs, For now, I avoided using an ephemeral cluster I bring up and destroy locally, like kind but that may change. So this is a mix of both worlds, not a perfect 3-master HA cluster with all the bells and whistles, but also not cluster running locally.

Flux and GitOps

I’ve used helmfile with a pipeline setup in Gitlab for my previous homelab and use ArgoCD at work. Flux is a bit different but I like it for its pure GitOps approach. I’m a bit tired of clicking, to be honest. I don’t want to click around in a UI, or follow pipelines along. I want to commit like a mad man and after a while have my changes propagated to the cluster. In theory, ArgoCD can do that too but there’s something about the way you write the argo manifests that doesn’t feel as clean as Flux. Don’t treat this as a valid technical argument, it’s just a personal preference. I also had a fun talk with a colleague once about the differences and this was his take:

I’ve been working in 3 companies in a row, first one was using helmfile, second - argo, and third - flux And in the first one we were just getting things done with k8s and having fun with other tasks, like we were working on an open source project, or trying complicated setups and different tools. But in the rest we were mostly trying to make GitOps work, re-writing manifests because of YAML typos, reviewing something like 700 lines of YAML code, merging, fixing, and trying to make GitOps work again.

Hopefully this cluster won’t grow to the extent that I have to deal with that :)

But yeah, the biggest time waster was following the logs of the Flux helm controller, trying to figure out why it didn’t apply the changes I made, redoing the changes, manually reconciling, etc. Dealing with all the different ways the helm charts of the tools organize their values is one of the reasons I’m becoming skeptical of helm recently. Also, using the Flux configMap and secret generators and integrating them with SOPS was a bit of a pain. However, when all the pieces fell into place and made a sound template, it was just a matter of replicating the same pattern for the rest of the cluster components.

01 Day 0: k3s no bs

I stripped k3s of the pieces I’d replace anyway — flannel, kube‑proxy, traefik. I love cilium because of its alien eBPF magic I don’t quite understand, the visibility and the node firewall rules it gives me so that’s the CNI, end of story. Everything was run manually with a set of simple scripts. This is one of those compromises I mentioned earlier.

Control Plane:

curl -sfL https://get.k3s.io | sh -s - server \
 --token {token} \
 --cluster-init \
 --flannel-backend=none \
 --disable-kube-proxy \
 --disable servicelb \
 --disable-network-policy \
 --disable traefik \
 --tls-san=sondas \
 --tls-san=sonda-01 \
 --tls-san=sonda-02 \
 --tls-san=sonda-core \
 --tls-san=127.0.0.1 \
 --kube-apiserver-arg=kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

Workers joined with:

curl -sfL https://get.k3s.io | sh -s - agent \
 --token {token} \
 --server https://sonda-01:6443

I kept both the NUC and the GPU box as workers and only later differentiated them with taints.

02 Software updates

Disclaimer: No automation regarding OS package installation or upgrading.

Host packages and kernel updates are just a manual dnf upgrade about once a month too - as well as OS upgrades, whenever a new Fedora Release comes out.

03 Storage layout

NUC nodes: single 512 GB disk given entirely to k3s.
GPU node: 2 TB NVMe for k3s data, 512 GB for the OS, and a separate 2 TB scratch disk for local experiments.

04 Intel GPU support: the bare minimum

On the GPU node only I enabled Intel’s oneAPI repo and installed the base bits so containers would see Level Zero and friends. The operator doesn’t install drivers; hosts must be ready beforehand.

# Enable Intel oneAPI repo and install base packages
sudo tee /etc/yum.repos.d/oneAPI.repo >/dev/null <<'EOF'
[oneAPI]
name=Intel oneAPI repository
baseurl=https://yum.repos.intel.com/oneapi
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
EOF
sudo dnf -y install intel-basekit intel-oneapi-runtime-libs

Installing the intel-basekit, intel-level-zero and the SYCL packages will result in a system that supports the Intel Arc GPUs and the Level Zero API.

This support is later extended in Kubernetes with the intel-device-operator and intel-device-plugins-gpu to expose the GPUs to the cluster, which you can follow in the next section at Intel GPU specifics on k8s level.

Sanity Test: To make sure everything is working at this stage, I usually go through the process of pulling an ipex image, which stands for intel-extension-for-pytorch.

You can run a simple sanity test to double confirm if the correct version is installed, and if the software stack can get correct hardware information onboard your system. The command should return PyTorch* and Intel® Extension for PyTorch* versions installed, as well as GPU card(s) information detected.

# Pull the image and run a simple test to check if the Intel Extension for PyTorch is working correctly
docker pull intel/intel-extension-for-pytorch:2.7.10-xpu
# Run the test in the container while giving it access to the GPU devices
docker run --rm -it --device /dev/dri intel/intel-extension-for-pytorch:2.7.10-xpu \
 python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"

[W817 18:49:52.781169416 OperatorEntry.cpp:154] Warning: Warning only once for all operators, other operators may also be overridden.
 Overriding a previously registered kernel for the same operator and the same dispatch key
 operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
 registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
 dispatch key: XPU
 previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:37
 new kernel: registered at /build/intel-pytorch-extension/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:186 (function operator())
2.7.0+xpu
2.7.10+xpu
[0]: _XpuDeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.32567+18', total_memory=15473MB, max_compute_units=512, gpu_eu_count=512, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=0, has_atomic64=1)
[1]: _XpuDeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.32567+18', total_memory=15473MB, max_compute_units=512, gpu_eu_count=512, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=0, has_atomic64=1)

Output should look like this, with the GPU name and properties printed. Congrats, you can be sure that the Intel Extension for PyTorch is working correctly and can access the GPU devices, otherwise you’ll receive something of the likes of XPU device not found or similar.

05 TLS: a local CA for now

I didn’t want public ACME in the lab. I minted a self‑signed root inside the cluster and used a CA ClusterIssuer for leaf certs. Trust is me installing the root CA on my devices, for the flex of having https on my ingress

Manifests were deployed at boostrapping the cluster: bootstrap/infra/cert-manager/kustomization/root-ca.yaml.

06 Secrets: SOPS, encrypted in Git

I keep secrets in the repo but encrypted with SOPS (age). Flux decrypts at apply time, so I get Git history without leaking credentials. Policy lives at the root: .sops.yaml. I encrypt per‑app secrets next to the app manifests. In a helm context, Flux uses its secret generator to decrypt the secrets and pass them to the helm release.

Keys are off‑repo. The cluster only knows the age private key needed by Flux to decrypt.

07 Architecture diagram

Below is the map I keep in my head. Central services (Postgres, Redis, MinIO) are shared by design; Harbor is just the first consumer but each later component requiring them will connect to them and not run its separate service. Ingress mainly fronts internal UIs — Harbor, MinIO Console, Grafana, and VictoriaLogs.

08 Component overview

Entry path for the reconciler: clusters/sonda-red/ Reusable bits for Helm releases: templates/flux/helm-release/

More or less self-explanatory infrastructure components:

Object storage: MinIO → infrastructure/minio/
Registry: Harbor (on Postgres + Redis) → infrastructure/harbor/
Datastores: Postgres → infrastructure/postgresql/, Redis → infrastructure/redis/
Ingress: ingress‑nginx → bootstrap/infra/ingress-nginx/
LB: MetalLB → infrastructure/metallb/
Metrics & dashboards: kube‑prometheus‑stack → infrastructure/monitoring/kube-prometheus-stack/
Logs: VictoriaLogs → infrastructure/monitoring/victorialogs/
CI runners: Actions Runner Controller → infrastructure/arc/

A note on observability:

My stack is intentionally simple. Full observability is a bit out of my scope, especially setting up alerting, etc. I have the basics down, as well as the default dashboards in Grafana, but I mainly stay in Lens to follow logs or deployment status while I’m working on the cluster.

Lens Workload Overview	Master Node Status View

Central services

I chose central Postgres, central Redis, and central MinIO as cluster services. Today only Harbor consumes them, but they’re for the whole cluster. That keeps state small in number of places and easy to back up. MinIO takes the largest chunk of storage from the 2 TB NVMe, as it’s intended to be the S3 backend for most apps and I’ll try to avoid persistent volumes when I can.

Harbor points to the shared services explicitly (values excerpt):

# harbor values
+ externalURL: https://harbor.sonda.red.local
+ database:
+ type: external
+ external:
+ host: postgres.postgres
+ username: harbor
+ coreDatabase: harbor
+ redis:
+ type: external
+ external:
+ addr: redis.redis:6379

Full manifests sit under infrastructure/harbor/.

k3s specifics

Upgrades: System Upgrade Controller → flux-system/system-upgrade-controller.yaml

k3s has a system-upgrade-controller, which is not built in but separately installed. It consists of an operator and a controller that watches for Plan resources and applies them to the nodes. The nodes are adequately drained, cordoned and upgraded. I use it to upgrade the k3s version from within the cluster, so I can just change the version in the Plan and it will apply it to all nodes, without having to run k3s scripts on the nodes manually.

An upgrade Plan looks like this:

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
 name: server-plan
 namespace: system-upgrade
spec:
 concurrency: 1
 cordon: true
 nodeSelector:
 matchExpressions:
 - key: node-role.kubernetes.io/control-plane
 operator: In
 values:
 - "true"
 serviceAccountName: system-upgrade
 upgrade:
 image: rancher/k3s-upgrade
 version: v1.33.3-k3s1
 tolerations:
 - key: node-role.kubernetes.io/master
 operator: Exists
 effect: NoSchedule
 - key: node.kubernetes.io/unschedulable
 operator: Exists
 effect: NoSchedule
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
 name: agent-plan
 namespace: system-upgrade
spec:
 concurrency: 1
 cordon: true
 nodeSelector:
 matchExpressions:
 - key: node-role.kubernetes.io/control-plane
 operator: DoesNotExist
 prepare:
 args:
 - prepare
 - server-plan
 image: rancher/k3s-upgrade
 serviceAccountName: system-upgrade
 upgrade:
 image: rancher/k3s-upgrade
 version: v1.33.3-k3s1
 tolerations:
 - key: dedicated
 operator: Equal
 value: gpu
 effect: PreferNoSchedule
 - key: node.kubernetes.io/unschedulable
 operator: Exists
 effect: NoSchedule

Intel GPU specifics

Node inventory: Node Feature Discovery → infrastructure/nfd/ It detects the presence of Intel GPUs and labels them accordingly. GPU plugin’s node selector is used to deploy plugin to nodes which have such a GPU label.
Intel Device Operator: Intel Device Operator → infrastructure/intel-device-operator/ Intel Device Plugins Operator is a Kubernetes custom controller whose goal is to serve the installation and lifecycle management of Intel device plugins for Kubernetes. It provides a single point of control for GPU, QAT, SGX, FPGA, DSA and DLB devices to cluster administrators.
Intel GPU plugin: Intel Device Plugin → infrastructure/intel-device-plugins-gpu/

As taken from the official docs: https://intel.github.io/intel-device-plugins-for-kubernetes/cmd/gpu_plugin/README.html

Intel GPU plugin facilitates Kubernetes workload offloading by providing access to discrete (including Intel® Data Center GPU Flex & Max Series) and integrated Intel GPU devices supported by the host kernel.

Use cases include, but are not limited to:

Media transcode

Media analytics

Cloud gaming

High performance computing

AI training and inference

Basically, it exposes the Intel GPUs to the cluster as resources, so you can request them in your pods. As soon as you install the plugin, the DaemonSet is created and the GPUs are advertised to the scheduler. A pod also spawns on each node that has the appropriate label, which prints:

I0817 13:01:11.792873 1 gpu_plugin.go:799] GPU device plugin started with none preferred allocation policy
I0817 13:01:11.793010 1 gpu_plugin.go:518] GPU (i915/xe) resource share count = 1
I0817 13:01:11.793909 1 gpu_plugin.go:540] GPU scan update: 0->2 'i915' resources found
I0817 13:01:11.793915 1 gpu_plugin.go:540] GPU scan update: 0->1 'i915_monitoring' resources found
I0817 13:01:12.794849 1 server.go:285] Start server for i915_monitoring at: /var/lib/kubelet/device-plugins/gpu.intel.com-i915_monitoring.sock
I0817 13:01:12.794851 1 server.go:285] Start server for i915 at: /var/lib/kubelet/device-plugins/gpu.intel.com-i915.sock
I0817 13:01:12.797391 1 server.go:303] Device plugin for i915 registered
I0817 13:01:12.797394 1 server.go:303] Device plugin for i915_monitoring registered
I0817 13:01:13.629689 1 gpu_plugin.go:92] Select nonePolicy for GPU device allocation
I0817 13:01:13.629703 1 gpu_plugin.go:138] Allocate deviceIds: ["card0-0"]

This log shows the Intel GPU plugin found the two Arc GPUs, registered them with the kubelet, and now pods can request them by adding resources.requests: { gpu.intel.com/i915: 1 }. The monitoring resource is also available if you deploy the sidecar/telemetry components.

Component updates

Wired Renovate so chart bumps show up as PRs I can merge (or ignore): renovate.json

09 To-do and next steps

Add bootstrap helmfile deployments to be managed by Flux
Research and add Flux dashboards as quality of life improvement
Begin building some containers with the arc runners and push them to Harbor
VPN or some kind of public endpoint for remote access to the cluster

Hopefully, this is where the fun begins. I want to experiment a bit with scheduling and running different kinds of workloads. No real solid plan for the next article but I have a few ideas:

Stable Diffusion
LLMs and different inference engines like vllm, llama.cpp, etc.
Explore model and prompt versioning
Observability of Intel GPUs and their utilization
Benchmarking and comparing different inference engines?
Loading bigger models on both GPUs

Lab Notes: Assembling an Intel-Powered Lab

Sat, 24 May 2025 00:00:00 +0000

Why Build This Homelab?

The Hardware Journey

In 2018, when my laptop died, I picked up an Intel NUC 8i3BEH. It was compact, efficient, and eventually got repurposed into a Docker/K8s node. My wife got one too. When the fan failed (twice), I replaced the chassis with a fanless Akasa case and turned it into a headless server.

Later, I added five Dell Wyse 3040 thin clients to build a Raspberry Pi-style HA cluster. The NUC did the heavy lifting, the Wyses ran the control plane. But performance was lacking, and eventually, when my wife’s NUC fan failed as well, I consolidated down to two NUCs: one control plane and one worker.

NUC 8i3BEH in Akasa case	NUC/Dell Wyse cluster	Double NUC setup

Why Build This Homelab?

My Linux journey started at 14 with a scratched OpenSUSE DVD and a broken GUI. I didn’t know who gave it to me or what version it was, but that terminal-only experience stuck. Homelabbing became my way of learning, and it’s the reason I landed my first DevOps job. What started as a hobby turned into a career and I still see great value in testing things out at home.

I want to explore AI from a DevOps perspective - how to run AI workloads, how to orchestrate them, and how to integrate them into a Kubernetes cluster.

Why Intel-Only?

The short answer: use what I own, and maximize price-to-VRAM. Intel Arc GPUs aren’t mainstream, but they offer solid specs for the cost if we consider the mantra that VRAM is King. Instead of chasing cloud credits or NVIDIA hardware, I chose to explore what Intel can do just because of the fun of it.

I expected it to be hard from my initial research online. Lack of documentation or best practices on how to deal with this hardware, not to mention the general warning signs from the community to just stay away. All this reminded me of my early Arch Linux installs, but it’s the intense troubleshooting that makes things click eventually on a bit of a deeper level than just copying steps from a guide. And in this new AI era, I’m honestly clueless about how to do things right. As reassurance, I told myself that a bit more tinkering will probably get me more familiar with the internals than just everything working out of the box.

At the time of writing, the new Arc Pro B60 with 24 GB of VRAM was announced at Computex 2025 along with a double 48 GB version, but they are not available yet. This gives me some optimism that Intel is serious about the AI market and will continue to improve their GPU drivers and software ecosystem.

Why Two Arc GPUs?

AI workloads are VRAM-hungry. At 16GB per card, Intel Arc’s wide 256-bit bus delivers 560GB/s bandwidth. Multi-GPU setups improve throughput, especially with larger models (7B–14B) but in the next posts I’ll shere there’s a really big “IT DEPENDS” in all of this. Regardless, it’s a cost-effective way to get started with AI workloads on a budget and running at least some of the popular models locally.

Arc doesn’t have the software maturity of NVIDIA. I’m not here to advocate—honestly (yet), it’s a bet that the ecosystem will mature, and it’s not certain the pain points will be solved. But I want to explore how well these chips work in a Kubernetes environment, how to set up multi-GPU workloads, and how to optimize them for performance, etc.

The AI Tidal Wave and Decision to Upgrade

The NUCs were great for learning Kubernetes, but when the world started buzzing about AI, I realized they were underpowered for the new workloads. I needed more VRAM and compute power to run LLMs and image generation models effectively.

Initially, I debated adding a third NUC but realized that by 2024, they were overpriced, nor did I need a third one for workloads exactly. I just had a dream of having 3 NUC masters for my cluster. I didn’t want to force this decision, so instead of spluring, I budgeted 250 EUR/month, hunted discounts, and built a powerful Intel workstation over six months for ~1500 EUR.

Full PC Part List

Highlights

CPU: Intel Core i9-12900K
The last generation of the i9 series with all the pros of the latest generations, without the controversial thermal issues. 16 cores (8P+8E), 24 threads.
GPU: 2× Intel Arc A770 16GB
ASRock Challenger OC, the most affordable 16GB models I could find. Not the blower type I wanted, but solid performance and adequate thermals.
RAM: 64GB DDR5
Storage: 4TB NVMe total
Motherboard: Asus ProArt Z790-CREATOR Wi-Fi
Two PCIe 5.0 x16 slots for dual GPUs, plenty of USB ports and M.2 slots. Both cards run at x8, which is sufficient for the Arc A770.
Power Supply: Corsair RM1000x Shift 1000W
Modular, with enough headroom for future upgrades.
Case: Fractal Design North
Excellent airflow, clean design, and matches my desk. Enough space for dual GPUs and keeps everything cool under load.

The front intake fans run at around 50% higher speed than the exhaust fans on the back and top, producing more static pressure to overcome the dust filter and creating a slight overpressure inside the case, which keeps dust intake at a minimum.

Inspired by this article: Fractal Design North Case with maxxed out air cooling.

Case internals	Workspace view

What I’m Exploring and What’s Next

For about 6 months, I’ve been running this Intel-only homelab as a local workstation. I’ve been getting used to which drivers on which kernel or distribution work best, and how to set up the GPUs for optimal performance. I’ve also been experimenting with running LLMs and image generation models locally, testing multi-GPU orchestration and containerization of these workloads.

I’m now ready to take it to the next level by integrating it into my Kubernetes cluster and exploring Intel’s GPU support in Kubernetes. In the next part of this series, I’ll document the architecture of the cluster itself, the reasons behind the choices I made, tech stack, etc.

Link to Part 2: Laying the Cluster Foundation

wrigo

Mon, 04 Sep 2023 00:00:00 +0000

When I first went into DevOps professionally, whenever an automation oppurtunity arose, my first instinct was to use bash for whatever script/tool needed to be built. That’s what I’ve been using for my personal needs and what I’m most familiar with.

I want to learn Go through use though and I’m actively trying to circumvent my instinct for bash, even for simple things. This post has metadata in the beginning you see as formatted title, date, description, tags, etc. They’re in the beginning of each <post>.md file like this for example:

+++
author = "System Scribe"
title = "wrigo"
date = "2023-09-04"
description = "helper tool for new hugo posts"
tags = ["go", "code", "bash",]
+++

I had a few thoughts regargind this:

Copy the last entry from each last post and edit it.
Create a template I fill each time.
Create a bash script to generate a new post file with filled metadata

I was set, catually on number 3, because writing in bash is always a fun experience but I reminded myself that the thing I want more was to exit a bit this comfort zone and try to use go for the job. So we got to:

Create Go program to generate a new post file with filled metadata with persistance of some fields

I knew it will be more complicated, more searching, more lines of code, etc. It actually took only a Sunday afternoon but was a good excercise. I’ve written two web scrapers about two months ago and felt rusty but things kept coming back. I reminded myself that maybe by consistently writing and progressively complicating such small innocent projects, I’ll learn the language better than going after a mastodon project and rewriting it everything I consider a new way of doing it better.

So it’s not much, definitely not a portfolio project for the future but a good start for this blog.

I’m following advice on how to structure my files and code by a friend but I’ve only processed only about 5% of what he’s told me for now.

You can check the code on my wrigo repo. Including the readme as well:

Wrigo

Wrigo (as in “Writer for Hugo”) is a simple tool to generate the metadata for your Hugo posts in the terminal.

Features

Generates the following metadata:
- Author
- Title
- Description
- Current Date
- Tags
Persists author and previous tag data in ~/.config/wrigo/config.yaml. This way in the next prompt, you don’t need to set author again and see last used tags.
Generates <lower-case-of-your-title>.md file in the current directory.
Makes starting a new post less of a hassle than using templates.

Get started

# Clone this repo
git clone gitlab.com/systemscribe/wrigo
cd wrigo

# Build a binary in your $PATH
go build -o ~/.local/bin/wrigo cmd/main.go

# Generate a new markdown file with metadata
wrigo

init

Mon, 17 Jul 2023 00:00:00 +0000

I’ve postponed the idea of a blog for quite a while. Didn’t really imagine myself writing entries on a computer, as I’ve mostly written inside real paper journals.

Life has however pushed me (willingly) in a more and more digital existance and it started feeling like a more natural thing to write about technology in a space like this, rather than on paper.

Using this entry as a test for this new hugo thing. Hopefully, the following writings are a bit more interesting.

Sonda Red – Sonda Red

Lab Notes: XPU Manager 2.0 finally lets me delete my custom image

Table of Contents

00 Finally, a boring XPU Manager install

01 What I used to do

02 The DRA monitor shape I was patching toward

03 What changed upstream

04 The new shape in my cluster

05 The metric name situation

06 What this removes from my life

Kleym: Intro to creating SPIFFE workload identity from inference intent

Table of Contents

00 Personal notes

Time, doubt, and AI help

Giving it a name

01 What am I doing here?

02 The problem I am testing

To get into a bit more detail

03 What Kleym does today

04 The two identity modes

PoolOnly: one logical model pool, multiple replicas

PerObjective: one pool, separate model objectives

05 Why not just write ClusterSPIFFEID YAML?

06 Reaching some limitations

07 What is ahead

Lab Notes: Why llm-d pushed me out of Ingress (and into agentgateway)

Table of Contents

00 I came back from vacation to a broken routing layer

01 Why Ingress couldn’t follow

02 The migration I didn’t plan for

03 When timeouts become product behavior

04 Where the stack landed

05 Three lessons from three migrations

References

Lab Notes: Intel AI Inference Platform MVP 2 with llm-d

Table of Contents

00 Scale a vLLM replica to 2 and see what happens

01 What made llm-d click for me

02 LLMs as CPUs, and what Kubernetes does not know yet

03 How llm-d slots into the Intel MVP

04 The three llm-d components in this cluster

04.1 modelservice: vLLM on Intel GPUs with DRA and ModelKit

04.2 InferencePool: an LLM aware backend instead of a Service

04.3 HTTPRoute: Gateway to InferencePool

04.4 Routing and DRA

05 Lighting the path

06 Conclusion: why llm-d matters to me

Lab Notes: Intel AI Inference Platform MVP

Table of Contents

00 Reflecting on a year of learning

01 The Stack at a Glance

02 Packaging and Storage

Aim

Stack

Actions I took

Supplying models to workloads

Challenges

Network considerations

03 GPU Allocation

Aim

Stack

Actions I took

What DRA is

Enable CDI in containerd

Intel resource driver DaemonSet

Challenges

If you’re still on the classic Intel device plugin (shares)

Conclusion:

04 Application Layer

Aim

Stack

Actions I took

OpenWebUI

vLLM with KitOps ModelKit and DRA claims

Challenges

Graceful shutdowns

05 Monitoring and Observability

Aim

Stack

Actions I took