#119 Your Web App Scaling Tricks Don’t Work for LLMs

In this episode, we talk with Abdel Sghiouar and Mofi Rahman, Developer Advocates at Google and (guest) hosts of the Kubernetes Podcast from Google.
Together, we dive into one central question: can you truly run LLMs reliably and at scale on Kubernetes?

It quickly becomes clear that LLM workloads behave nothing like traditional web applications:

• GPUs are scarce, expensive, and difficult to schedule.
• Models are massive — some reaching 700GB — making load times, storage throughput, and caching critical.
• Containers become huge, making “build small containers” nearly impossible.
• Autoscaling on CPU or RAM doesn’t work; new signals like GPU cache pressure, queue depth, and model latency take over.
• LLMs don’t run in parallel, so batching and routing through the Inference Gateway API become essential.
• Device Management and Dynamic Resource Allocation (DRA) are forming the new foundation for GPU/TPU orchestration.
• Security shifts as rootless containers often no longer work with hardware accelerators.
• Guardrails (input/output filtering) become a built-in part of the inference path.

And then there’s the occasional request from customers who want deterministic LLM output —
 to which Mofi dryly responds:
 “You don’t need a model — you need a database.”

Powered by: ACC ICT

Stuur ons een bericht.

ACC ICT Specialist in IT-CONTINUÏTEIT
Bedrijfskritische applicaties én data veilig beschikbaar, onafhankelijk van derden, altijd en overal

Support the show

Like and subscribe! It helps out a lot.

You can also find us on:
De Nederlandse Kubernetes Podcast - YouTube
Nederlandse Kubernetes Podcast (@k8spodcast.nl) | TikTok
De Nederlandse Kubernetes Podcast

Where can you meet us:
Events

This Podcast is powered by:
ACC ICT - IT-Continuïteit voor Bedrijfskritische Applicaties | ACC ICT