Hey {{first name | there}}. OpenAI, Anthropic, Google, NVIDIA, AMD, and nearly every infrastructure startup are optimizing the same problem: inference. Training may create the model, but inference is what customers actually pay for.
As serving billions of tokens becomes the bottleneck, inference engines are quietly becoming one of the most important pieces of modern infrastructure.
In today's Technical Notes:
What is an inference engine?
Where did they come from?
Why this matters
🧠TOGETHER WITH FLOX
Docker was supposed to end "it works on my machine." For local dev, it never quite did.
Spin up a dev container, and you're bind-mounting directories and port-forwarding just to reach your own tools. The isolation that makes containers great in production is the same thing fighting you on your laptop.
The fix everyone points to is Nix. I tried it. I gave up.
Then Flox gave me a way back in: all the power of Nix, without the PhD.
↳ Reproducible by design
↳ One definition, laptop to production
↳ 190,000+ packages from Nixpkgs
And it doesn't ask you to rip out Docker. Flox defines the environment, containers can ship it. Different jobs, no either-or.
Resolve AI builds with Flox, and simplicity is what they love most.
Still saying "it works on my machine"? Try Flox in 5 minutes.
Thank you @Flox for sponsoring this newsletter!
📰 TECHNICAL NOTES: So what is an inference engine?
An inference engine is the runtime between a trained model and the application using it. It loads the model, schedules requests, manages GPU memory, batches work across users, and streams generated tokens back to the client.
The model determines what answers can be generated. The inference engine determines how efficiently those answers are delivered.
That distinction matters. Two teams can deploy the same model on identical hardware and see dramatically different latency, throughput, and infrastructure costs depending on the inference engine underneath.
Where did they come from?
Although machine learning serving frameworks have existed for years, the current generation of inference engines is relatively new.
In March 2023, Georgi Gerganov released llama.cpp, demonstrating that Meta's LLaMA models could run efficiently on commodity CPUs. Local inference suddenly became practical instead of experimental.
A few months later, researchers at UC Berkeley's Sky Computing Lab introduced vLLM alongside the PagedAttention paper. Their work focused on GPU serving, showing that better memory management could dramatically increase throughput without changing the model itself.
Those projects established two directions that continue to define the ecosystem today. One focuses on efficient local and edge inference. The other focuses on maximizing GPU utilization for production-scale deployments.
How does an inference engine work?
Regardless of the implementation, the execution pipeline is broadly the same.
A user submits a prompt.
The engine first tokenizes the text into the integer IDs understood by the model.
Next comes the prefill stage. The entire prompt passes through the model once, producing the initial hidden state and building the KV cache. This cache stores the attention information for every token processed so far.
Generation then enters the decode loop.
Instead of recomputing the entire prompt for every new token, the engine reuses the KV cache and predicts one token at a time. Each generated token is appended to the sequence and immediately becomes part of the context for the next prediction.
As tokens are generated, they are streamed back to the client. That is why responses appear one token at a time instead of arriving all at once.
Most of the engineering effort in modern inference engines is concentrated in this loop. Better scheduling, batching, memory allocation, and cache management allow the same hardware to serve more concurrent users while reducing latency and infrastructure cost.
Why this matters
I think Inference engines are worth paying attention to if you are trying to understand how models work on the infrastructure side. For many months, I have been looking for how to get my feet wet with inference on Kubernetes, and I’m still on the fence.
Inference engines seem to be a worthwhile starting point.
🌍IN THE ECOSYSTEM
PagedAttention paper: The original 2023 paper introducing PagedAttention, the memory technique behind vLLM. Borrowing virtual memory paging from operating systems, it cut KV cache waste and unlocked far higher serving throughput.
vLLM documentation: The official architecture overview for vLLM, the widely adopted open source inference engine. Walks through the engine, scheduler, memory, and model execution. Start here to understand how it serves models in production.
SGLang blog : A code heavy deep dive into SGLang's internals, built for structured generation. Unpacks RadixAttention, zero overhead scheduling, and cache aware routing, contrasting each choice against vLLM. Long, advanced, for the deeply curious.
💡PRESNTED BY SPACELIFT
AI agents are changing who, or what, uses your Terraform modules.
Join Jinger Meilani, Senior DevOps Engineer at MNTN, to learn what changes when the consumer is not human and which interface design principles still hold.
Walk away with practical patterns for creating reusable, self-service building blocks that help non-infrastructure developers onboard faster and prepare your IaC for its next generation of users.
⏱️UNTIL NEXT TIME
The AI ecosystem is moving fast, but not every change deserves your attention.
Today’s issue is to separate the infrastructure worth understanding from the noise.
Know an engineer who needs to hear this? Share this link with them.
See you next week.
Jubril Oyetunji
CTO, EverythingDevOps
