Happy Monday! ☀️
Welcome to the 1169 new hungry minds who have joined us since last Monday!
If you aren't subscribed yet, join smart, curious, and hungry folks by subscribing here.
📚 Software Engineering Articles
🗞️ Tech and AI Trends
👨🏻💻 Coding Tip
Optimize LLM inference using vLLM's token-level batching for maximum throughput
Time-to-digest: 5 minutes
Serving a 100K-token prompt costs dramatically more than a short one—not because the model is larger, but because of the KV cache, working memory that stores key and value vectors for every input token. For a 70B-parameter model at 128K tokens, this alone consumes ~40GB of GPU memory, scaling linearly with context length and batch size.
The real cost comes from reading this cache, not storing it. During token generation (decoding), the model must sweep through the entire cache on every step to compute attention. This memory bandwidth bottleneck—not compute—drives the expense of long-context inference.
The challenge: KV cache memory and bandwidth scale linearly with context and batch size, making long-context serving prohibitively expensive at scale.
Implementation highlights:
Grouped-query attention reduces key-value head count by having multiple query heads share one KV head, cutting cache size by ~8x compared to standard multi-head attention with negligible quality loss.
Multi-head latent attention (DeepSeek) compresses keys and values into smaller latent representations before caching, achieving 70KB per token versus 192-328KB for grouped-query models, though serving overhead increases.
KV cache quantization drops stored values from 16 bits to 8 or 4 bits, halving cache size per reduction step; 8-bit quantization incurs <1% accuracy loss, while 4-bit shows measurable degradation on retrieval tasks.
Paged attention borrows OS memory paging: splitting the cache into small fixed-size blocks allocated on-demand instead of reserving one contiguous block per request, reducing fragmentation from 60-80% to <4% and boosting throughput 2-3x.
Prefix caching leverages paged attention to share cache blocks across requests with identical prefixes, achieving 50-90% cost/latency reduction on cache hits (common in agent workloads with repeated system prompts).
Results and learnings:
Architectural changes like grouped-query attention have become standard because they save memory and bandwidth with near-zero quality cost, reshaping how modern models are designed.
Serving optimizations like paged attention and prefix caching dramatically improve utilization without touching model internals, making them the easiest wins in production systems.
Quantization and eviction trade accuracy for savings; 8-bit quantization is safe while 4-bit and aggressive token-dropping introduce measurable quality drops on demanding tasks like multi-needle retrieval.
The KV cache turned a quadratic compute problem into a linear memory problem—then serving systems had to solve the bandwidth problem. Turns out the real bottleneck wasn't the size of the cache; it was how many times you read it.
ARTICLE (tests that understand themselves)
AI Can Write Tests Faster Than Your Team Can Understand Them
ARTICLE (speed at scale)
Presentation: Keeping ChatGPT Fast as AI Development Accelerates
ARTICLE (tpu-powered learning)
Run Ray on TPU, Part 2: Ray AI libraries
ARTICLE (agents multiplying chaos)
Zawinski's Law of MultiAgents
ARTICLE (hacking evaluation systems)
AI Escaped Its Sandbox — What Does That Actually Mean?
ARTICLE (stack-based minimalism)
Multistack Concatenative Programming Languages
ARTICLE (talks worth your time)
some software talks i like
ESSENTIAL (transformers demystified)
Before Q, K, and V: Reconstructing the Transformer
Want to reach 200,000+ engineers?
Let’s work together! Whether it’s your product, service, or event, we’d love to help you connect with this awesome community.
Brief: Model labs are co-designing harnesses with their models to optimize performance, forcing agent labs to move into model training or face obsolescence in their domains.
Brief: Google restructured its AI leadership with Demis Hassabis stepping into a new Chief Scientist role at Alphabet to focus on AGI strategy, while Koray Kavukcuoglu becomes SVP of Google DeepMind.
Brief: Jeff Dean and Sanjay Ghemawat, two of Google's longest-serving senior fellows, are leaving to start an AI startup, alongside key DeepMind and Google Brain researchers.
Brief: AMD acquired AI chip startup Taalas, which etches model weights directly into silicon to achieve 48x faster inference than Nvidia GPUs on select benchmarks.
Brief: Cloudflare launched Wallets to enable AI agents to autonomously purchase APIs and services using stablecoin micropayments through virtual accounts with spending limits.
Brief: As AI tools collapse the barrier between idea and working software, taste—the hard-won judgement to distinguish good from merely plausible—becomes the only irreplaceable skill.
This week’s tip:
Model Serving with Token-Level Streaming and Batching
Optimize inference throughput using continuous batching with dynamic token-level scheduling to maximize GPU utilization for LLMs.
Wen?
When latency from sequential request processing degrades throughput
When need to maximize GPU utilization during multi-user inference
When serving open-source or fine-tuned models at scale
Hope means hoping when everything seems hopeless.
Gilbert Chesterton
That’s it for today! ☀️
Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!
If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦
Thanks for spending part of your Monday morning with Hungry Minds.
See you in a week — Alex.
Icons by Icons8.
*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).