🍔🧠 How Google's Policy Bug Took Down The Whole Internet

Happy Monday! ☀️

Welcome to the 322 new hungry minds who have joined us since last Monday!

If you aren't subscribed yet, join smart, curious, and hungry folks by subscribing here.

📚 Software Engineering Articles

• Build and deploy agents that scope issues, code, and review PRs

• Master ACID database principles with this clear explanation

• Learn why concurrency differs from parallelism in system design

• Build robust systems with this distributed rate limiter design

• Critical insights from data-intensive applications book

• Discover powerful features of AI agents in modern applications

🗞️ Tech and AI Trends

• WhatsApp introduces ads for the first time

• AWS challenges Nvidia with custom AI chips

• Walmart and Amazon enter crypto with stablecoin plans

👨🏻‍💻 Coding Tip

• Use sed with backup files to safely replace lines after pattern matches

Time-to-digest: 5 minutes

Big thanks to our partners for keeping this newsletter free.

If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you love. 💚

One human. Dozens of AI agents. Infinite possibilities.

Build and deploy agents that scope issues, code, and review PRs alongside teams building products in Linear.

Linear is inviting engineers to build agents on their API.

How a Simple Bug Took Down Google Cloud: The 2025 Global Outage 🐛 

A null pointer exception in Google Cloud's Service Control system cascaded into a massive global outage, affecting millions of users and hundreds of services. What started as a minor code change in quota policy checks ended up bringing down everything from Gmail to Spotify, showcasing how interconnected our modern cloud infrastructure really is.

The challenge: Prevent a single point of failure in a critical service from cascading across globally distributed infrastructure while maintaining rapid policy updates across regions.

Implementation highlights:

• Service Control gateway: Acts as the central authorization and quota enforcement layer for all GCP API requests

• Global policy replication: Uses Spanner to instantly sync policy updates across all regions

• Kill switch mechanism: Implemented emergency "red button" to disable problematic code paths

• Regional independence: Designed for isolated regional operations but failed due to shared policy data

• Recovery orchestration: Required careful throttling and load redistribution to prevent secondary failures

Results and learnings:

• Widespread impact: 50+ Google Cloud services affected across 40+ regions

• Extended downtime: Full recovery took 2+ hours, with some regions taking longer

• Communication failure: The status dashboard itself went down, leaving customers in the dark

The incident shows why feature flags, proper error handling, and isolated observability systems are crucial in cloud infrastructure. Even tech giants can fall victim to the "it works in testing" trap.

Remember, folks: always check for nulls, or your code might pull a Google and take half the internet down with it! 🎯

ESSENTIAL (wise-guy wisdom)
Expert Generalists

ESSENTIAL (AI bodyguards)
How Can You Secure Your AI Agents?

GITHUB REPO (chat-no-internet)
Jan is an open source alternative to ChatGPT that runs 100% offline on your computer

ARTICLE (no-prop-tangles)
Use Async Local Storage to prevent props drilling in Next.js Route handlers

ARTICLE (JS oopsies)
Things to avoid in JavaScript

ARTICLE (vibey-craftsmanship)
The Case for Software Craftsmanship in the Era of Vibes

ARTICLE (env-var secrets)
What every dev should know about using Environment Variables

ARTICLE (go-go-design)
Modern application design

ARTICLE (AI sidekicks)
AI Agents

ARTICLE (SQL-over-AI)
This AI Agent Should Have Been a SQL Query

Want to reach 190,000+ engineers?

Let’s work together! Whether it’s your product, service, or event, we’d love to help you connect with this awesome community.

💰 Walmart and Amazon Explore Stablecoin Payments for Retail (2 min)

Brief: Retail giants Walmart and Amazon are reportedly exploring the use of stablecoins to facilitate payments, signaling a broader push into crypto-based transactions for everyday commerce.

📱 WhatsApp Finally Adds Ads to Status Updates After Years of Resistance (2 min)

Brief: Meta introduces ads in WhatsApp's Status feature, targeting users based on limited data while promising no ad interruptions in chats and no sharing of personal messages.

🤖 Google’s Gemini 2.5 AI Upgrades: Stable Pro & Budget Flash-Lite Now Live (2 min)

Brief: Google rolls out stable Gemini 2.5 Pro for developers and a cost-efficient Flash-Lite variant, slashing AI workload expenses while expanding integration into Google Search and AI tools.

🔥 Elon Musk’s xAI Burns $1B Monthly Amid Soaring AI Costs (2 min)

Brief: Musk’s xAI is spending $1 billion per month on AI development, far outpacing its revenue, highlighting the sky-high costs of cutting-edge artificial intelligence.

🚀 Sam Altman Reveals Meta's Failed $100M Offers to Poach OpenAI Talent (3 min)

Brief: OpenAI CEO Sam Altman claims Meta unsuccessfully attempted to lure top AI researchers with $100M+ compensation packages, boasting that OpenAI’s mission-driven culture kept its team intact.

⚡ AWS’ Custom Chips Challenge Nvidia’s AI Dominance (2 min)

Brief: Amazon’s Graviton4 CPU and Trainium2 GPUs are gaining traction in AI infrastructure, offering cost-effective alternatives to Nvidia’s chips, with Project Rainier already powering Anthropic’s Claude Opus 4 model.

This week’s coding challenge:

This week’s tip:

Use sed -i.bak '/pattern/!b;n;c\new text' to replace the line after a matching pattern while creating a backup. This advanced sed pattern works by matching a line, then using the n command to load the next line into pattern space for replacement.

Wen?

• Config file updates: Automatically update values following specific headers or markers.

• Code generation: Inject new implementations after function declarations or interface definitions.

• Log file processing: Replace dynamic content while preserving surrounding context and keeping backups.

“Become the kind of leader that people would follow voluntarily, even if you had no title or position.”
Brian Tracy

That’s it for today! ☀️

Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!

If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦

Thanks for spending part of your Monday morning with Hungry Minds.
See you in a week — Alex.

Icons by Icons8.

*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).