• Hungry Minds
  • Posts
  • 🍔🧠 Palantir's Database Index Pattern That Cut Query Times By 85%

🍔🧠 Palantir's Database Index Pattern That Cut Query Times By 85%

PLUS: How Claude Code Works 🤖, Master Rate Limiting Patterns 🚦, OS Kernel From Scratch 🏗️

Author

Alex Zajac
September 25th, 2025 • Time-to-digest: 10 minutes

Today’s issue of Hungry Minds is brought to you by:

Happy Monday! ☀️

Well, no, it’s Thursday! I’m testing getting content out twice a week.

Would you like to receive content bi-weekly?

If you aren't subscribed yet, join smart, curious, and hungry folks by subscribing here.

📚 Software Engineering Articles

🗞️ Tech and AI Trends

👨🏻‍💻 Coding Tip

  • Use objects instead of switch statements for cleaner, maintainable code

Time-to-digest: 5 minutes

Big thanks to our partners for keeping this newsletter free.

If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you love. 💚

Aiven built a disk-less Kafka with 2 KIP:

↳ ~90% lower cloud data transfer fees by eliminating cross-AZ disk replication
↳ Zero-copy migration between classic and diskless topics
↳ Production-ready now with transactions and queues support

Palantir operates 300+ Elasticsearch clusters powering mission-critical workflows from geospatial defense operations to batch job orchestration. When upstream services use poorly-configured refresh policies, they can completely lock up entire ES nodes, bringing down essential infrastructure.

The challenge: Build defensive mechanisms into open-source Elasticsearch without forking the codebase, while maintaining API compatibility for existing clients across a complex microservices architecture.

Implementation highlights:

  • TransportInterceptor plugin: Hook into ES request pipeline to rewrite dangerous refresh policies before they execute

  • Concurrent sync-refresh detection: Track active immediate refreshes per shard and convert subsequent ones to wait_for to prevent lock contention

  • Smart policy downgrading: Automatically convert wait_for requests to none when refresh intervals exceed safe thresholds (>1s or -1)

  • Telemetry-driven alerts: Emit metrics on every policy rewrite to notify client teams about problematic usage patterns

  • Thread pool protection: Prevent write thread exhaustion by eliminating scenarios where all threads block on refresh locks

Results and learnings:

  • Zero thread pool exhaustion: Eliminated complete node lockups from concurrent immediate refresh requests

  • Bounded wait times: No more indefinite blocking on wait_for requests with misconfigured refresh intervals

  • Seamless deployment: Maintained full API compatibility while adding defensive behavior across 300+ production clusters

Palantir's approach proves that defensive infrastructure can be built into third-party systems without sacrificing functionality. Their plugin strategy shows how to harden critical dependencies while preserving upgrade paths and community collaboration.

Diskless 2.0: Unified, Zero-Copy Apache Kafka

We’ve unified Apache Kafka Tiered Storage (KIP-405) and Diskless Topics (KIP-1150) into a single path with zero-copy migration.

How Claude Code is built

A rare look into how the new, popular dev tool is built, and what it might mean for the future of software building with AI. Exclusive.

How Fine-Tuning Transforms Generic AI Models into Specialists

In this article, we will understand how fine-tuning works at a conceptual level, the different approaches available, and breakthrough techniques like LoRA that have democratized access to AI customization.

Trio vs Asyncio: The Ultimate Guide to Async Programming in Python

See how Trio makes Python async programming easier than asyncio with clean code, structured concurrency, and real-world examples you can use today.

Ace Your Next JavaScript Interview: Promises, Async/Await, Event Loop (Part 4) ✨ 

Learn the deeper concepts in JavaScript, such as Promises, Async/Await, and Event-Loop (5 min)

5 Rate Limiting Strategies Explained, Simply 🚦

#90: Break Into Rate Limiting (5 Minutes)

Less is safer: how Obsidian reduces the risk of supply chain attacks

Supply chain attacks are malicious updates that sneak into open source code used by many apps. Here’s how we design Obsidian to ensure that the app is a secure and private environment for your thoughts.

Leveraging CARL Method in Behavioral Interviews

How to crush your next behavioral interview

ARTICLE (code like an AI)
Shipping like Claude Code

ESSENTIAL (load balancer secrets)
What All Can A Load Balancer Do?

ARTICLE (duck beats spark)
DuckDB benchmarked against Spark

Want to reach 190,000+ engineers?

Let’s work together! Whether it’s your product, service, or event, we’d love to help you connect with this awesome community.

Brief: Google integrates its flagship Gemini AI model into the Chrome browser, enhancing search capabilities with AI-powered features for millions of users worldwide.

Brief: Apple achieves complete in-house control of iPhone chip production, marking a strategic shift to optimize AI capabilities and reduce dependency on external suppliers while enhancing device performance.

Brief: Tesla's Chief Designer suggests company is considering a Cyber SUV and smaller Cybertruck for international markets, with a potential mock-up design spotted in recent promotional video despite earlier statements about not using stainless steel exoskeleton.

Brief: Meta reveals new Ray-Ban display glasses for $799, upgraded smart glasses with better battery life, athlete-focused Oakley glasses, and major updates including real space VR scanning, an entertainment hub, and enhanced Horizon Engine for the metaverse.

Brief: Developer demonstrates how to serve images over DNS by leveraging TCP connections and TXT records up to 64KB, challenging common assumptions about DNS limitations and potentially raising new security considerations.

This week’s coding challenge:

Build Your Own Grep

Real-world proficiency projects designed for experienced engineers. Develop software craftsmanship by recreating popular devtools from scratch.

This week’s tip:

Use fetchpriority="high" on critical above-the-fold images to bypass browser heuristics and ensure immediate loading. This attribute overrides default browser prioritization which may delay important content.

Wen?

  • Hero images on landing pages: Ensures the main visual loads before other resources

  • Product photos in e-commerce: Prioritizes the primary product image over thumbnails

  • Critical UI elements: Avatar images or logos that define page structure

Take care of your body. It's the only place you have to live.
Jim Rohn

That’s it for today! ☀️

Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!

If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦

Thanks for spending part of your Monday morning with Hungry Minds.
See you in a week — Alex.

Icons by Icons8.

*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).