Joe Beda

There's been some talk around storage unification and zero-copy lakehouse integrations recently, and I wanted to better define these terms as well as look at ways we should evaluate different design decisions in this space. So I’ve published a new post: A Conceptual Model for Storage Unification Storage unification isn't about “one storage format to rule them all,” but virtualizing them into a unified logical model. In this post, I lay out a conceptual framework for thinking about storage unification (aka data virtualization). I explore: 🔹 Internal vs. shared tiering (and why shared tiering is so tricky). 🔹 When materialization makes more sense. 🔹 Client vs. server-side stitching. 🔹 Lifecycle and schema management across evolving formats. 🔹 Seven key considerations for evaluating different approaches. This is the groundwork for a set of posts where I’ll dig into how some real-world systems deliver storage unification, primarily about unifying real-time and historical data. https://lnkd.in/dBiS2kxJ

45

My good friend Kevin Yu is back with the third season of his podcast “Making of the SRE Omelette” — and it’s better than ever. Just like in a great kitchen, building reliable systems isn’t only about following recipes. It’s about balancing: - Methodology with intuition - Professional tools with hands-on experience - Precision with experience SREs aren’t just keeping systems up — they’re crafting reliability that users feel. That means scalable, resilient infrastructure paired with fast, seamless, and predictable interactions. If you care about the how and why behind resilient systems — and the human stories behind them — give it a listen. Kevin brings in voices from across the industry with insights, stories, and lessons you won’t want to miss. 🎧 https://lnkd.in/du7kSUZZ #SRE #Podcast  

28

1 Comment

Accidentally discovered a new way to work with GitHub Copilot today: write new tests based off test coverage reports. Copilot (or I) run this command to generate a report of all the uncovered lines in the current diff: pytest --cov --cov-report=xml && \ diff-cover coverage.xml --html-report coverage_report.html && \ open coverage_report.html Then Copilot reviews the report, summarizes the uncovered situations, and proposes new test cases to add, or if the code is truly unreachable, suggests deleting it. Just another reason why you should add diff-cover (or similar tool) to your projects!

115

4 Comments

New blog post: A Fork in the Road: Deciding Kafka's Diskless Future. Kafka is getting serious about S3 and finds itself at an architectural crossroads that will shape its next decade. Several new KIPs (1150, 1176, 1183) aim to reduce replication costs across cloud availability zones, but the implications go far beyond networking cost. It’s a mistake to think of S3 as simply a cheaper disk or a networking cheat. Building on object storage opens the door to operational benefits such as elastic, stateless compute — something many modern analytics systems already exploit. In my latest post, I outline two competing future paths: 🔹 Evolutionary path: Reuse large parts of existing Kafka components to reduce code changes and long-term maintenance. 🔹 Revolutionary path: Separate stateless and stateful layers to realize the full operational benefits of disaggregated storage. The post examines the trade-offs of each path, how the current KIPs map to these two paths and poses a broader question: what should Kafka become? And what will keep it relevant in the decade ahead? https://lnkd.in/d4E2BAe4

149

12 Comments

In 2025, one of the most underrated skills is learning JSON prompting. It changes how LLMs respond by giving them clear, structured instructions that read almost like machine code. Instead of writing broad, open-ended prompts, you break your request into simple key–value pairs—things like task, audience, or tone. Models like ChatGPT, Gemini, and Claude can then deliver sharper, more relevant results. This works because LLMs are trained on structured data—code, docs, APIs—so JSON feels natural to them. The result is output that’s cleaner and better aligned with what you want. Credits: heysehajsingh on X Follow @airesearches to stay updated on the world's most fascinating AI developments. ---------------------------------------------------- Learn AI in 3 Minutes a Day 👉 https://lnkd.in/eTNnQjaz ----------------------------------------------------

1

1 Comment

Absolutely blown away by all the positive feedback on the #Hardwood 1.0.0.Alpha1 release 🥳! Thanks a lot to all the comments, discussions on issues in the tracker, pull requests, stars on the repo! So cool to see that this parser for #ApacheParquet, optimized for minimal dependencies and great performance, seems to itch a scratch for folks. I got invited to present Hardwood on the Parquet community call where we'll explore opportunities for collaboration, and I'll hop on a few podcasts to talk about the project, too. For the 1.0 release, the key items we're planning to implement initial support for predicate push-down, reading files from S3, and a compatibility layer with parquet-java. And, of course, further performance improvements: just today, Rion Williams opened a PR which substantially cuts down allocation rates, thus reducing GC pressure and improving overall latency. Onwards 🚀!

63

1 Comment

POV: It’s 2026 and your manual infrastructure is a ticking time bomb. 💣 If you’re still running terraform apply from your local terminal, we need to talk. Production-grade IaC isn't just about "making it work"—it’s about making it unbreakable. The 2026 Playbook for Terraform Elites: 🔒 State Isolation is King: Local state is a myth. Use S3/GCS with DynamoDB locking. Separate your Dev/Prod states or prepare for a $100k "oopsie." 📦 Modular or Bust: Stop copy-pasting code. Build versioned modules. Pin everything. If a provider update can break your stack, your stack is already broken. 🛡️ Zero-Trust Secrets: Hardcoding passwords in .tfvars? Straight to jail. 🚩 Fetch them dynamically from HashiCorp Vault or AWS Secrets Manager. 🤖 Policy-as-Code (PaC): Don't just plan. Use Open Policy Agent (OPA) to auto-kill any PR that tries to open Port 22 to the world. Stop clicking buttons. Start scaling logic. 🚀 What’s your #1 Terraform "lesson learned" the hard way? Drop it below. 👇 #DevOps #Terraform #CloudNative #SRE #InfrastructureAsCode

8

What does a production-ready GenAI stack actually look like? We’re sharing real architectures, lessons from the field, and why a Gateway layer is now essential. 📅 May 15 | 10 AM PST 🎙️ @Nikunj Bajaj & @Abhishek Choudhary 🔗 Link in comments

16

2 Comments

LiteLLM is an open-source proxy that connects your app to LLMs through one unified interface. Paired with Redis, it gives AI and ML teams a simple yet powerful way to unify access to LLMs, accelerate response times, and make AI apps real-time. Redis handles performance, memory, and data coordination that modern AI apps demand, while LiteLLM handles abstraction and routing. Our AI Product Marketing Manager Rini Vasan dives into: ▶️ What LiteLLM is ▶️ Why it works well with Redis ▶️ How LiteLLM and Redis work together Learn how to scale your LLM gateway with LiteLLM & Redis: https://lnkd.in/gqsTebF7

32

As LLMs evolve from static models into dynamic components of real-world applications, they must do more than perform tasks; they must also adapt to context, personalize interactions, and continuously improve through feedback. This blog outlines a practical, modular approach to designing adaptive, production-ready LLM systems using: - Supervised Fine-Tuning (SFT) for core task competence - Reinforcement Fine-Tuning (RFT) for behavioral alignment - Context-aware routing for persona and adapter control - RAG for real-time knowledge injection - Telemetry loops for continuous learning and refinement It also includes a curated toolchain (LoRA, trl, vLLM, Langfuse, and more). https://lnkd.in/gBtjeg_X

17

2 Comments

🚨 New Claude Code plugin: Lisa's Notes 🚨 I just released Lisa's Notes, a Claude Code plugin that automatically takes notes whenever Claude thinks it’s “done” by creating a Jujutsu (jj) commit. Because jj auto-stages everything, each commit becomes a clean, chronological record of what the agent actually did. No manual note taking. No lost context. Just a durable trail of progress. 👉 Plugin: https://lnkd.in/gt6YKWEe A few notes: • Works well on its own, as you can work through a number of steps and clean-up the code afterwards • Compliments the Ralph Wiggum loop nicely, as you can let Ralph do his thing and look at all the individual steps afterwards • Jujutsu makes it really easy to squash, split, re-order and remove commits afterwards • Works really well with my current AI native workflow: https://lnkd.in/gXy5-yjn

6

If there’s any low level coders in my network please check out Troy’s GitHub and his project to build a new, faster shell! Some details on Troy: ### Windows Accessibility and TTS Contributions (Red Eagle Team) - Description: As part of the Red Eagle Team (a Georgia-backed initiative with Microsoft support in the mid-2000s), Mallory developed foundational code for universal Text-to-Speech (TTS) systems and cross-platform handicap accessibility features. This included compatibility layers for screen readers, app navigation, and voice input (e.g., influencing tools like Dragon NaturallySpeaking). - Impact on Windows: Microsoft adopted and evolved his logic starting with Windows Vista, integrating it into core OS features for impaired users. If you've used modern Windows TTS, clipboard integrations, or basic accessibility navigation, elements trace back to this work. - Modern Extensions: He's currently exploring IoT/smart home integrations for disabled veterans, building on this foundation. Troy’s Post “Okay guys think tank. I maybe over complicating this or may habe to do a bit of tweaking with some C code. Rubian has been going smoothly. The new boot and dynamic system discovery deamons work great. As standalone. The new stack needs tweaking for efficiency but works and is stable. Here's the stack im working on with the prototype test. /etc/rc.local ->calls boot.rb as bg process. ->boot.rb bootstraps rubian's system discovery and array logic. User login -> rubian3.rb (repl) -> chsh -s ./rubian3.rb USER Rubian boot.rb logic. ->starts C ruby and Jruby C ruby starts the bootstrap process while jruby warms up. -> Cruby passes workload to jruby when its ready to finish boot system. The deamon system is the server-side, where shells and env are the client. (See my UNIX socket prototype here) https://lnkd.in/gf6rWs46 C and j ruby can now natively talk to each other. I know that works. The issue im having is the new login shell isnt hooking into the deamon when it loads. https://lnkd.in/g5_2ZuPH A further explanation: “Yes. I've been building from the Shell working down to the kernel. The Shell side of things mimic Bash but in ruby expression. On the logic side of things, ruby treats everything as an object. What im doing is taking all the system information and building Arrays. Each directory and file then has an index that we can iterate through with .each_with_index, .each_with_object, etc. So instead of a static file tree, i have arrays of nested file paths and directories being pattern matched. Any ruby app will be ran and communicate to it. This allows for dynamic file utils systems and more. Each array index points to a file, directory, process, thread, interpreter, shell, hardware, everything that is a defined method object.”

3