Aaron Batilo

"Beyond Indexes: How Open Table Formats Optimize Query Performance" is another profoundly great post on Open Table Formats and their evolution by Confluent's Jack Vanlightly. He walks you through the history of primary and secondary indices in OLTP databases, then contrasts that with the requirements for analytic workloads in Open Table Formats, showing that complex secondary indices don't help there. Instead, "in open table formats, layout is king. Here, performance comes from layout (partitioning, sorting, and compaction) which determines how efficiently engines can skip data, while modeling, such as star or snowflake schemas, make analytical queries more efficient for slicing and dicing." https://lnkd.in/gEdizbSY

5

Multi-node GPU training feels “fragile” because your model is basically running a distributed systems benchmark. Everyone blames “slow GPUs.” But at scale, the bottleneck is almost always the interconnect + tail latency. A single AllReduce is a global barrier: if one rank gets delayed (microburst, ECN/PFC pause, noisy neighbor, bad route, a single retransmit), everyone waits. RDMA helps—a lot—but it’s not magic. ✅ What RDMA actually fixes • Cuts CPU overhead and kernel involvement on the fast path (after setup) • Reduces copies / buffering (especially important for high message rates) • Lowers latency variance compared to “socket path” networking ❌ What RDMA does not fix • Congestion + queueing (microbursts will still stall collectives) • Packet loss / retransmits (RoCE + mis-tuned fabric = “mystery hangs”) • Topology mismatch (oversubscription, cross-rack hops, uneven paths) • Stragglers (one slow node turns into cluster-wide slowdown) • Control plane fragility (timeouts, retries, NCCL error handling, driver quirks) The production truth: RDMA is a necessary ingredient for high-scale training… …but the stability comes from end-to-end engineering: Engineer-proof checklist (what actually moves the needle) • Placement & topology: same rack/leaf when possible, avoid oversubscription, pin ranks with awareness • GPU-to-NIC path: ensure you’re not silently staging through host memory when you expected GPU-direct • RoCE/IB tuning: MTU, ECN, PFC pause behavior, congestion control (and verify counters, don’t assume) • NCCL config: timeouts, algorithm selection, channel count, correct interface binding • Observability: retransmits, ECN marks, PFC pauses, NIC queue depth, NCCL step time p99 (not average) • Resilience: frequent checkpointing, fail-fast + restart policies, “blast radius” limits for retries If you want multi-node training to stop feeling fragile, treat it like what it is: a latency-sensitive distributed system where p99 dominates your throughput. Curious: when your jobs “randomly slow down,” what’s the #1 culprit you see in practice—network congestion, stragglers, or NCCL timeouts? #RDMA #RoCE #InfiniBand #NCCL #DistributedTraining #AIInfrastructure #HPC #Networking #MLOps #CloudComputing

3

Curious what it takes to fine-tune a reasoning agent on a cluster? Read a new post in Lambda blog on using dstack to orchestrate distributed training on their 1-CLick-Clusters. The tutorial features the use of RAGEN, verl, and Ray. https://lnkd.in/eiqDjVYY

16

Anthropic shared a wild case study on Claude Opus 4.6. When the model hit a wall during a websearch test, it didn't just fail as it correctly hypothesized it was being evaluated, identified the specific benchmark, and then wrote its own code to find and decrypt the answer key 🤯 As I’ve written recently on my blog, the shift from "software that executes" to "agents that act" changes everything. This is a perfect example of why governance is an architectural problem, not a compliance one. When an agent is smart enough to "hack the test" to achieve its goal, traditional static gates and simple prompts aren't enough. We need a true platform mindset…one built for autonomous actors that can recognize the boundaries of their sandbox and actively look for a way out….If you’re still treating AI as a deterministic tool, it’s time to rethink your stack The full deep dive from Anthropic is worth a read: https://lnkd.in/gPXdffpy

91

5 Comments

Absolutely blown away by all the positive feedback on the #Hardwood 1.0.0.Alpha1 release 🥳! Thanks a lot to all the comments, discussions on issues in the tracker, pull requests, stars on the repo! So cool to see that this parser for #ApacheParquet, optimized for minimal dependencies and great performance, seems to itch a scratch for folks. I got invited to present Hardwood on the Parquet community call where we'll explore opportunities for collaboration, and I'll hop on a few podcasts to talk about the project, too. For the 1.0 release, the key items we're planning to implement initial support for predicate push-down, reading files from S3, and a compatibility layer with parquet-java. And, of course, further performance improvements: just today, Rion Williams opened a PR which substantially cuts down allocation rates, thus reducing GC pressure and improving overall latency. Onwards 🚀!

63

1 Comment

🚨 New Claude Code plugin: Lisa's Notes 🚨 I just released Lisa's Notes, a Claude Code plugin that automatically takes notes whenever Claude thinks it’s “done” by creating a Jujutsu (jj) commit. Because jj auto-stages everything, each commit becomes a clean, chronological record of what the agent actually did. No manual note taking. No lost context. Just a durable trail of progress. 👉 Plugin: https://lnkd.in/gt6YKWEe A few notes: • Works well on its own, as you can work through a number of steps and clean-up the code afterwards • Compliments the Ralph Wiggum loop nicely, as you can let Ralph do his thing and look at all the individual steps afterwards • Jujutsu makes it really easy to squash, split, re-order and remove commits afterwards • Works really well with my current AI native workflow: https://lnkd.in/gXy5-yjn

6