Jelani Nelson

Love this direction. Designing rewards that are dense while keeping benchmarks hard to hack is the right way to push LLM evaluation forward. I also appreciate how the paper introduces example-specific rubrics — making LLM judging aligned with actual user expectations instead of vague, artificial criteria. Clear, grounded rubrics are exactly what the evaluation ecosystem needs. Highly recommend reading. 👇

3

New work on Rethinking Thinking Tokens: LLMs as Improvement Operators: https://lnkd.in/errhNuuz Reasoning training encourages LLMs to produce long chains of thought (CoT), improving accuracy via self-checking but increasing context length, compute cost, and latency. This work studies whether frontier models can achieve better trade-offs, higher accuracy with lower cost. The paper develops a simple yet effective Parallel-Distill-Refine (PDR) procedure: Generate diverse drafts in parallel, Distill them into a compact textual workspace, and Refine conditioned on this workspace. This decouples context length from total token count, allowing control over compute via parallelism. PDR yields higher accuracy than long CoT at lower latency. Training an 8B model with RL to align with PDR further shifts the Pareto frontier. On math benchmarks, PDR achieves +11% (AIME 2024) and +9% (AIME 2025) over single-pass baselines. With Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal.

78

1 Comment

I came across the Agentic Context Engineering paper by Stanford, SambaNova and UC Berkeley earlier this week and it reframes how we think about adaptation in language models. Instead of fine-tuning model weights, ACE evolves the context itself - transforming prompts and agent memories into living playbooks that self-improve over time. ACE introduces a modular generation, reflection and curation loop that enables LLMs to: · Prevent context collapse and overcome brevity bias · Achieve scalable self-improvement through grow-and-refine cycles · Learn effectively from natural execution feedback, without labeled supervision It's exciting to learn how ACE expands the agentic design space beyond autonomous reasoning, positioning context as a first-class learning substrate. As learning shifts from model-centric to context-centric paradigms, ACE paves the way for self-improving agentic architectures. Paper: https://lnkd.in/g4QXJcAb

21

3 Comments

🚨 According to the World Health Organization’s June 2024 report, over 3 million people die each year due to alcohol and drug use, with men making up the majority of these deaths. In our latest episode, we talk with Faiza Khan Khattak, PhD, CTO of a startup developing a personalized therapy platform for Alcohol Use Disorder (AUD). This conversation is a must-listen for anyone interested in mental health innovation, women-led tech, and how AI can address urgent health crises. 🎧 Tune in to learn more about this inspiring initiative! #WiAIR #WomenInTech #MentalHealth #MedTech #AlcoholUseDisorder #AUD #HealthTech #TechForGood

7

1 Comment

Israeli researchers at meta published a breakthrough work in LLM inference acceleration that slashes the number of forward passes required for LLM generation by an impressive 3-5x, while maintaining the exact same performance as traditional next-token prediction! They introduce Set Block Decoding, that accelerates generation by integrating standard next token prediction and masked token prediction within a single architecture, via a simple, flexible fine-tuning scheme. Set Block Decoding allows the model to sample multiple, not necessarily consecutive, future tokens in parallel, a key distinction from previous acceleration methods. This flexibility allows the use of advanced solvers from the discrete diffusion literature, offering significant speedups without sacrificing accuracy. SBD requires no architectural changes or extra training hyperparameters, maintains compatibility with exact KV-caching, and can be implemented by fine-tuning existing next token prediction models. https://lnkd.in/d7XmAyq7 #LLM, #LLMInference #Meta #Israel #GenAI

13

Stanford, SambaNova, and UC Berkeley just published ACE (Agentic Context Engineering)—a framework that improves LLM performance by evolving context rather than updating weights. The approach treats context as a living "playbook" maintained by three roles: Generator executes tasks, Reflector distills lessons from trajectories, and Curator merges delta items incrementally to avoid context collapse. https://lnkd.in/d-iwE9gH

3

Neural networks act locally: each layer transforms its input and moves on, without revisiting earlier computations. The only trace of the past lies in their parameters, not in memory. In game theory, Kuhn’s theorem shows that local probabilistic decisions can stay consistent with a global plan, but only when players have perfect recall. Neural networks face the same structural problem without that memory: each layer acts without full awareness of the computation’s history. And unlike games, they are not trees with single, traceable paths but complex graphs with loops and shared dependencies. This recent work suggests that quantum discord, a type of quantum correlation, might fill that gap — and even define a kind of Kuhn-equivalence for AI: a way to measure or enforce how well local computations stay aligned with a global objective through quantum correlation rather than recall. https://lnkd.in/gsHcWhXJ #QuantumDiscord #KuhnsTheorem #MachineLearning #ArtificialIntelligence #QuantumComputing

14

1 Comment

The recent Waymo testimony making the rounds isn’t really about robotaxis. It’s about the reality gap between AI demos and AI at scale in the real world. In Senate testimony earlier this month, Waymo confirmed that when its autonomous vehicles encounter ambiguous situations, they can request guidance from remote human agents. The vehicles remain in control, but humans are still in the loop. This shouldn’t be read as “AI failure.” It should be read as a reminder of how hard real-world deployment actually is. Every serious AI system that touches safety, money, or operations ends up with: • human fallback layers • escalation paths • edge-case handling • governance and oversight That’s not a bug. That’s operational maturity. The lesson for leaders isn’t “autonomy is fake.” It’s this: scaling AI safely is less about models, and more about systems. Real-world AI = model + data + workflow + humans + policy + monitoring. If you ignore any of those layers, you don’t get innovation. You get incidents. We’re entering the phase of AI where credibility won’t come from benchmarks or demos, but from disciplined deployment: where safety, transparency, and operational design matter as much as algorithms. The companies that understand that will scale. The ones that don’t will stall — or worse. #AILeadership #HumanCenteredAI #EnterpriseAI #AITransformation #AIatScale https://lnkd.in/eVVfiUPP

12

4 Comments