Panpan Xu

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

By building a taxonomy for compound AI systems, researchers are showing that when agents are optimized using structured evaluation frameworks, downstream task accuracy improves. Better evaluation → better agents. Featuring Melissa Pan and Mert Cemri of University of California, Berkeley / CSGE Berkeley Interviewed by Jessica Dai and SAIL Media Thank you to ‪Lambda‬ for making these interviews possible! 👉 Subscribe to our calendar for priority invites to upcoming events: lu.ma/twill #AI #ArtificialIntelligence #MultiAgentSystems #LLM #LLMAgents #CompoundAI #MachineLearning #AIResearch #BerkeleyAI #CSGEBerkeley #TechEvents #StartupCommunity #SiliconValley #FutureOfAI #AI #ArtificialIntelligence #MultiAgentSystems #LLM #LLMAgents #CompoundAI #MachineLearning #AIResearch #BerkeleyAI #CSGEBerkeley #TechEvents #StartupCommunity

34

2 Comments

New work on Rethinking Thinking Tokens: LLMs as Improvement Operators: https://lnkd.in/errhNuuz Reasoning training encourages LLMs to produce long chains of thought (CoT), improving accuracy via self-checking but increasing context length, compute cost, and latency. This work studies whether frontier models can achieve better trade-offs, higher accuracy with lower cost. The paper develops a simple yet effective Parallel-Distill-Refine (PDR) procedure: Generate diverse drafts in parallel, Distill them into a compact textual workspace, and Refine conditioned on this workspace. This decouples context length from total token count, allowing control over compute via parallelism. PDR yields higher accuracy than long CoT at lower latency. Training an 8B model with RL to align with PDR further shifts the Pareto frontier. On math benchmarks, PDR achieves +11% (AIME 2024) and +9% (AIME 2025) over single-pass baselines. With Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal.

78

1 Comment

Great article by Ashpreet B. on adaptive AI agents and continuous learning AI. Highly recommended reading for agentic engineering. https://lnkd.in/gZQknWrn

16

📢 New Blog Post: NeurIPS Datasets & Benchmarks Track - From Art to Science in AI Evaluations we just published a new blog post that provides an update on the 2025 Datasets and Benchmarks Track, reflecting on how the new hosting and metadata requirements affected submissions and the review process. We present submission statistics, survey findings from 851 authors and 155 reviewers, and identify areas requiring continued development. This differs from our earlier post on the review process by focusing on empirical outcomes rather than procedural changes. with Konstantina Palla , Francesco Locatello , Joaquin Vanschoren , Megan Risdal https://lnkd.in/eHxQTvnh

94

Love this direction. Designing rewards that are dense while keeping benchmarks hard to hack is the right way to push LLM evaluation forward. I also appreciate how the paper introduces example-specific rubrics — making LLM judging aligned with actual user expectations instead of vague, artificial criteria. Clear, grounded rubrics are exactly what the evaluation ecosystem needs. Highly recommend reading. 👇

3

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

967

21 Comments

Anthropic shared a wild case study on Claude Opus 4.6. When the model hit a wall during a websearch test, it didn't just fail as it correctly hypothesized it was being evaluated, identified the specific benchmark, and then wrote its own code to find and decrypt the answer key 🤯 As I’ve written recently on my blog, the shift from "software that executes" to "agents that act" changes everything. This is a perfect example of why governance is an architectural problem, not a compliance one. When an agent is smart enough to "hack the test" to achieve its goal, traditional static gates and simple prompts aren't enough. We need a true platform mindset…one built for autonomous actors that can recognize the boundaries of their sandbox and actively look for a way out….If you’re still treating AI as a deterministic tool, it’s time to rethink your stack The full deep dive from Anthropic is worth a read: https://lnkd.in/gPXdffpy

91

5 Comments

Anthropic's recent blogpost discusses the important shift from evaluating simple LLMs ---> testing complex AI agents.  It highlights why standard benchmarks fall short for agentic workflows and offers a practical roadmap for building robust, multi-turn evaluations." Packed with insights, but couple of takeaways that stood out to me:  🎯 1. 𝐂𝐨𝐦𝐩𝐨𝐮𝐧𝐝𝐢𝐧𝐠 𝐯𝐚𝐥𝐮𝐞 𝐨𝐟 𝐭𝐡𝐞𝐬𝐞 𝐬𝐲𝐬𝐭𝐞𝐦𝐬: where costs are visible upfront but their benefits accumulate over time. (quicker iterations, faster deployment) 🎯 𝟐. 𝐓𝐡𝐞 𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲 𝐅𝐢𝐱 / 𝐃𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐢𝐬𝐦 𝐢𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧𝐬: Regardless of agent type, agent behavior varies between runs. Sometimes, we want to measure how often (what proportion of the trials) an agent succeeds for a task, while for others, we want to measure if Agent succeeds atleast once. 🚀 Single success is enough across runs ? Use pass@k. 🚨 Need reliability across runs? Use pass^k. The post contains many more valuable insights worth exploring. Read more here: https://lnkd.in/gwCxqhDj

59

2 Comments

This new paper on Policy Compiler for Agentic Systems (PCAS) sits at the intersection of three ideas I've been writing about: policy-as-code, graphs, and the collapsing boundary between scaffolding and workflow engines. When frontier models self-enforce using policies embedded in prompts, compliance can be poor. With PCAS (Datalog rules enforced by a reference monitor intercepting every action), the authors show compliance is above 90% with zero violations. Caveat here is that the paper only tested prompt embedding, not safety reasoners or runtime policy retrieval. The 48% characterizes one design choice, not model-level enforcement as a category. IMO the interesting contribution is how PCAS enforces policy. Standard policy-as-code engines like OPA / Cedar evaluate individual requests against rules. That works for per-call authorization (e.g., is this agent allowed to invoke this tool with these arguments?). PCAS goes further by building a dependency graph (a DAG tracking causal relationships across agents, tool calls, and messages as the system executes) then evaluates Datalog rules over that graph with recursive queries. This unlocks policies that per-call engines cannot express like: ▪️ Transitive taint tracking: taint from untrusted sources propagates through the dependency graph. Actions transitively depending on tainted data that touch sensitive resources get blocked, even if the action looks clean in isolation. ▪️ Cross-agent provenance: agents accessing FDA data must obtain supervisor approval within their own causal context. Prior approvals from other sessions don't carry over. The graph enforces per-session authorization scope. ▪️ Prerequisite sequencing: policies requiring get_order_details before mutations are enforced structurally. The relevant graph state only exists after the prerequisite completes. That last example connects to my recent substack post on scaffolding meeting workflow (link in comments). The PCAS reference monitor is hard scaffolding: it sits outside the model, intercepts actions, enforces constraints the model cannot reason past. When it enforces step ordering and prerequisite checks, it absorbs a governance function that traditionally belonged to workflow engines. Scaffolding gaining enforcement-grade properties. But the dependency graph captures runtime execution provenance: what happened during this run. The organizational knowledge governance draws on, which approval chains apply, what roles are authorized, what reasons qualify, is encoded manually in Datalog rules. Someone had to write CoveredReason("medical_emergency"). In an enterprise, that knowledge lives in the organizational model workflow platforms maintain. PCAS shows scaffolding can absorb enforcement. But it also shows the organizational model that enforcement depends on has to come from somewhere outside scaffolding. Links in comments. #aisecurity #aigovernance #agenticai #trustworthyai #aievaluation

44

7 Comments

A new paper from CDS founding director Yann LeCun and Brown's Randall Balestriero introduces LeJEPA, a simpler way to train AI systems without labeled data. Instead of relying on many common training tricks, the method uses a clean mathematical idea to help models learn useful representations on their own. The approach scaled efficiently across many architectures and datasets, while still performing strongly on benchmarks like ImageNet. The work argues that self-supervised learning can be both simpler and more reliable when guided by clear theory. https://lnkd.in/g3zaZPrh #SelfSupervisedLearning #MachineLearning #AIResearch

690

5 Comments

RLHF allows us to refine a language model’s behavior based on feedback that captures what we consider to be good or useful outputs. Policy optimization and policy gradients form the foundation of RLHF. These techniques are especially relevant for aligning language models, as they are central to the training algorithms used in practice. Despite its importance, reinforcement learning remains less familiar to many of us in the AI field. Most of our training and practical experience tends to be grounded in supervised learning methods, where we learn from fixed input-output pairs. This gap in familiarity makes RL seem less approachable or harder to use in practice. In this blog post, I aim to close that gap by developing an understanding of policy gradient methods, which play a central role in RLHF. Full blog post: https://lnkd.in/gJfWEcif #GenAI #RLHF

1