Tao Lin

Most existing #lean4 datasets contain only correct proofs. Models learn error correction with RL, that's expensive. With UW Math AI lab we release a dataset of 260k erroneous Lean proofs with - compiler feedback - reasoning trace - corrected proof Improvements in Error Correction: - Goedel 8B: 2x - Kimina 8B: 3x Paper: https://lnkd.in/gaybt4bd

117

2 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

** Big update on the roadmap for Meta's MTIA roadmap (our Meta-designed AI chips) ** We just shared today a big update on the MTIA (Meta Training & Inference Accelerator) roadmap. The main takeaway is we're accelerating the whole set of programs to get rapid iterations on chips (given how fast the model needs are evolving). 👉 Overall blog is here: https://lnkd.in/gaHuxUfg For the networking aficionados, you can see a whole lot of networking that we've been working on inside MTIA - we're excited to finally be able talk about it! 👉 The scale-up and scale-out networks we've already talked about. It's sort of buried in there, but we are already running distributed training clusters with MTIA 300 for core ranking & recommendation (even as we focus future generations on Inference). 👉 The communication library (HCCL) that is purpose-built to take advantage of networking-specific capabilities we've built into the chips to offload the main compute. 👉 The in-built support for fused compute/communication kernels to minimize latency (key for both training and inference workloads). Super-proud of the whole MTIA team!

143

4 Comments