Zhuoran Yu

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

69

2 Comments

Tom Aarsen recently came across two interesting Sentence Transformer models - one finetuned from ModernBERT-base, the other from Pinecone ConstBERT, which surprised me a bit since ConstBERT is typically used as a multi-vector model. Even more surprising? On a simple yet telling benchmark (TripletEvaluator: % of times sim(query, positive) > sim(query, negative)), the ConstBERT-based model slightly edged out the ModernBERT one: 🔹 ModernBERT: 98.8% 🔹 ConstBERT: 99.1% It's a small margin, but fascinating that a model designed for late interaction is holding its own—even excelling—in single-vector settings. Makes you wonder about architectural flexibility and how far you can stretch these models with the right fine-tuning. Curious if others have seen similar results or explored hybrid use cases this way—happy to chat! #NLP #InformationRetrieval #SentenceTransformers #BERT #IR #ML

101

6 Comments

OSS Bench for Agentic Coding.. in steps Cline-Bench: "A task that frontier LLMs struggle to complete are ideal candidates for cline-bench. If you find yourself having to intervene, or write code manually because cline was not able to complete the job - your task captures the failure boundary of state-of-the-art models and is precisely what cline-bench seeks to formalize." https://lnkd.in/gHcGYxiZ

6

Comparing models.. Finding that a small 1B #llm hallucinates less than a 4B for semantic to structural matching. Small llms for targeted tasks for the win. The new Meta llama3.2 1B is better than expected and cheap on token cost.

3