Lezhi Li

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

👉 My Key Takeaways from Chip Huyen's Recent Interview on Lenny's Podcast Chip Huyen is the author of the widely recognized "AI Engineering: Building Applications with Foundation Models". Link to the podcast: https://lnkd.in/dxZ-tFWX 💡 Importance of post-training.  Pre-training gives you raw capabilities (next token prediction on massive data), but post-training is what makes the model actually usable. SFT on high-quality examples + RLHF. Fine-tuning should be your last resort, not first. Most problems can be solved with better prompts, better data, or RAG. 💡 Evals.  You can't improve what you can't measure. Need multiple types: unit tests (does this specific prompt work?), integration tests (does the whole pipeline work?), regression tests (did we break something?), and user feedback loops. The hardest part isn't writing evals; it's maintaining them as your product evolves. 💡AI products.  Reliability and UX matter more than models. Most AI product failures aren't about bad models: they're about reliability (API limits, latency spikes, poor monitoring) and UX (users don't understand how to use it, doesn't fit workflow). Building reliable platforms and talking to users constantly beats chasing SOTA models. Most insights come from watching users, not from benchmarks. 💡How to improve AI-powered apps.  What people think improves apps: staying current on AI news, chasing newest agentic framework, obsessing over vector database choice, constantly evaluating model benchmarks, fine-tuning models. What actually improves apps: talking to users, building reliable platforms, preparing better data, optimizing end-to-end workflows, writing better prompts. Better prompt engineering beats switching models 90% of the time. A well-crafted system prompt, clear instructions, good examples (few-shot), and proper output formatting can transform a mediocre experience into a great one. 💡 Advice for builders.  Start with user problem, not with cool AI technique. Use the simplest solution that works (often that's a good prompt, not a fine-tuned model). Build evals early. Focus on end-to-end experience. Don't fine-tune unless you've exhausted everything else. Don't treat AI as deterministic (it's not, you need to handle variability). Don't ignore data quality (garbage in, garbage out).

1

1 Comment

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

Our team ByteBrain, in collaboration with Tsinghua University, proposed ChatTS, a time-series multimodal LLM, accepted to VLDB 2025. Here is a brief introduction: Understanding time series is crucial for its application in real-world scenarios. Recently, large language models (LLMs) have been increasingly applied to time series tasks, leveraging their strong language capabilities to enhance various applications. However, research on multimodal LLMs (MLLMs) for time series understanding and reasoning remains limited, primarily due to the scarcity of high-quality datasets that align time series with textual information. This paper introduces ChatTS, a novel MLLM designed for time series analysis. ChatTS treats time series as a modality, similar to how vision MLLMs process images, enabling it to perform both understanding and reasoning with time series. To address the scarcity of training data, we propose an attribute-based method for generating synthetic time series with detailed attribute descriptions. We further introduce Time Series Evol-Instruct, a novel approach that generates diverse time series Q&As, enhancing the model’s reasoning capabilities. To the best of our knowledge, ChatTS is the first MLLM that takes multivariate time series as input for understanding and reasoning, which is fine-tuned exclusively on synthetic datasets. We evaluate its performance using benchmark datasets with real-world data, including six alignment tasks and four reasoning tasks. Our results show that ChatTS significantly outperforms existing vision-based MLLMs (e.g., GPT-4o) and text/agent-based LLMs, achieving a 46.0% improvement in alignment tasks and a 25.8% improvement in reasoning tasks. We have open-sourced the source code, model checkpoint and datasets at https://lnkd.in/gYDJ9S4j Paper: https://lnkd.in/gqXW9tPY About ByteBrain Team: ByteBrain is a platform of AI for Infra (AI for System) at ByteDance. The goal is to leverage AI to automatically optimize system performance and reduce labor and resource costs. ByteBrain integrates machine learning, LLM and OR techniques to support both internal infrastructure systems at ByteDance as well as the Volcano Engine Cloud.

153

3 Comments

black-forest-labs/FLUX.2-klein-9B is crazy fun! Under FLUX Non-Commercial License, it bring: - Real-time AI agents with visual reasoning. - Instant, iterative design workflows. - Low-latency production pipelines on consumer-grade hardware.  - Sub-second speed - Text-to-Image + Native Editing in one pipeline. - Fits in ~20GB VRAM (RTX 4090 ready). The 4-step distilled version is a game-changer for anyone looking to deploy high-quality vision models without the massive VRAM overhead. #GenerativeAI #MachineLearning #FLUX2 #OpenSource #AIEngineering

6

3 Comments

🚨 breaking - ai apps & models launched in last 48 hours 1. hunyuan-a13b model open-sourced by tencent - compact yet powerful, moe architecture, 13b active, 80b total - hybrid model with flexibility to choose fast or slow thinking modes - 256k context window - enhanced agent task capabilities - efficient inference using gqa & multiple quantization methods 2. pangu models open-sourced by huawei - 7b & moe based pro 72b model - optimized for inference on huawei's ascend chips 3. ernie 4.5 models open-sourced by baidu - moe architecture with 10 variants - 0.3b dense model, 3b, 47b, 424b models - multimodal models 4. qwen-vlo by alibaba (already open-sourced) - upgrade from qwen2.5-vl - supports text to image & image to image generations - unique progressive generation feature where user can see steps of image generation

2

AI's Next Chapter: Key Trends from China's NetEase Future Conference: At the 2025 NetEase Future Conference, Academician Pan Yunhe outlined a pivotal AI shift: a "two-wheel drive" of thought simulation (e.g., LLMs) and action simulation (e.g., robotics). China's new "AI Plus" action plan focuses on deep industrial integration. Here are three critical trends he identified: 𝐓𝐫𝐞𝐧𝐝 𝟏: 𝐓𝐡𝐞 𝐑𝐢𝐬𝐞 𝐨𝐟 𝐒𝐩𝐞𝐜𝐢𝐚𝐥𝐢𝐳𝐞𝐝 𝐋𝐚𝐫𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 The "hallucination" problem becomes critical as AI Agents take action. • Solution: Use high-quality, "textbook-level" data from academia, industry, and real-world applications. • New Path: Proposes a dual approach: continue general models while also training specialized models first, then federating them into a general intelligence. 𝐓𝐫𝐞𝐧𝐝 𝟐: 𝐄𝐦𝐛𝐨𝐝𝐢𝐞𝐝 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐄𝐱𝐩𝐚𝐧𝐝𝐬 The concept is evolving beyond humanoid robots. • Broader Scope: Now includes drones, smart vehicles, and intelligent machinery. • Deeper Integration: Future systems will combine LLMs with visual, action, and mechanical models (e.g., sensors). Lightweight exoskeletons are a leading example. 𝐓𝐫𝐞𝐧𝐝 𝟑: 𝐀𝐈 𝐃𝐫𝐢𝐯𝐞𝐬 𝐭𝐡𝐞 "𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐄𝐜𝐨𝐧𝐨𝐦𝐲 2.0" The next wave moves beyond life-service platforms (e-commerce) to industrial-tech platforms. • The New Frontier: Platforms for scientific research and industrial design, like Group Core Technology's interior design platform that automates processes for a network of 50,000 enterprises. These trends highlight AI's maturation towards reliable, industrial-scale application. #ArtificialIntelligence #AI #ChinaTech #Innovation #LargeLanguageModels #EmbodiedAI #PlatformEconomy #IndustrialAI #QiGoalCapital #QiGoal

1