Lina Li

Introducing Nemotron 3 Super, our new open 120B/12B hybrid Mamba‑Transformer MoE model built for real agentic reasoning at scale. 🧠 Latent MoE + multi‑token prediction + hybrid Mamba‑Transformer = up to 5x higher throughput and 1M‑token long‑context reasoning for multi‑agent systems. 🦞 All of this comes together to create a model that is well suited for long-running autonomous agents. Full parameter checkpoints for Nemotron 3 Super are available on Hugging Face and through NVIDIA NIM. 🔗https://bit.ly/4ukdMQl

2

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

3

🎉 New paper from #NVIDIAResearch 🧩 Puzzle: Distillation-Based NAS for Inference-Optimized LLMs+ Puzzle is a distillation-based NAS approach, that transforms trained LLMs into far more efficient, inference-aware (heterogeneous) architectures.

1

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

Breaking the memory bottleneck for streaming video understanding! I'm excited to share our solution in a new paper, a great collaboration between my exceptional student, Minsoo Kim, and Qualcomm AI Research. Imagine your AR glasses understanding hour-long videos in real-time, but here's the challenge: current multimodal LLMs require 100GB+ of memory for just 15 minutes of video—that's 3x more than most edge devices can handle! Our new research introduces InfiniPot-V, the first training-free framework that enforces a hard memory cap regardless of video length. The key breakthrough lies in exploiting temporal redundancy, where static backgrounds repeat across frames, and spatial importance, recognizing that not all image patches contribute equally to understanding. This enables us to compress KV caches by up to 94% while preserving accuracy. What makes this transformative is that it requires no costly retraining and operates query-agnostically, compressing video content before knowing what questions will be asked. Whether processing a 5-minute clip or a 5-hour stream, memory usage remains constant. This breakthrough could enable practical streaming video assistants on phones, AR glasses, and robots, closing the critical gap between research demos and real-world deployment. Please check out our paper for the full technical details: https://lnkd.in/gtcawhdg #EdgeAI #VideoUnderstanding #AR #VR #VLM #KVCacheCompression

92

3 Comments

🚧 #AIinference is reaching its memory ceiling, and GPUs aren't the barrier anymore — context is. Meet #NVIDIA BlueField-4, powering Inference Context Memory Storage, a new AI-native layer designed for long-context, multi-turn, #agenticAI at gigascale. By moving KV cache into a purpose-built, network-accelerated architecture powered by Spectrum-X #Ethernet, we deliver: 🚀 Up to 5× higher tokens per second 🔋 Up to 5× better power efficiency 🌐 Fast, RDMA-based context sharing across nodes The result is AI factories that scale inference efficiently, lower cost per token, and keep GPUs focused on what matters most: reasoning. 👉 See how we’re reinventing the storage stack for the agentic era: https://bit.ly/4c7pJCg #Networking #GPU #DataCenter

3