Ming J.

🎉 New paper from #NVIDIAResearch 🧩 Puzzle: Distillation-Based NAS for Inference-Optimized LLMs+ Puzzle is a distillation-based NAS approach, that transforms trained LLMs into far more efficient, inference-aware (heterogeneous) architectures.

1

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

“Prune Gently, Taste Often?” Our ACL’25 work on LLM pruning is now live on Amazon Science as a blog post. Pruning is still “half-baked.” Delivering in-range accuracy while meeting compute budgets remains a puzzle. Researchers from Amazon (Aram Galstyan Sravan Bodapati (hiring) Jack FitzGerald Abhishek Kumar Bhavana Ganesh Goeric Huybrechts Markus Müller Jonas M. Kübler Rupak Vignesh Swaminathan Nathan Susanj Athanasios Mouchtaris) and UCSB (Yifan Yang, Zheng Zhang) introduced a “Prune Gently, Taste Often” methodology to offer a new perspective on this challenge. The work stems from Yifan Yang's internship at Amazon Pittsburgh Inference Science Team - thanks Denis Filimonov for being the principal shepherd. Thanks to Kate Greene, Jenna Larson, and Larry Hardesty for keeping the bite while tightening the prose: I’ve always missed the New Yorker style of critics. #amazonscience #acl #pruning #efficiency #llm

21

5 Comments

Most existing #lean4 datasets contain only correct proofs. Models learn error correction with RL, that's expensive. With UW Math AI lab we release a dataset of 260k erroneous Lean proofs with - compiler feedback - reasoning trace - corrected proof Improvements in Error Correction: - Goedel 8B: 2x - Kimina 8B: 3x Paper: https://lnkd.in/gaybt4bd

117

2 Comments

Caught my eye at #NeurIPS2025: a position paper calling for a "Refutations and Critiques" track for ML conferences. As an applied scientist, some of my most valuable lessons come from understanding why methods fail, not just celebrating what works. Critical examination isn't negativity, but it's how science advances. The poster asks: should we formalize space for challenging existing work? Would this strengthen our field or create perverse incentives? Curious what the community thinks. #MachineLearning #Research #AppliedML

72