Anwar Shaikh

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

Rachit Jain's walkthrough shows how to profile kernels with PyTorch tools, import traces into NVIDIA Nsight or Intel Advisor, and read the slanted ceilings that separate compute-bound from memory-bound regions.

5

IBM just published their roadmap for bringing first-class PyTorch support to the Spyre Accelerator, fully ecosystem-first. Instead of building a custom fork, they’re extending torch.inductor with tile-based layouts, multi-core work division, and scratchpad optimizations so Spyre’s dataflow architecture plays nice with PyTorch’s compiler. Key pieces: • SuperDSC → KernelTile IR (KTIR) as a community-aligned spec for other dataflow accelerators • Full out-of-tree device registration with <5% overhead • vLLM integration with a new Spyre attention backend (no sequence-length constraints) • Layered testing + contributions back (OpenReg primitives, KTIR spec, CI patterns) This is the kind of thoughtful work that makes new hardware actually usable for real PyTorch teams, especially in healthcare AI where you need reliable, portable inference on diverse accelerators. Full technical note: https://lnkd.in/gnvg-r7F #PyTorch #AIInfrastructure #Hardware #MachineLearning

6

Engineering teams are turning AI into real results, not just experiments. Last month at the ELC Annual in SF, Redis Principal Field Engineer Josh R. joined Vinay Perneti from Augment Code, Anand K Sankaran from Workday, and Forrest Thomas from Garner Health for a fireside chat on where AI is truly adding value. Hear how they drove adoption across their teams, improved workflows, and made AI deliver real results: https://lnkd.in/gDAjZsDF

23

DeepSeek AI Signals a New Direction in Efficient Model Design : Latest Paper Suggests a Rethink of Large-Scale Training A newly published paper on arXiv proposes structural changes to neural networks that could meaningfully improve training stability and cost efficiency and it feels like a familiar pattern. The paper introduces mHC, a technique designed to stabilize large-scale training with minimal additional compute overhead. Tested on 3B, 9B, and 27B parameter models, mHC delivered stronger benchmark performance, particularly on reasoning-heavy tasks. Notably, CEO Liang Wenfeng co-authored and personally uploaded the paper, reinforcing how hands-on DeepSeek leadership remains with core research. If this sounds familiar, it should. Similar research breadcrumbs appeared ahead of R1 and V3, both of which preceded major releases. Why this matters Last year’s “DeepSeek moment” showed how R1 approached frontier-level performance at a fraction of the cost. This paper suggests they’re still unlocking efficiency gains. Combined with greater access to advanced AI chips, these architectural improvements point to something bigger. Looking ahead to 2026 Between cost-efficient training methods and accelerating hardware access, Chinese AI releases are set to become even more competitive—not just cheaper, but smarter and more stable at scale. This doesn’t look like a one-off paper. It looks like a preview. Read the paper here : https://lnkd.in/gR222q9P #AIResearch #DeepSeek #NeuralNetworks #LLMs #AIInfrastructure #MachineLearning #AITrends

7

Are you still measuring your LLM inference throughput solely by FLOPS? If you are scaling production-grade LLMs like Llama-3-405B or running complex RAG systems, optimizing for raw compute might be the wrong strategy. In large-scale inference, we often see teams trying to run these models on standard H100 clusters—only to run into a memory wall. To fit model weights and massive KV cache footprint, you are often forced to rely on aggressive tensor parallelism, sharding the model across 8 to 16 GPUs. Essentially, you are paying for the compute of 16 units just to acquire the HBM Capacity of 8. 📉 The technical reality: ⚠️ Drastically reduced Model FLOPS Utilization (MFU). ⚠️ High Time-to-First-Token (TTFT) due to interconnect bottlenecks. ⚠️ Skyrocketing inference unit economics. To escape this trap, the architecture must shift from pure compute to memory density. Architectures with significantly higher HBM capacity, such as 141GB-class HBM3e designs, enable this transition... 1️⃣ Reduced Complexity: Minimize sharding and orchestration overhead. 2️⃣ Lower Latency: Keep data on-chip and reduce network dependency. 3️⃣ Better TCO: Stop paying the "Memory Tax" on legacy infrastructure. As models continue to scale, the bottleneck has undeniably shifted. In the current era of #AIInfrastructure, memory bandwidth is the new Compute. #LLM #HPC #HBM3e #NVIDIAH200 #Scitix #TensorParallelism #AIInfra

4