Brij Chauhan

Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need In the world of large language models (LLMs), memory bandwidth and capacity are pivotal for efficient inference workloads. Discover how optimizing these elements can significantly boost per-user throughput and drive system-level efficiency. Key Takeaways: - Serving LLMs demands 100s of GB per server for model instances, highlighting the importance of memory capacity. - High memory bandwidth is crucial for maximizing throughput. - DRAM-based designs offer superior efficiency in terms of cost and power usage. These insights, drawn from a comprehensive study by NVIDIA Research, highlight the core performance bottlenecks and potential hardware advancements that can redefine LLM deployment strategies. Read the full article here: https://lnkd.in/gNhFm8Ki #AI #LLM #MemoryBandwidth #InferenceWorkloads

18

1 Comment

I am excited to welcome Sankara Venkateswaran to the Ayar Labs team as our new VP of Engineering. As we move into 2026, the industry is hitting a critical inflection point. The "copper wall" is a real-world bottleneck for hyperscale AI, and co-packaged optics has become an infrastructure necessity in future roadmaps. Sankara joins us at a pivotal moment as we move towards mass production. His deep expertise in  compute and networking acceleration silicon from his time at Intel, Qualcomm, and Atheros perfectly complements our leadership in co-packaged optics. He understands the discipline required to deliver production-grade silicon that integrates into the world's most complex packaging flows. With Sankara leading our engineering organization, we are doubling down on our commitment to: - Scale our TeraPHY™ optical engines for high-volume deployment. - Break the power and bandwidth barriers of traditional interconnects. - Deliver predictable, high-quality CPO solutions that meet the aggressive timelines of our partners for the delivery of hyperscale AI infrastructure. Please join me in welcoming Sankara to the team!

89

6 Comments

From "General-Purpose GPUs" to "Custom Silicon": A Hardware Inflection Point Indian Startups Shouldn’t Miss Broadcom recently announced it has received over $10 billion in orders for custom AI racks and accelerators. That’s not just a procurement milestone. This marks a decisive shift in AI infrastructure—from broad-spectrum GPU reliance to purpose-built silicon engineered for high-efficiency, scalable performance. This surge reflects growing demand from frontier AI developers like OpenAI and others building large-scale foundation models—who increasingly require specialized infrastructure that goes beyond general-purpose GPUs to meet the scale, efficiency, and latency needs of next-generation training and inference. General-purpose GPUs, while versatile, are increasingly being outpaced by custom silicon when it comes to inference-heavy workloads. ASICs and other specialized architectures offer better power efficiency, reduced latency, and tighter integration for predictable, high-volume operations. And Broadcom isn’t alone. I see, Google continues to evolve its Tensor Processing Units (TPUs). Amazon has deployed Inferentia and Trainium across its cloud stack. Microsoft is building its own Azure Maia and Cobalt chips. There are startups that are reimagining compute from the ground up—focusing on memory bandwidth and wafer-scale innovation. But my main point is, this shift opens a rare window for Indian startups. With the right partnerships and design focus, there’s a clear opportunity to: - Build domain-specific silicon for EV, Automotive, AgriTech, Space, wearables, audio, and other edge AI domains - Leverage fabless models and global foundries to bypass capital-heavy barriers - Create IP-led differentiation in smart devices. There’s no need to replicate legacy architectures. The real opportunity lies in out-specializing them—especially in segments they won’t prioritize. This isn’t just a hardware story. It’s a blueprint for deep-tech sovereignty. Let’s not just observe the AI infrastructure race. Let’s build lanes of our own. Thoughts? #AIHardware #CustomSilicon #IndianStartups #SemiconductorIndia #DeepTech #StrategicAutonomy #TechLeadership #FutureProofIndia

40

2 Comments

As part of the IndiaAI Mission, Avataar is building UAI (Unified AI Interface) - a foundational DPI layer with thin, durable rails for trust and interoperability, connecting models, data and compute into a cohesive, discoverable ecosystem. We are privileged to collaborate with Prof. Ramesh Raskar (MIT) and Project NANDA: Architecting the Internet of AI Agents in shaping a broader “Internet of AI Agents” vision - exploring how a discovery layer for India’s evolving digital public infrastructure can unlock openness, innovation and responsible scale together. Equally grateful to Aafreen Siddiqui Sherwani (United Nations) for anchoring the dialogue, and to Pradeep Gupta (NVIDIA), Tanvi Lall (EkStep Foundation), Vineet Singh (Digital Green) and Jibu Elias (Mozilla Foundation) - for a thoughtful, forward-looking exchange on what it truly takes to build open AI platforms for India. The real opportunity ahead is not just technical, it is architectural. How we design these rails now will shape how intelligence compounds across citizens, enterprises and governance systems for decades. What principles should guide the design of AI infrastructure at national scale? #IndiaAI #DigitalPublicInfrastructure #ViksitBharat

184

9 Comments

𝘚𝘵𝘰𝘱 𝘉𝘶𝘺𝘪𝘯𝘨 𝘛𝘳𝘢𝘪𝘯𝘪𝘯𝘨 𝘊𝘩𝘪𝘱𝘴 𝘧𝘰𝘳 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦 𝘞𝘰𝘳𝘬𝘭𝘰𝘢𝘥𝘴: 𝘛𝘩𝘦 𝘛𝘊𝘖 𝘙𝘦𝘢𝘭𝘪𝘵𝘺 𝘊𝘩𝘦𝘤𝘬 There is a dangerous conflation in the market right now: assuming the hardware that built the model is the right hardware to serve the model. From a TCO (Total Cost of Ownership) perspective, this is financial suicide at scale. 𝐓𝐡𝐞 𝐓𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐆𝐚𝐩: • 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 requires massive FLOPs, high precision (FP8/BF16, sometime higher), and massive scale-up bandwidth (NVLink, Ethernet). Utilization is high because you control the batch size. • 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 (𝐒𝐞𝐫𝐯𝐢𝐧𝐠) is latency-sensitive, memory-bound (KV cache), and bursty. Using a massive training GPU for batch-1 inference is like using a Ferrari to deliver Uber Eats (or Grab, Swiggy). You are paying for compute capacity you cannot utilize because you are waiting on memory IO. 𝐓𝐡𝐞 𝐄𝐜𝐨𝐧𝐨𝐦𝐢𝐜 𝐒𝐡𝐢𝐟𝐭: As we move from "Model Building" phase to "Model Deployment" phase, the metric changes. • Training Metric: 𝐓𝐢𝐦𝐞-𝐭𝐨-𝐓𝐫𝐚𝐢𝐧 (𝐒𝐩𝐞𝐞𝐝) • Serving Metric: 𝐓𝐨𝐤𝐞𝐧𝐬-𝐩𝐞𝐫-𝐒𝐞𝐜𝐨𝐧𝐝-𝐩𝐞𝐫-𝐃𝐨𝐥𝐥𝐚𝐫 (𝐘𝐢𝐞𝐥𝐝) 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐜 𝐎𝐮𝐭𝐥𝐨𝐨𝐤 𝐟𝐨𝐫 2026: We will see a bifurcation in the data center: 1. 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐂𝐥𝐮𝐬𝐭𝐞𝐫𝐬: Homogeneous, high-bandwidth, expensive (Nvidia/AMD). 2. 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐅𝐥𝐞𝐞𝐭𝐬: Heterogeneous, optimized for TCO. This is where we see: • 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧-𝐍𝐚𝐭𝐢𝐯𝐞 𝐇𝐚𝐫𝐝𝐰𝐚𝐫𝐞:Chips designed specifically for INT4/INT8 precision. • 𝐃𝐚𝐭𝐚𝐟𝐥𝐨𝐰-𝐬𝐭𝐲𝐥𝐞 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞𝐬: Deterministic dataflow for low latency. • 𝐂𝐏𝐔 + 𝐍𝐞𝐮𝐫𝐚𝐥 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐨𝐫 𝐇𝐲𝐛𝐫𝐢𝐝: For cost-effective, smaller model serving at the edge. 𝐋𝐞𝐚𝐝𝐞𝐫𝐬𝐡𝐢𝐩 𝐈𝐧𝐬𝐢𝐠𝐡𝐭: If you are building an AI strategy, decouple your training infrastructure from your inference roadmap. The architectures winning in 2024 (training) will not be the ROI winners of 2026 (inference). 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞 𝐟𝐨𝐫 𝐲𝐢𝐞𝐥𝐝, 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐩𝐞𝐚𝐤 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞. #AIInference #TechStrategy #TCO #Semiconductors #EdgeAI #AICompute #ExecutiveInsights #DeepTech

17

1 Comment

“What does NVidia’s effective acquisition tell us about the semiconductor industry? Data movement is the problem !” There is a lot to like about Groq’s LPU, but the most interesting thing in my opinion is Groq’s choice to use SRAM instead of DRAM. Why ? It turns out that the limiting factors - both in energy and speed - is moving data from CMOS to HBM. A majority of heat is generated through this data transfer. Even the SRAM is way bigger than DRAM and more expensive, it appears to be worth it because of the higher throughput and lower power consumption. It also turns out that packaging with HBM is very complex too and very difficult to test. So by using SRAM only, there is a good chance that we will see significant packaging reduction costs, even though SRAM is much expensive. Another note - HBM GPUs are still better for training (simply because there is lots of memory), but because adding more parameters to LLMs does not necessarily improve performance, its not really that necessary. What’s really going to make the difference in training in the future is synthetic data generation, which LPUs with SRAM appears to also have the advantage. My prediction is that future training clusters will have large percentage of compute move from doing SGD to synthetic data generation. I’m quite interested in the dataflow SDKs that LPUs are building too. (Note Cerebras has a similar architecture). I think this is going to be a major theme in 2026 and beyond. What does that mean ? I’m long on TSMC and not so bullish on Samsung and Hynix. Also, I think there is going to be a huge push in the future to develop RRAM / MRAM alternative to SRAM. This is going to be VERY interesting. If anyone know any MRAM groups that are building AI accelerators, please put me in touch ! Groq NVIDIA #LPU #GPU

25

2 Comments

A major vote of confidence in India’s deep-tech ecosystem. U.S. chipmaker Qualcomm has announced plans to invest up to $150 million in India’s AI and deep-tech startup landscape through a Strategic AI Venture Fund led by Qualcomm Ventures. The capital deployment will span early to growth-stage startups, with focused bets on: • AI-powered automotive technologies • Internet of Things (IoT) platforms • Robotics and Physical AI systems • Mobile and edge computing innovation This is not just a funding announcement—it reflects a long-term strategic commitment to India’s evolving innovation economy. As AI workloads move closer to the edge and silicon becomes central to competitive advantage, partnerships between global semiconductor leaders and Indian startups will accelerate commercialization and scale. For founders building in AI, semiconductors, embedded systems, and robotics, this signals expanded access to capital, global networks, and deep technical expertise. India’s AI infrastructure story is gaining serious momentum. #Qualcomm #AI #IndiaAI #Semiconductors #AIChips

4

1 Comment