Ang Li

From Kernel to User Mode: The Ring 3 🦀Lets Run GTA 5 in JC-OS 😎 #RustLang #OsDev #SystemsProgramming #X86_64 #JCOS #TheRustEagle

2

The most impactful and important SDK release Trainiun Neuron team has worked on since inception: Native PyTorch support for Trainium, open source kernel (NKI) compiler and pre-optimized NKI kernel library, Neuron Explorer (IMO, the fastest and most intuitive ML profiler in industry), optimized inference implementation for major MoE and video models (GPT-OSS, Qwen, Pixtral….), complete vLLM v1 support !! All with full support for Trainium3. Using Trainium, whether for research, training , or large scale inference just became more intuitive and simple. https://lnkd.in/gjfhjv4j

626

10 Comments

Nvidia GPUS and Broadcom, Marvell ASICs Data & Insights for the Datacenter value chain … As I digested the Nvidia, Marvell, and Broadcom earnings and reviewed the technical differences in outlooks across GPUs and ASICs, a few lingering questions kept popping up: 1. Where is this market headed in 2030? 2. What are the associated opportunities in these hyper datacenters? So, I thought it was time to put the baby in action! The AI baby - the creation of these GPUs and ASICs. After watching Google I/O, the choice was simple - Google Gemini 2.5 Pro. The first challenge was to ask highly specific questions as I wrote in a previous Substack blog. While this is true for all models, it is specifically important (and powerful) for Gemini 2.5 Pro, due to its innovative step-by-step thinking model supported by industry-leading context window (2M tokens). A deep knowledge of the industry and function is a prerequisite. With my “a priori” questions and contextual hypotheses about the GPUs vis-a-vis ASICs, I wrote my prompt, with ~10 rounds of subsequent back-and-forth fine-tuning. Prompt: “Assume you are a deep subject matter expert across the semiconductors and data center infrastructure industry. Assume you focus on industry value chain-related data collection from reputed sources and curating/consolidating those as inputs for long-term product growth strategy and investment roadmap development. With this context and background, please answer these questions with sufficient detail, each one of them backed by projected 2025 market size and 2025-2030 market growth rate, as applicable, from reputed sources. 1. For the AI and HPC related computing chips, which of GPU or ASIC is projected to grow faster, and why? Break this data down across training and inference. 2. For the AI and HPC related networking chips, which architecture is projected to grow faster between Ethernet and InfiniBand? Break down the data for each of Ethernet and InfiniBand market size and growth a. across AI training and AI inferencing demands, and b. across pluggable optics and converged packet optical. What would be your technical rationale for what is projected to grow faster? 3. For the AI and HPC related demand, which of DRAM or NAND is projected to grow faster, and why? Break down the data for each of NAND and DRAM market size and growth across a. AI training and AI inferencing. and b. GPU vs. ASIC driven compute. 4. Using the answers to questions above as demand drivers, show the 2025 market sizes of the various components that go into building an HPC/AI datacenter, and how they are projected to grow in the next 5 years? Break down the sub-categories. 5. For different components in question 4 above, which ones are likely to remain highly constrained due to technology development complexity, to satisfy the technical requirements for AI?".. More questions, Gemini-generated public html report, and my strategic insights are in Substack here:  https://lnkd.in/gqMZGVnC

4

A couple of weeks back, I ran a project using Gemini to draw insights about AI's impact on demand trends for 1. GPU / ASIC and 2. Datacenters. The prompts I used are in my previous post: https://lnkd.in/gaD57YDR Here are my insights: 1. Compute - from a much lower base, ASIC is projected to grow faster than GPUs, driven by accelerating growth of Inference. 2. Networking - a similar pattern emerges as Inference is driving the growth. The Gemini predicted numbers look very conservative to me based on the latest earnings/forecasts. Despite multiple rounds of prompting, Gemini refused to increase this! As scale-out becomes more important in Inference (vs. scale-up in Training), Ethernet is accelerating growth relative to InfiniBand, a pattern we observed in recent earnings announcements across Nvidia and Broadcom. Multi-vendor ecosystem and open standards such as Ultra Ethernet Consortium are helping. 3. Optics - As expected, CPO is accelerating at 50%+ growth rate from a very low base. It is also likely undercounted as Gemini research did not identify any part of InfiniBand (Mellanox) adopting CPO. Based on announcements, we know that to be the most likely case. 4. Memory - As advanced packaging / hybrid integration remains highly in demand due to latency needs while still remaining significantly constrained in capacity, HBM4 is poised for explosive growth. 5. Cooling is hot! - Especially liquid cooling is expected to grow nearly at ~50% rate. 6. Energy - Power-hungry AI datacenters are expected to create a widening gap between demand and supply. Massive efforts are underway to close the supply gap through expensive efficiency gains, grid modernization, on-site generation, and of course renewables (as seen in many recent announcements). 7. GHG emissions - Using the Science Based Target Initiative (SBTI) benchmarks and 2025 as the base year, at the current pace, the target-actual gap widens to 150%+ in 2030! Carbon credits would not be the solution. We will need massive reductions through innovation in both energy demand (for product development) and carbon intensity of the grid (source, generation, and distribution) to have any realistic chance of keeping our planet cool! Mark Patel and team at McKinsey just published an excellent report on this here: https://lnkd.in/gNVivjdt Final thoughts: Agentic AI is powerful. However, in deep science/technology analysis and research, critical thinking, expert questions, close supervision, and detailed review of the generated content, backed by domain knowledge, would be prerequisites to get reliable outcomes. Agentic AI holds the promise to increase the productivity of such a human expert 10-fold. I created the prompt and the entire report in 48 hours.

2

On-device AI becomes powerful only when it works reliably on real hardware. In this Melange Walkthrough, Yeonseok Kim and Sun-Q Kim break down how the full on-device pipeline works, from model selection and quantization to real-device benchmarking and SDK deployment. Select. Benchmark. Deploy. This is how production-ready on-device AI should operate. Watch the full walkthrough below. #OnDeviceAI #EdgeAI #MobileAI #AIInfrastructure #ZETIC #Melange

26

Excited to see real-world proof of ARM’s impact on streaming. This collaboration between Amazon Web Services (AWS), Playtech, and the Wowza team makes this a must-watch for anyone focused on streaming performance and cost.

18

1 Comment

We are reimagining PyTorch on TPU (https://lnkd.in/gCrM-TZD). This approach still calls XLA but in a profoundly different way, enabling a solid eager mode. In the limit, this new approach will enable a true JIT approach with recompilation and re-optimization, online feedback-directed optimizations, avoidance of OOMs, support for HW fungibility, hyper-optimizations of kernels, and more - all with close to zero code changes!

341

2 Comments

** OCP Ethernet for Scale-Up Networking (ESUN) 1.0 Spec Published ** The Networking Project in OCP has just officially published/announced the OCP ESUN 1.0 spec. It's been an exciting last few months, seeing leading AI & networking companies come together to ensure Ethernet can be the basis for the high-performance scale-up networks inside rack-scale AI systems. Huge thanks to partners at all the various companies who have been working through the tradeoffs, suggesting improvements, and getting convergence on this 1.0 spec. There's still work for us to continue beyond 1.0, so please get involved with the project if you're invested in Ethernet for Scale-Up Networking! 👉 Blog from OCP on ESUN 1.0: https://lnkd.in/gXyc4zFT 👉 Direct link to OCP ESUN 1.0 Specification: https://lnkd.in/gDMVk988 👉 OCP ESUN wiki with info on how to get involved! https://lnkd.in/g87atvhv #OCP #Meta #Networking #AI #Infrastructure

287

3 Comments

Today, I’m excited to share that Meta is deepening our partnership with AMD to help us rapidly scale our AI infrastructure. Lisa Su and her team have been great partners and we are taking this one step further.  This collaboration will bring up to 6GW of AMD Instinct GPUs into our data centers, enabling us to scale and accelerate our AI innovation at an unprecedented level.  We partnered closely with AMD to co-design end-to-end, from the chip to the rack, optimizing for performance per watt and per dollar. Working with AMD is one of the several ways we’re expanding our AI compute resources. By diversifying our partnerships and technology stack, we’re building a more resilient infrastructure that can adapt quickly as workloads evolve. The future of AI cannot be defined by models alone– it will be defined by the infrastructure that powers them at scale. That’s why we launched Meta Compute last month: a top-level initiative to ensure Meta can deliver world-class AI to billions of people, now and for years to come. Given the scale of our business and our products, no single chip can meet the demands of building personal superintelligence. We’ve embraced a flexible, portfolio-based strategy, driving our own innovations like our superclusters Prometheus and Hyperion, advancing our MTIA silicon program, and working with world-class partners like @AMD to diversify and strengthen our supply. Thanks @Lisa Su and her team at AMD for partnering with us to bring this agreement to life, and the many people across Meta who made this a reality. Compute is the currency of next-generation AI and we’re at the frontlines leading it. Let’s go! 🚀 https://lnkd.in/gvpk_2ij

2,066

56 Comments