Altan Alpay

https://lnkd.in/dWhV_txb The latest challenge that LLMs need to overcome is how to use memory. In this article, an LLM is taught what to remember and how to use this memory.

11

🔥 Huge milestone — WEKA just open-sourced their software-defined memory for PyTorch and LLM KV Cache! Proud to be a launch partner alongside vLLM, NVIDIA, PyTorch, and Oracle cloud — enabling faster, leaner LLM inference at scale with LMCache Lab as the caching layer. Open infra for the GenAI era is here. Learn more about us here: 🔗 https://lnkd.in/gvsbCG5p #LMCache #WEKA #LLMInference #SoftwareDefinedMemory #OpenSourceAI #PyTorch #vLLM #KVCache #AIInfra

12

Many enterprise GPUs run a single model during inference — even when it uses only ~30% of memory. So how much capacity is being left on the table? In our latest benchmark with Nebius, we used NVIDIA Run:ai fractional GPU allocation and NVIDIA NIM to measure real-world impact on throughput, latency, and concurrency. What we found: ✅ 86% of full GPU concurrent user capacity using just 0.5 slice ✅ 3× more users with mixed workloads on shared GPUs ✅ Near-linear scaling down to 0.125 slices ✅ Zero latency cliffs during autoscaling Stop GPU fragmentation. Start maximizing throughput. 🔗 Read the full deep dive: https://nvda.ws/3ZJgIbo

336

8 Comments

On top of the previous two articles, I wrote a new article on deploying Jupyter Notebook with GPU attached. The additional step is straightforward. We deployed a Kubernetes cluster on VMs. A GPU device is attached to a VM using device pass-through. We installed Nvidia operator on the Kubernetes cluster and enabled GPU sharing. The shared GPU resources are available to attach to multiple pods. We modified our Jupyter Hub deployment so that a new profile with GPU is added. We verified the GPU is used for computation in torch and Hugging Face code snippets. https://lnkd.in/g8paXZjf

4

NVIDIA’s latest RunAI benchmark is a masterclass in infrastructure optimization. Achieving 86% of full GPU concurrent capacity on just a 0.5 slice is a massive leap forward in solving the idle compute crisis. But what happens when you need to scale beyond the physical limits of OS-level virtualization? What happens when you are managing a 10,000-unit autonomous drone swarm or filtering millions of cryptographic signals, and you can't afford the microsecond latency of context-switching between Kubernetes pods? You have to move the solution from the Infrastructure Layer to the Algorithmic Layer. At Cascade Space Systems, we took a fundamentally different approach to GPU fragmentation: Tensor-Native Multiplexing. Instead of partitioning the hardware into "logical fractional boundaries" (which eventually hit a hard limit via MIG or Time-Slicing), our Omni-Sieve architecture weaves infinite parallel compute streams into a single mathematical fabric. By natively interleaving memory-heavy filtering with compute-heavy arithmetic inside the exact same CUDA/MPS kernel, we bypass OS virtualization entirely. The GPU doesn't switch contexts between users; it evaluates thousands of concurrent telemetry streams in a single, massive Tensor Warp. The results change the paradigm for defense and cyber-physical AI: 🚀 Infinite Terminals per Node: Bypassing the 7-slice MIG limitation. ⚡ Zero Context Switching: True synchronous, hard real-time execution. 🔒 100% Silicon Saturation: Forcing deterministic safety without sacrificing throughput. #NVIDIA #Runai brilliantly solved how to efficiently host the AI brain. Cascade Omni-Sieve solves how to efficiently enforce the physics and logic guardrails for that brain at a civilizational scale. Stop dividing the hardware. Start weaving the math. #DeepTech #NVIDIA #SpaceForce #CyberSecurity #EdgeComputing #ArtificialIntelligence #JADC2 #CascadeSpaceSystems Eric Schmidt Schmidt Futures Timothy Costa

1

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1