Sadeep Jayasumana

🧠 Breaking Through AI's Memory Paradigm Barrier Many of us have likely faced a fundamental architectural constraint that shaped our interaction and system design with LLMs: the rigid split between immutable parameter memory (the model's pre-trained knowledge) and ephemeral, stateless retrieval systems like RAG. This creates a challenging dichotomy - one end is crystallized knowledge that's expensive to update and prone to forgetting, while the other is a series of temporary patches lacking true integration or lifecycle management. A new paper, MemOS: A Memory OS for AI System, proposes a cognitive architecture that treats memory as a first-class, manageable resource, moving beyond the current limitations. What truly excites me is the concept of MemCubes, which are unified memory abstractions that bridge three distinct layers. 1. Parameter Memory: Deep semantic knowledge embedded in the model's weights. 2. Activation Memory: Dynamic in-the-moment inference states and KV-cache. 3. Plaintext Memory: Explicit knowledge fragments from external sources. Significantly, MemCubes aren't static. They can be combined, moved, and merged, allowing knowledge to shift easily between different types of memory. This structure reflects how human memory consolidates. Experiences gradually blend and strengthen over time. A memory can grow from a brief activation to a more stable parameter, similar to how our own memories become stronger through repetition or rehearsal. This has important implications. 1. Transferable Intelligence: Memory can be shared, making it possible to transfer "experience" between agents without expensive retraining. 2. Continuous Learning: It sets up systems that can progressively learn and adapt from each interaction. 3. Deep Personalization: Agents can be tailored by adjusting their memory to fit specific contexts and users. MemOS reportedly achieves state-of-the-art performance, outperforming strong baselines (mem0, LangMem, Zep, OpenAI-Memory), especially in complex multi-hop and temporal reasoning tasks. The authors term this new paradigm as "Mem-training" (beyond pre and post training) - where the continuous evolution of memory, not just sporadic parameter updates, drives capability advancement. It suggests we're nearing an inflection point where memory architecture could become more critical than raw model scale. This could represent a major shift from perception-based AI to true memory-driven intelligence. The future may not only focus on creating larger static models but also on developing systems that can genuinely learn from experiences. This foundation is essential for the truly adaptive, continuously learning agents of tomorrow. 🤗 📄 Paper: https://lnkd.in/gdHDrZEa #AI #MachineLearning #LLM #MemoryArchitecture #Research #Innovation

8

🚀 Smarter, Faster RAG: A Policy-Optimized, Memory-Efficient Framework for Test-Time Reasoning 📄 New arXiv print 🔗 Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding https://lnkd.in/dErdTDf7 🔍 The Problem Retrieval-Augmented Generation (RAG) boosts LLMs with external knowledge but suffers from: ❌ Static retrieval schedules ❌ Distractor-induced hallucinations ❌ Linear memory growth in KV caching ❌ Inefficient test-time compute without reward alignment. 🧠 Our Framework: PORAG + ATLAS + CRITIC We propose a three-component architecture that dynamically optimizes retrieval, rewards factual and high-quality outputs, and reduces compute/memory costs — all without altering the pretrained base model at inference. 1️⃣ PORAG: Policy-Optimized Retrieval-Augmented Generation • Built on Group Relative Policy Optimization (GRPO) • Fine-tunes with QLoRA adapters, guided by two specialized reward heads: – Retrieval Fidelity (factual correctness) – Response Quality (fluency, coherence, utility) • Enables single-shot decoding with KL-regularized policy updates for stability 2️⃣ ATLAS: Adaptive Token-Layer Attention Scoring • Detects when to retrieve via Multi-Layer Attention Gradient (MLAG) • Builds what to retrieve via Layerwise Representation Pooling (LRP) • Leverages hybrid attention-embedding relevance, with entropy-aware scaling to manage retrieval load 3️⃣ CRITIC: Cache Reduction via Importance-based Token Inclusion Criteria • Selectively retains high-impact tokens in KV cache • Hybrid importance signal: attention strength, entropy spread, gradient sensitivity • Reduces memory and latency for long-context generation, with adaptive compression ⚙️ Exhaustive Test-Time Inference Scaling — No Fine-Tuning Required Our framework integrates advanced inference-time scaling techniques that allow LLMs to trade off compute, latency, and answer quality — dynamically and intelligently. These include: ✔️ Entropy-Guided Decoding — invoke deeper reasoning under uncertainty ✔️ Monte Carlo Tree Search (MCTS) — strategic decoding via multi-branch simulation ✔️ Mixture of Agents — task-routed decoding using a pool of specialized policies ✔️ Lookahead Decoding — speculative multi-token generation and verification These methods are plugged into our test-time control module, which uses token-level entropy, compute budget, and reward models to dynamically select the optimal strategy for each input. 🧠 Conclusion Our framework enables scalable, retrieval-aware LLM inference by combining reward-optimized generation, introspective retrieval, and memory-efficient decoding. It is: ✔️ Transformer-agnostic ✔️ Plug-and-play for SLMs ✔️ Ready for enterprise-scale, compute-sensitive deployments 📌 Full paper on arXiv: https://lnkd.in/dErdTDf7 🤝 With Akash Das, SHIVAM GUPTA, and Venkataramana Runkana #LLMInference #TestTimeScaling

17

🚀 Excited to share Compressive Light-Field Tokens (CLiFT) — a new scene representation for neural rendering developed by my students (Zhengqing Wang, Yuefan Wu, Jiacheng Chen, Fuyang Zhang). CLiFT is a Light-Field Token, compressed for compute-efficient and adaptive neural rendering, achieving (2x to 16x) reduction in storage and (66% to 100%) increase in FPS, while preserving high visual quality with only (0.54dB to 3dB) drop in PSNR. 🔗 Project: https://lnkd.in/gBqvMWi8 📦 Code coming 8/1!

168

3 Comments

Anyscale has contributed Ray to the PyTorch Foundation, joining PyTorch, vLLM, and DeepSpeed under open governance to advance scalable, distributed AI infrastructure. Janakiram MSV (Forbes) explains how this governance model reduces fragmentation and supports transparent collaboration across the AI ecosystem in this article: 🔗 https://lnkd.in/dwy4HpTC #PyTorchFoundation #Ray #OpenSourceAI #AIInfrastructure #DistributedAI

137

1 Comment

A new RL algorithm that fixes a hidden flaw in PPO The authors propose CE-GPPO (“Coordinating Entropy via Gradient-Preserving Policy Optimization”), a variant of PPO that restores gradient contributions from clipped actions in the policy update. They argue that traditional clipping discards useful gradient signals from low-probability tokens, which play an important role in controlling the agent’s entropy during training. By bounding those gradients in a controlled way, CE-GPPO maintains exploration–exploitation balance more stably than prior methods. They provide a theoretical analysis showing that CE-GPPO mitigates entropy instability, and empirically test it on mathematical reasoning benchmarks. Their results indicate consistent improvement over strong baselines across different model scales, demonstrating that preserving clipped gradients can lead to better performance in reinforcement learning for reasoning tasks. https://lnkd.in/gYZsJ-8f

1