Cheng Li

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

967

21 Comments

We just published our first technical research note. "Low-Latency Event-Driven Architecture for Real-Time Options Execution on IBKR" It covers how we eliminated compounding feed lag at SPY market open — from a growing 45-second backlog to stable sub-second bar sealing — through: → Queue-first callback offloading (Python GIL problem) → Per-venue DOM isolation across ARCA, ISLAND, IEX → Event-time binning across 6 independent intervals → Single source of truth for all technical indicators No signals. No alpha. Pure infrastructure. This is the kind of work that happens before a single trade fires — and the kind that determines whether your system survives the opening bell. 📄 Download: https://lnkd.in/g-XJsYwG #AlgoTrading #OptionsTrading #MarketMicrostructure #SystematicTrading #Python #QuantitativeFinance #TradingInfrastructure

1

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Our latest PyTorch community blog from the SGLang Team covers how SGLang advances hybrid Mamba plus Attention models with improved memory efficiency, prefix caching, speculative decoding, and serving performance. 📖 Read the blog: https://lnkd.in/gfDZkZWF #PyTorch #AIInfrastructure #SGLang #LLM

110

1 Comment

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1

Rachit Jain's walkthrough shows how to profile kernels with PyTorch tools, import traces into NVIDIA Nsight or Intel Advisor, and read the slanted ceilings that separate compute-bound from memory-bound regions.

5

🔥 Huge milestone — WEKA just open-sourced their software-defined memory for PyTorch and LLM KV Cache! Proud to be a launch partner alongside vLLM, NVIDIA, PyTorch, and Oracle cloud — enabling faster, leaner LLM inference at scale with LMCache Lab as the caching layer. Open infra for the GenAI era is here. Learn more about us here: 🔗 https://lnkd.in/gvsbCG5p #LMCache #WEKA #LLMInference #SoftwareDefinedMemory #OpenSourceAI #PyTorch #vLLM #KVCache #AIInfra

12

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments