yangying xu

New from O’Reilly + Redis: Managing Memory for AI Agents 📘 AI agents are only as good as their memory. This new report explores how short-term, long-term, and persistent memory shape agent performance—and how Redis provides real-time, scalable architectures for production-ready AI. 📖 Read the full report: https://lnkd.in/g7t--57p

50

1 Comment

Databricks Data Engineer with DevOps Skills Los Angeles CA (Hybrid) Long term No H1Bs&GCs     Must Demonstrate (Critical Skills & Architectural Competencies) Designing and implementing Databricks-based Lakehouse architectures on AWS Clear separation of compute vs. serving layers Ability to design low-latency data/API access strategies (beyond Spark-only patterns) Strong understanding of caching strategies for performance and cost optimization Data partitioning, storage optimization, and file layout strategy Ability to handle multi-terabyte structured or time-series datasets Skill in requirement probing, identifying what matters architecturally  12+ years of experience Strong hands-on experience with Databricks, including: Delta Lake Unity Catalog Lakehouse Architecture Delta Live Pipelines Databricks Runtime Table Triggers Databricks Workflows Proficiency in PySpark, Spark, and advanced SQL. Expertise with AWS cloud services, including: S3 IAM Glue / Glue Catalog Lambda Kinesis (optional but beneficial) Secrets Manager Strong understanding of DevOps tools: Git / GitLab CI/CD pipelines Databricks Asset Bundles Familiarity with Terraform is a plus. Experience with relational databases and data warehouse concepts. Please share suitable profiles to syam@s2jsolutions.com

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

VP & GM of AI & Infrastructure at Google Cloud, Amin Vahdat, shows off Ironwood, our seventh-generation TPU—from inside one of Google’s labs. Ironwood is purpose-built for the most demanding workloads—from large-scale model training and complex reinforcement learning (RL) to high-volume, low-latency AI inference and model serving. Learn more about our seventh-generation TPU → https://goo.gle/4osYw0o

1,974

20 Comments

In this post, we explore how to monitor and manage Amazon Bedrock batch inference jobs using Amazon CloudWatch metrics, alarms, and dashboards to optimize performance, cost, and operational efficiency.

1

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1

The harsh reality: 73% of distributed systems fail to scale beyond initial deployment. Is your AI startup heading for a compute crisis? ML workloads need 40x more resources than traditional applications, but most teams are using infrastructure strategies that break at production scale. The companies that make it past this bottleneck understand something critical about distributed architecture that the 73% who fail completely miss. It's not about adding more machines. It's about three specific design decisions that determine whether your system scales or collapses under load. Link in comments for the full breakdown of what separates the 27% who succeed.

20

1 Comment

🚨 Big AI + data news! - IBM to acquire Confluent🚨 🟦IBM announced its intent to acquire Confluent in an $11B all-cash deal ($31/share) to build a “smart data platform” for enterprise generative & agentic AI. ✅ Real-time data for AI; Confluent (Kafka at scale) gives IBM a powerful streaming backbone so AI apps & agents can tap into trusted real-time data across clouds, apps & APIs. ✅ AI stack gets deeper; this plugs directly into watsonx + IBM’s hybrid cloud & automation portfolio to offer a more end-to-end data + AI platform for enterprises. ✅ Signal for data engineers; Data-in-motion, event-driven architectures & governed streams are now firmly in the critical path of enterprise GenAI strategies. If you work in data, platforms or AI, this is one to watch 👀 #IBM #Confluent #Kafka #DataStreaming #GenerativeAI #AgenticAI #DataEngineering #MergersAndAcquisitions #EnterpriseAI #HybridCloud #technews 👉 https://lnkd.in/duY2z6Xp ♻️ IBM, Reuters

2