Dmitry Kotlyarov

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7

🚨 Data streaming is due for its Iceberg moment. In this new post, our CEO, Sijie Guo, and Principal Sales Engineer, David Kjerrumgaard, explore why tightly coupled streaming architectures like Apache Kafka are hitting their limits—and what the future looks like. 👉https://hubs.ly/Q03tR4QS0 Drawing lessons from the lakehouse revolution (e.g., Apache Iceberg), they propose a three-layer architecture for streaming:
 🔹 Data – Durable, scalable object storage
 🔹 Metadata – Decoupled and authoritative
 🔹 Protocol – Stateless, multi-interface routing Streaming should be as modular, cloud-native, and vendor-neutral as modern analytics: 📉 Cut costs by up to 90%
 ⚙️ Evolve each layer independently 🔓 Enable open, multi-tool access Projects like #ApachePulsar and #StreamNativeUrsa are already paving the way!🌟 #StreamingData #Kafka #Iceberg #Lakehouse

6

Airflow migrations don’t have to be painful. In “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI," we talk with Ethan Shalev at Wix about migrating ~8,000 DAGs to Airflow 3, using Cursor to speed fixes and cutting CPU from 50% to 4% with deferrable operators. Click the link in the comments for the episode. #AI #Automation #Airflow #MachineLearning

13

1 Comment

LLMs look impressive in demos — but become inconsistent once they're embedded in real workflows. BASE Studio helps teams fine-tune models from SFT to RLHF using data built for real-world use, not toy examples: 🔵 Multi-turn conversations that preserve context 🔵 Clear, followable instructions models can reliably execute 🔵 Code data that explains *why* decisions are made — not just the final answer 🔵 Reasoning data that strengthens step-by-step thinking For multimodal models, we also align prompts with visual inputs to reduce inconsistency in text-to-image and text-to-video generation. The result? Models that behave more consistently — even after deployment. Learn more 👉 https://boden.ai #EmbodiedAI #Robotics #AITraining #TrainingData #MachineLearning #DataPipeline #RobotLearning #MultimodalAI #Automation #ComputerVision #AIInfrastructure #Annotation

1

🚀 Metaflow fans—what if getting real-time data into your flows was… simple? We just dropped a step-by-step guide showing how to use Conduit to stream data from Kafka into S3, then process it with Metaflow. No duct tape. No endless configs. Just clean pipelines from event to model. 🔁 If you’re tired of batch jobs and want something faster, this one’s for you. 👉 Read the full post: https://lnkd.in/dgGXaGhV #Metaflow #MLOps #StreamingData #Kafka #DataEngineering #OpenSource #Conduit #RealTimeML #Python #AWS #S3

11

Has Postgres become hard to change? In this T3D Postgres Pet Peeves episode, JD chats with Inferal founder Yurii Rashkovskii explore a tension many engineers feel but rarely say out loud: stability vs innovation. They dig into storage architecture, tightly coupled subsystems, the cost of experimentation, and whether Postgres’ greatest strength, predictability, is also slowing its evolution. If you’ve ever pushed Postgres beyond the “average install,” this one will hit close to home. Listen here: https://lnkd.in/db2nqpei

3

Learn how to build an image search app using a series of guided code commits with Developer Advocate Guy Royse. He’ll guide you through generating image embeddings using freely available Hugging Face models and storing them in Redis for fast, scalable semantic search.  The workshop covers: ▶️ What embeddings are and how they work ▶️ How to generate image embeddings and store them in Redis ▶️ Building a semantic search app with Redis, Python, and JavaScript Get access here: https://lnkd.in/gbZ3imbz

44

Build a real-time lakehouse with Apache Doris + AWS Glue + S3 Tables ➡️ Join our meetup in SF to see the full demo: https://lnkd.in/gRmeZWUd 🔧 Stack: - S3 stores Iceberg table data - AWS Glue as a central metadata catalog for schema and table management - Apache Doris (via VeloDB) brings unified batch + real-time SQL analytics on top 💡 Pros of this setup: - Real-time analytics: Doris delivers sub-second, high-concurrency queries directly on S3 data. - Open architecture: Keep data in open formats (Parquet, Iceberg) for multi-engines access. - Simple AWS integration: Glue makes tables easily managable across AWS services (Athena, EMR, Redshift Spectrum...) - Cost-efficient: Cheap S3 storage, elastic Doris compute, and no complex ETL pipelines. Our quick demo below shows connecting Glue and S3 to Doris through VeloDB Studio, then create and query a new table, all in real time. 🎟️ RSVP link at the top, join our event in SF to see our demo and demos from Uber, Pinterest, and Datastrato. #AWS, #ApacheDoris, #Gravitino

13

As more of you leverage foundational models in your organizations, remember they weren't trained or tuned on your internal metadata and taxonomies. Get a headstart with the metadata that naturally sits inside of Astronomer and is kept up-to-date automatically as your organization evolves. Companies leveraging orchestration metadata and lineage achieve higher quality answers with a fraction of the tokens. More to come!

9

“1 million rows per second.” In his opening keynote, our CEO Max Liu walked through what that really means in production: - TiDB processing 1M rows/sec - 10x higher throughput for cross-region replication - A data layer built to make multi-tenant architectures actually practical at scale If you’re thinking about SaaS, multi-region, or AI-heavy workloads, this is the kind of infrastructure ceiling you want. 🎥 Watch the full keynote from Max Liu at #TiDBSCaiLE to see what’s behind the numbers: https://ow.ly/BgKl50YetBq #TiDB #DistributedSQL #MultiTenancy #CloudNative #Scalability

25

The most effective architectural changes are often subtractive - removing entire layers of cost and complexity. Behind the scenes, OpsHelm, Inc.'s streaming backbone was migrated from a costly MSK and unreliable NATS setup to a single diskless Kafka architecture. The results were immediate: - Annual spend dropped from >$50,000 to <$10,000. - Data loss incidents went from recurring to zero. - Manual scaling and cross-cloud networking fees were eliminated. Read the full story of their migration and 78% cost reduction: https://lnkd.in/dS6UK9-7 #disklesskafka #datastreaming #Aiven

31

How Attentive uses Gatling and Datadog to achieve 360° observability at scale For teams operating at scale, performance issues are rarely clear-cut. The real challenge is understanding whether traffic is flowing, where it slows down, and why. In an upcoming live session, Bian Jiang, Principal Software Engineer on Attentive's Developer Experience team, will walk through how Attentive connects Gatling and Datadog to gain end-to-end visibility during high-scale load testing. We’ll explore: - How Attentive correlates generated load with real system behavior - Why observability is critical to trusting load test results - Lessons learned testing systems under real constraints - How Datadog and Gatling together close blind spots at scale 📅 January 22, 2026 - 6PM CET / 12PM EST 🎙 Bian Jiang, Principal Software Engineer, Attentive 🗃️ Diego Salinas, Enterprise Content Manager, Gatling 🗃️ Heather Thacker, Developer Relations, Gatling 🔗 Register here: https://lnkd.in/eqvEuMhB

18

2 Comments

Common GPU Bottlenecks During Fine-Tuning and How to Avoid Them in 2025 Fine-tuning LLMs and diffusion models is expanding fast, but many teams still lose 30-50% of training time because of avoidable GPU bottlenecks. These slowdowns increase cloud spend, delay iterations & reduce model throughput. Knowing these bottlenecks early helps teams train faster and scale more efficiently. 1. Underutilized GPUs GPUs often operate at only 40-60% utilization because the data pipeline is slow to keep up. This forces GPUs to wait between batches, creating wasted compute hours. Many teams misdiagnose this as poor GPU performance when the real issue is upstream in data loading.  Fix: Use parallel data loaders, prefetching, NVMe storage and larger batch sizes to keep the GPU saturated. 2. VRAM Overflow Large models like LLaMA 3, Qwen & Mistral often exceed VRAM limits, which pushes computation onto the CPU. If this happens, training can slow down by as much as 40× because CPUs cannot handle GPU-level tensor operations. This not only hurts performance but also increases training instability.  Fix: Use gradient checkpointing, QLoRA (4-bit or 8-bit) and higher-VRAM GPUs like A100 80GB, H100 or L40S. 3. CPU Bottlenecks Even the fastest GPUs perform poorly when paired with underpowered CPUs that cannot keep up with preprocessing, tokenization or batch preparation. These CPU delays create idle time, making GPU performance appear inconsistent. As models grow, poor CPU throughput becomes an even bigger constraint.  Fix: Choose GPU servers with balanced CPU ratios, enable multi-threading and move heavy preprocessing to async workers. 4. Disk I/O Limitations When datasets sit on slow disks, the GPU repeatedly pauses to wait for data reads. This issue becomes even more noticeable with large datasets, high-resolution image pipelines where read operations dominate. The result is lower GPU utilisation & unnecessary delays.  Fix: Use NVMe SSDs, keep datasets local and avoid slow network-attached storage during training. 5. Wrong Precision Modes Many teams train in FP32 even though it offers no extra accuracy for most fine-tuning tasks. This wastes compute, restricts batch sizes and increases VRAM pressure, slowing down training overall. Lower precision modes deliver equal performance with dramatically faster throughput.  Fix: Use BF16/FP16 for most workloads and apply INT8/4-bit quantized fine-tuning when mem is tight. Want Faster Cost-Efficient Fine-Tuning?😄 If you want high-performance GPUs with predictable pricing, reservation options & architecture designed to remove these bottlenecks, Dataoorts is optimised exactly for fine-tuning. You get stable A100, H100 and L40S compute, fast provisioning and cost-saving reservation plans built for long training runs. Scale your fine-tuning with faster infrastructure, transparent pricing and smarter GPU planning. #AI #MachineLearning #LLM #GPUCloud #DeepLearning #FineTuning #A100 #H100 #Dataoorts #NVIDIA #GPURental #on-demand_cloud

4