Srilatha Kothur

Building on top of large language models is fun, but getting consistent performance and reliability is extremely challenging. I love this post from Kimberly Patron about how Sierra uses health and performance-based traffic routing and request hedging to get meaningfully better performance and tail latency from high scale LLMs https://lnkd.in/g4qWRCb5

284

13 Comments

Document-centric tasks sit at the core of many enterprise, business, and government workflows. Search is important, but the real challenge is going beyond retrieval, enabling systems to reason across documents, verify facts, and handle multi-step information tasks. Great to see a new model from Databricks moving in this direction. I expect we’ll see many more models, including open-weight ones, designed specifically for document-driven workflows, an area with huge potential across enterprise and government use cases.

23

1 Comment

🚧 Lessons from AI in Production – #1: Understand Licenses As the Engineering Manager leading the GenAI Platform at Verizon — powering 1000's of experiments across teams and having been in the production AI space for over half a decade— I’ve spent a lot of time working with platform, legal, security, and responsible AI teams to address one core challenge: ⚙️ What does it take to get AI into production? I'm kicking off a new series to share real-world learnings and lessons from the production side of AI. ⚖️ This first post answers a common question: 👉 “Can we use this model or tool in production?” To answer that, we need to understand: ✅ Licenses that are enterprise-safe ⚠️ Licenses that come with obligations 🚫 Licenses you should avoid in production 💡 My advice: Before integrating any open-source model or framework, always understand the license. Because performant ≠ legally usable. ➡️ Swipe through the carousel to dive deeper. #GenAI #AICompliance #MLOps #LLM #AIInProduction #EngineeringLeadership #OpenSource #AIInfrastructure

28

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://gag.gl/WwWso7

1

Inference at scale is evolving. The future is composable systems where offline batch, online batch (near-realtime), and realtime traffic can all be served from a single, unified inference endpoint for most use-cases. This simplifies your architecture, reduces operational overhead, and increases throughput. Our new tutorial series will show you how to build it step by sep. In the first installment, Erik Saarenvirta walks you through creating a scalable image classification system on GKE that can be adapted to a variety of use-cases. Leave us feedback in comments or ask questions we are happy to answer. Kent Hua Ishmeet Mehta Erik Saarenvirta https://lnkd.in/dVNiUhwE

16

Benjamin Vaughan, Ph.D. shares some insightful wisdom here on the recursive dangers of synthetic data as well as the novel approach Vurvey Labs is taking to solve for the best of what #syntheticdata can be and the realities of needing #AIPoweredByPeople.

2