Ali Ghodsi

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7

If you’ve seen my explainer video on MCP and would now like to see actual practical things you can do with it, _watch this video_. Sami walks you through using the Confluent MCP server to do things in Confluent Cloud under voice control. Now imagine taking this demo and extending it to run from your own agentic microservice. This stuff is all very, very doable now…

62

1 Comment

New research from Databricks. It's about training enterprise search agents via RL. KARL introduces a multi-task RL approach where agents are trained across heterogeneous search behaviors, constraint-driven entity search, cross-document synthesis, and tabular reasoning. It generalizes substantially better than those optimized for any single benchmark. KARL is Pareto-optimal on both cost-quality and latency-quality trade-offs compared to Claude 4.6 and GPT 5.2. With sufficient test-time compute, it surpasses the strongest closed models while being more cost efficient.

396

8 Comments

Andrew Ng reported from Davos, that AI deployment for transformative impact will require workflow redesign. It is "theory of constraints" all over again (remember Eli Goldratt's "The Goal"?), we have to understand that real business benefits come only from working across silos, from analyzing flow bottlenecks, from understanding that optimizing non-bottlenecks creates waste. As Ng noted, letting a thousand (AI) flowers bloom has failed to lead to significant payoffs. Obviously! We in the silicon industry are no strangers to this. Workflows have been stagnant for decades. Technologies for high level specification, high level synthesis, and formal verification are not yet integrated into the full silicon development flow. They are often done piece-meal, and enthusiasts' pilots are dropped at the hint of a failure or delay. People say they want formal verification in their flow, but want it yesterday and do not have the time to invest in training engineers for its rigorous usage. Ashish Darbari, responding to a question on "if formal verification is so powerful then how come it is not widely used", noted that the problem is in the name - the actual task to be done is formal analysis while the bug detection is only a corollary, but the industry puts the cart before the horse. Based on my experience with industrial practice of high level specifications and formal verification, I tend to agree with his succinct comment. Jan Rabaey, in a DAC keynote speech, said that someday he expects to see a comic book series entitled "adventures in formal verification". AI deployment faces the same problem, if not actively worked on by business leaders. To be fair, deployments of all disruptive technologies face this mental barrier, we are so used to thinking in terms of the old flow (using the old technology) that thinking of a new flow literally means us having to begin again from first principles - few leaders have the courage and the time to do that. Goldratt mentioned that he himself had faced the same mindset problem when deploying the early ERP solutions and was surprised why he wasn't getting the expected large benefits, and he said it took him many years before he could identify the core problem and define solutions for it, which he named (rather academically) as "theory of constraints". AI leaders would do well to rework the conventional workflows in a holistic manner in order to get the real benefits from this new technology. As Henry Ford had said, "If you always do what you've always done, you'll always get what you've always got".

21

As more of you leverage foundational models in your organizations, remember they weren't trained or tuned on your internal metadata and taxonomies. Get a headstart with the metadata that naturally sits inside of Astronomer and is kept up-to-date automatically as your organization evolves. Companies leveraging orchestration metadata and lineage achieve higher quality answers with a fraction of the tokens. More to come!

9

What are the most underrated Databricks capabilities? Watch my conversation with Shiv Trisal (Global Industrials GTM @ Databricks) on the topic below. For the full, practical, no-fluff discussion on real-world data + AI: https://lnkd.in/gMPsFHbY

89

Databricks and NVIDIA introduce Genesis Workbench, an open source blueprint for running modern life sciences and drug discovery AI workflows on Databricks. Learn how open source biology models and NVIDIA’s accelerated BioNeMo stack can be used to run large-scale inference, fine-tune models on governed enterprise data, and move drug discovery experiments from the wet lab to in silico—faster, cheaper, and at scale. Genesis Workbench is open source and available today!!! This is a big deal for #lifesciences and #drugdiscovery. Scientists can discover drugs faster. They can fail fast, iterate faster, and move promising drug candidates forward with confidence. Faster. Cheaper. At scale. Srijit Chandrashekhar Nair Mayank Tahilramani Adam Crown Mark Lee Oliver Chiu Prasad Kona Nima Badiey Joel Caruso

34

6 Comments

Starburst has always believed in providing customers with the freedom of choice. Nowhere is this more true than the emerging developments around interoperable compute among data platforms. Recently, I sat down with Snowflake’s Ryan C. Green to talk about the Open Semantic Interchange (OSI), an open, vendor-neutral standard for sharing semantic models across AI, BI, and analytics tools. OSI gives customers a common, open way to define and share business metrics. This helps them stay consistent across dashboards, notebooks, and machine learning models, no matter which tools they use. For Starburst, this furthers one of our core goals. Choice. Customers should be able to keep one trusted set of business definitions inside their Snowflake environment while still having the freedom to choose the tools that work best for them. OSI helps make that possible. It marks an important step in bringing Starburst’s engine to Snowflake-interoperable compute. The video from my chat with Ryan is coming soon 👀 In the meantime, you can get more details from our recent press release: https://lnkd.in/ev2XGB4u #OpenSemanticInterchange #OSI #Interoperability #AI #Data #Snowflake #Starburst

187

8 Comments

LLMs look impressive in demos — but become inconsistent once they're embedded in real workflows. BASE Studio helps teams fine-tune models from SFT to RLHF using data built for real-world use, not toy examples: 🔵 Multi-turn conversations that preserve context 🔵 Clear, followable instructions models can reliably execute 🔵 Code data that explains *why* decisions are made — not just the final answer 🔵 Reasoning data that strengthens step-by-step thinking For multimodal models, we also align prompts with visual inputs to reduce inconsistency in text-to-image and text-to-video generation. The result? Models that behave more consistently — even after deployment. Learn more 👉 https://boden.ai #EmbodiedAI #Robotics #AITraining #TrainingData #MachineLearning #DataPipeline #RobotLearning #MultimodalAI #Automation #ComputerVision #AIInfrastructure #Annotation

1

.@Databricks, @OpenAI form $100 million partnership http://bit.ly/4gD9BIY Databricks said OpenAI's foundational models will be available in the Databricks Data Intelligence Platform and Agent Bricks natively in a partnership worth $100 million.

1

Snowflake Cortex Search is redefining how we interact with data — blending AI, vectors, and search natively within the warehouse. It’s powerful, fast, and deeply integrated. And here’s the real magic: what if you didn’t even have to think about how to set it up? That’s where #Frosty (PATENT PENDING) comes in. At Gyrus, we’ve built automation that makes #CortexSearch effortless. #Frosty automatically discovers the right data to enable, determines which columns should be used for vector indexing and which for filtering, and even adapts to your data’s freshness — configuring the service to refresh results exactly when they should be. With #Frosty, #CortexSearch no longer waits on enterprises — it’s automatically ready wherever it fits best. #Snowflake #CortexSearch #Frosty #AI #Automation #AIAgents #AutoPilotForSnowflake

8

Great news for Spanner and BigQuery users. Datastream now provides a seamless, serverless bridge from Spanner’s transactional power to the high-performance analytical capabilities of BigQuery and BigLake. Get the most scalable and available database that runs Google with BigQuery and BigLake’s serverless AI-Native Lakehouse!

517

3 Comments