Zheng Shao

Starburst has always believed in providing customers with the freedom of choice. Nowhere is this more true than the emerging developments around interoperable compute among data platforms. Recently, I sat down with Snowflake’s Ryan C. Green to talk about the Open Semantic Interchange (OSI), an open, vendor-neutral standard for sharing semantic models across AI, BI, and analytics tools. OSI gives customers a common, open way to define and share business metrics. This helps them stay consistent across dashboards, notebooks, and machine learning models, no matter which tools they use. For Starburst, this furthers one of our core goals. Choice. Customers should be able to keep one trusted set of business definitions inside their Snowflake environment while still having the freedom to choose the tools that work best for them. OSI helps make that possible. It marks an important step in bringing Starburst’s engine to Snowflake-interoperable compute. The video from my chat with Ryan is coming soon 👀 In the meantime, you can get more details from our recent press release: https://lnkd.in/ev2XGB4u #OpenSemanticInterchange #OSI #Interoperability #AI #Data #Snowflake #Starburst

187

8 Comments

Deep-dive write-up of the final day of #FabCon by Carmel Eve: - Accelerating Fabric Migration: New Assistant Tools for Data Engineering and Warehousing (Jenny Jiang & Charles Webb) - Powering AI Foundry Agents with Azure AI Search (Farzad Sunavala) - Democratize Generating Business Insights with Azure Databricks through AI and No-Code (Adam Wasserman & Isaac Gritz) - Scaling and Protecting Data Engineering in Fabric: Best Practices for Success (Santhosh Kumar Ravindran & Ashit Gosalia) - Microsoft Purview Data Security Protections in the AI Era (Shilpa Ranganathan & Anton Fritz) 🔗 To read the blog post, see the link in the comments 👇 #MicrosoftFabric #AIFoundry #Purview #Databricks

7

2 Comments

Exa has raised $85M in Series B funding, at a $700M valuation, to build a new kind of search engine. One made for AI agents. Founded in 2022 by Harvard grads Will Bryk and Jeffrey Wang, Exa powers search for companies like Databricks, AWS, Vercel, and Anysphere (maker of Cursor). Unlike Google, which optimizes for ads and clicks, Exa charges per query—pushing it to deliver the highest-quality results directly to AI systems. Benchmark, led by Peter Fenton (early Twitter and Yelp investor), led the round and joined Exa’s board, betting that AI search will create a company worth tens of billions. The new funding will go toward expanding Exa’s GPU cluster, “Hephaestus,” and growing its San Francisco team. Bryk says the mission is simple: “Google was built for people searching the web. Exa is built for AI from the ground up.” Congrats to the team on their Series B! https://lnkd.in/gPgM85eG

1,023

46 Comments

Engineering teams are turning AI into real results, not just experiments. Last month at the ELC Annual in SF, Redis Principal Field Engineer Josh R. joined Vinay Perneti from Augment Code, Anand K Sankaran from Workday, and Forrest Thomas from Garner Health for a fireside chat on where AI is truly adding value. Hear how they drove adoption across their teams, improved workflows, and made AI deliver real results: https://lnkd.in/gDAjZsDF

23

Our portfolio company, TileDB , announced a partnership with Databricks to harness multimodal data to power AI in healthcare and life sciences. https://lnkd.in/g9Z56Bwk

6

It’s been the headline of the week: 𝐈𝐁𝐌 𝐢𝐬 𝐚𝐜𝐪𝐮𝐢𝐫𝐢𝐧𝐠 𝐂𝐨𝐧𝐟𝐥𝐮𝐞𝐧𝐭. A big question on everyone’s mind is how this will reshape the streaming landscape. In Yingjun Wu’s words, this acquisition does not mark the end of anything! It only accelerates a shift that was already in motion. The whole data infra stack is moving upward, away from raw plumbing and toward products that deliver visible value. This shift shows up in three places: ▪️ How analytics vendors treat ingestion ▪️ How Apache Iceberg changes the role of Kafka ▪️ How Kafka itself has to evolve. For details, read: https://lnkd.in/gXUMRyPA #risingwave #streamprocessing #datainfra

5

New research from Databricks. It's about training enterprise search agents via RL. KARL introduces a multi-task RL approach where agents are trained across heterogeneous search behaviors, constraint-driven entity search, cross-document synthesis, and tabular reasoning. It generalizes substantially better than those optimized for any single benchmark. KARL is Pareto-optimal on both cost-quality and latency-quality trade-offs compared to Claude 4.6 and GPT 5.2. With sufficient test-time compute, it surpasses the strongest closed models while being more cost efficient.

396

8 Comments

Ruokai Yin, our summer intern from Yale to Microsoft Azure, worked with Xuan Zuo, @Sattwik Deb Mishra, Hokchhay T. and Preyas Shah and me. We published a paper based on his project: https://lnkd.in/eHuG3BSt Summary of paper: Finding optimal sharding is hard especially for large LLMs. A large number of devices have to be divided into TP, PP, EP dimensions. For TP, we have to decide the tensor dimension e.g. head, hidden size to shard for every op. Additionally, TP can be one-dimensional/two-dimensional etc. This makes for a large sharding space that is hard to cover exhaustively. SoTA methods use heuristics to select sharding e.g. GSPMD, Sequence Parallelism, Helix etc. There are significant time gaps, months or years, between the discovery of each of these methods. We want to fast-forward the process of discovery by 1) using a formal representation of the sharding space, and, 2) using reinforcement learning to search the sharding space.    This paper demonstrates that our method is 3.5x better than random search, and on par with hand-tuned heuristics when looking only at 1D sharding and without allowing sequence parallelism. We aim to extend the work to multi-dimensional sharding as well as sequence parallelism. Beyond that, we wish to combine the problem of sharding with fusion, tiling and collectives as well.

58

YellowDog.ai just set a 10x benchmark uplift in scale computing, delivering 40,000 tasks per second (TPS) and managing 100,000 compute nodes in the cloud. What's even more interesting, that's 2x IBM Symphony and opens an intriguing pathway for these until-know closed/captive systems.

11

We co-authored this new white paper with JBF Consulting exploring what changes when transportation optimization moves beyond CPU-based architectures to GPU-accelerated optimization. One insight stands out clearly: the historic trade-off between optimization runtime, scale, and solution quality isn’t a law of physics - it’s an architectural limitation. In the paper, we go deeper into why CPU-based solvers hit that ceiling, how GPU-accelerated optimization changes the equation, and what that means for real-world, highly constrained transport networks. 👉 Read the full white paper - see the link in the first comment. #SupplyChainTechnology #TransportationOptimization #LogisticsStrategy #GPU #TMS #Chora #Tramm

10

3 Comments

Most teams don’t fail at Iceberg because of the tech — they fail because of everything around it. You can have the best format, fastest engine, and cleanest tables… and still sink your Lakehouse if your tactics and strategy don’t align. On October 16, Yingjun Wu and I will talk about why Iceberg adoption can be tricky — and how not to mess it up. Expect real stories, tactical advice, and maybe one or two jokes about data teams turning schema evolution into a full-time job. 🧊 This isn’t just about config files — it’s about mindset, ownership, and the reality of running open data infra in production. 🎟️ Save your seat: https://luma.com/fga9usb8 #Lakehouse #Iceberg Apache Iceberg #DataEngineering Vakamo RisingWave

17

1 Comment

Why MCP Matters for the Future of AI As LLMs move from demos to real-world systems, the biggest challenge isn’t model capability—it’s context, action, and integration at scale. In this blog, I break down Model Context Protocol (MCP): • Why prompt-based integrations don’t scale • How MCP solves the M×N integration problem • Why MCP is emerging as the operating system for AI agents If you’re building agentic systems, RAG pipelines, or enterprise AI, MCP is a concept worth understanding early. 🔗 Read the blog here: https://lnkd.in/gJpcAHdt #AI #LLM #MCP #AgenticAI #Architecture #FutureOfAI

4

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1

Modern data stacks rarely live in one system. That’s why we've expanded Cortex Code CLI beyond Snowflake workflows to support any data, anywhere, starting with dbt and Apache Airflow® patterns natively. From schema-aware discovery to DAG generation and debugging, it brings deeper context to multi-system data engineering – all while maintaining enterprise governance and model flexibility. And, we're opening access with a self-serve subscription model, so you can get started without a Snowflake account. Details here: https://lnkd.in/gf23cJA5 

50