Krishna Balasubramanian

It was great catching up with Sean M. Kerner recently and discussing how many classical labeling workflows are now being adopted for evaluation, especially within agentic workflows. To quote the article: "If evaluation is just data labeling for AI outputs, then the shift from models to agents represents a step change in what needs to be labeled. Where traditional data labeling might involve marking images or categorizing text, agent evaluation requires judging multi-step reasoning chains, tool selection decisions and multi-modal outputs — all within a single interaction." For product leaders, evaluation needs to be a core part of building any AI-powered application. To get beyond demos, you need a reliable way to generate high-signal human feedback that continually improves your models, and this feedback need to be in consensus between PMs, Data Science and SMEs. Its the most important investment to make to get and stay in production. https://lnkd.in/eMKhBThV

53

4 Comments

Today the Pulse team published a deep dive on why a single “accuracy” score doesn’t tell you if a document extraction system will survive in production. The goal here is to lay out an introductory but still rigorous evaluation methodology - we have an exciting open-source benchmark building on this research coming out very soon. Let’s do the math: take 1,000 pages, each with 200 data elements. A model that’s 98% “accurate” on paper still produces 4,000 incorrect values. Now make some of those: 1/ Broken reading order that scrambles multi-column layouts 2/ Tables with shifted columns or missing headers 3/ Cross-page context lost entirely That’s enough to silently corrupt an entire dataset without throwing a single error. We’ve processed hundreds of millions of pages and built a multi-axis evaluation framework to measure what actually matters: reading order validation, region-level ANLS, reading order accuracy, TEDS for table structure, and continuity checks across page boundaries. The result? Fewer silent data corruptions, more predictable performance, and pipelines that keep working on the next million documents you haven’t seen yet. Full technical write up in the comments!

36

2 Comments

When talking with folks about evaluating AI coding tools, one metric that comes up frequently is "acceptance rate". It sounds straightforward: track all of the code that AI writes, and use how often a human developer accepts or rejects proposed changes as a measure of tool efficacy. But if you stop for a moment to think about the difference between, say, inline autocomplete in Copilot versus interactive agentic coding across multiple files with Claude Code, things start to get... confusing. For example, Cursor suggestions get accepted 81% of the time, but developers only end up keeping about half the lines. "Accepted" and "kept" are very different things — and that gap is exactly why this is harder to measure than it looks. Are you as confused as I am? Check out this Jellyfish Research post by my colleague Tomas Pardiñas where he breaks down the different coding tools and the different ways we use them and talks about what "acceptance rate" means across various types of AI development workflows. Worth a read, especially if you're trying to make sense of your own team's numbers: https://lnkd.in/eRGK5TPZ

35

2 Comments

“The AI-powered system is anti-fragile. Every user interaction, especially corrections, becomes a signal that enriches the system for everyone. This model enables continuous improvement. Traditional semantic layers are most accurate the day they're built, then slowly decay until someone rebuilds them.” In the very near future hopefully we’ll stop talking about these traditional semantic layers altogether. MotherDuck has come to the same conclusion as us at Fabi.ai : traditional semantic layers are a story told by vendors and they’re expensive, fragile and not all that useful in an AI world. True story: I was chatting with the head of engineering at one of the big BI firms peddling semantic layers, and he told me “to be honest, we see very little adoption, we’re mostly hoping people adopt it for AI” (so basically hoping AI needs it so they can sell their product)

7

7 Comments

Just-in-Time UI: At Aligned Labs we frequently share reports about the data collection, fine-tuning and eval work that we do with clients.  They often need different views or refined analysis, which is why we built Whiz-bang Boom! Our new tool takes a CSV or JSON file and generates a shareable, interactive report. You can easily modify the report or focus the analysis with a simple prompt, and it creates a new revision. You can organize reports into spaces and customize those spaces with a simple prompt. Try it out. https://wizbangboom.com #businessIntelligence #AI

25

2 Comments

As someone who's been intimately involved in AI for 18 years, I'm both excited and apprehensive about what tools like OpenClaw mean for the future. That said, this is the future SeerAI, Inc. has been planning for since day one. Check out this article on what SeerAI means for your data initiatives.

7

2 Comments

I came across this today: AI in Procure-to-Pay: Structural constraints holding back P2P platforms (link in comments). “Structural constraints?” At Silkline, we’ve been thinking about these problems since Day 0 - over two years ago. Not as academic critiques, but as real blockers to building procurement software people actually trust and use. Every step of our journey, the team has made explicit product and engineering decisions to answer one question: "How should this be built today, given what’s possible in 2026 - not how it’s been done for the last 20 years?" Given our growth last year (and an even stronger trajectory this year), it’s time to start showing the how. Next week, I’m challenging myself to do 5 posts in 5 days, breaking down how we’ve addressed each of these so-called “structural constraints” in practice: 1. Document-centric data models: Fragmented documents → fragmented context → weak AI reasoning. When data is trapped in PDFs and forms, AI can’t see patterns, history, or risk. 2. Weak state management: It’s not enough to know where you are. You need to know how you got here. Most systems lose the story. 3. Brittle integration architectures: The real problem isn’t “big data.” It’s lots of small, messy, constantly changing data. 4. Policy encoded as static rules: Real procurement is full of exceptions. Static rules break the moment reality shows up. 5. Explainability and trust: If you don’t know why, when, or how something happened, you won’t let AI touch real decisions. These aren’t “constraints” to work around. They’re design choices, and we’ve been intentional about them since day 1. More next week.

19

1 Comment

Three core problems we’ve encountered AT Votal.ai when deploying Reinforcement Fine‑Tuning (RFT) models in Prod - Reward definition is a product problem. A reward must simultaneously capture business goals, legal constraints, data validation requirements, and ethical policies—and stay up to date with external knowledge sources. - Reward collection is a data problem. High‑quality signals are costly or slow to obtain; cheap signals are often noisy and can degrade performance below that of an untuned frontier LLM. Striking the right balance of human, synthetic, and implicit feedback is an R & D challenge in itself. - Reward verification is a systems problem. Test‑time adjudication requires latencies in the millisecond‑to‑second range. Ideally, an LLM should be able to call external services—APIs, database checks, UI validations—for formal verification, rather than relying solely on text‑based heuristics. This is especially important when agents make multiple external or internal system calls like updating multiple tables in an multi-agent ai system or while calling multiple API's in sequence || parallel to external systems for an user op. We don’t yet have a universal, fully formalized solution yet, instead, we co‑design reward functions with each customer, tailoring them to specific use cases. This handcrafted approach lets us bootstrap larger datasets from a small seed and avoid the high costs of broad SFT based approach which needs expensive labeling. If you are working on these problems, DM me we can hang around to discuss a better way to attack this problem. It is a important problem to solve for us as it gives us a better performance compared to untuned frontier model and higher user satisfaction

34

2 Comments