sreeprasad Govindankutty

🧪 If verification asks “is it correct?” but we also need to ask “what is happening over time?” In an earlier post, I argued that evals shouldn’t be treated as QA artifacts added after the fact and that they should be executable intent: acceptance criteria made runnable, measurable, and repeatable. The research on AI‑assisted development reinforces the next step of that argument. As David and I have been discussing, we need to expand how we think about quality: Verification → correctness against rules Validation → fitness against requirements Scientific evaluation → emergence, drift, accumulation of complexity, long‑horizon effects of local optimizations, and whether the system stays within its intended operating envelope. This is standard thinking in science but still rare in software engineering practice. Why does this matter? AI‑driven development collapses the boundary between building and optimizing. When intervention happens faster than understanding: velocity becomes indistinguishable from progress, short‑term gains hide long‑term drag, and technical debt becomes invisible until it’s systemic! That’s exactly what the Cursor study shows: speed first, complexity later, slowdown eventually. If your AI development process optimizes speed without continuously evaluating what that speed is doing to the system, you are accumulating invisible technical and organizational debt. The next maturity curve isn’t better prompts or faster agents. It’s treating evaluation as a scientific instrument, not just a software gate. #Evals #AIGovernance #AIProductManagement #AIInfrastructure #AgenticAI

9

3 Comments

Struggling to ensure your LLM performs reliably in production? Here is an interesting article from my wife Sonakshi Singh This framework turns SME-labeled ground truth into a scalable evaluation pipeline. Goal Classify Q&A pairs into Correct, Incorrect, and Partially Correct in a live, production-grade system. Challenges 1.SMEs have limited availability and often provide only partial ground truth, without offering rationale for every record. 2.Understanding the rationale behind SME responses at scale is difficult. 3.Once we understand these rationales, the challenge is to use them to classify new Q&A pairs into Correct, Incorrect, or Partially Correct. Multistep Solution Step 1 – Use LLMs to extract the rationale behind SME responses. The LLM will classify them into the top 5 rationales (labels) with confidence scores. To achieve this, we provide all the available ground truth to the LLM and ask it to derive the top 5 rationales with confidence scores that illustrate why SMEs classified the pairs as they did. Step 2 – For each new Q&A pair, pass these rationales with confidence scores to another LLM and ask it to classify the pair into Correct, Incorrect, or Partially Correct. This, however, is a simplified view—using only 5 rationales and associated confidence scores per Q&A pair. In reality, a production-grade system would involve many more rationales and millions of Q&A pairs, resulting in very high token consumption and cost. Step 3 – To reduce dependency on an LLM-based classifier, we could build an in-house multi-class classifier. By combining the ground truth with LLM-generated labels and confidence scores, we can train this classifier to predict Correct, Incorrect, or Partially Correct. Step 4 – For each new live Q&A pair, use the LLM only to generate rationales (mapped to the predefined 5 labels) along with confidence scores. Step 5 – Pass this enriched dataset as input to the in-house multi-class classifier, which will then predict Correct, Incorrect, or Partially Correct. Conclusion With this approach, combining LLMs (to generate rationales) and a traditional ML classifier (to perform scalable classification), we can efficiently classify outputs into Correct, Incorrect, and Partially Correct at scale, while also creating a feedback loop to continuously improve performance. Closing the Gap: Production-Grade Evaluation Frameworks for RAG https://lnkd.in/eMJv8JNU

2

Existing LLMs are only as good as their training data. So what happens if a new frontend library is built that is wholly different to existing ones like React, Vue .js etc? Will models need to be updated with enough code examples from Github, etc or will official documentation be sufficient for it to build a frontend in said technology? Also, if LLMs are really good at building React apps, would that mean that overtime React will continue to solidify its position as the de-facto standard, especially amongst vibe coders? And going one step further, does this mean that with AI we would see less technological innovation, as the burden of development is moved from the developer to the agentic coder? Do we not risk long term future innovation if we go full auto-pilot and stop building systems ourselves?

19

14 Comments

𝗟𝗟𝗠 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝘁𝗼 𝗼𝘂𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝘁𝘂𝗿𝗻𝗲𝗱 𝗱𝗲𝗯𝘂𝗴𝗴𝗶𝗻𝗴 𝗶𝗻𝘁𝗼 𝗮 𝗻𝗶𝗴𝗵𝘁𝗺𝗮𝗿𝗲. 𝗛𝗲𝗿𝗲'𝘀 𝗵𝗼𝘄 𝘄𝗲 𝗳𝗶𝘅𝗲𝗱 𝗶𝘁 𝗶𝗻 𝟱 𝗱𝗮𝘆𝘀 𝘄𝗶𝘁𝗵 𝘇𝗲𝗿𝗼 𝗻𝗲𝘄 𝘁𝗼𝗼𝗹𝘀. 𝗧𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: A single user request triggers 200+ LLM calls. Which one failed? Where did context break? Our MTTR became as high as 7 hours. As a Startup, we cannot afford to have our reputation at stake. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗿𝘁𝘂𝗽 𝗗𝗶𝗹𝗲𝗺𝗺𝗮: Everyone suggested: LangSmith, LangFuse, Helicone. - Evaluating tools = 2-3 weeks - Every new tool = Learning curve + cost + vendor management - We don't even know what we need yet 𝗙𝗶𝗿𝘀𝘁 𝗣𝗿𝗶𝗻𝗰𝗶𝗽𝗹𝗲𝘀 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀: What exactly do we need to debug? Input sent, response received, post-processing results, and relationships between 200+ calls. The insight: Not a logging problem. It's a 𝘁𝗿𝗮𝗰𝗶𝗻𝗴 𝗽𝗿𝗼𝗯𝗹𝗲𝗺. Each LLM call is stateless. Our application maintains context. Same applies to observability. 𝗧𝗵𝗲 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻: We already had Grafana Loki, Tempo, and OpenTelemetry. The breakthrough: Connect all LLM calls under one parent span. 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 (𝗿𝗲𝗳𝗲𝗿 𝗱𝗶𝗮𝗴𝗿𝗮𝗺): 𝟭. 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿 OpenTelemetry emits logs and traces 𝟮. 𝗟𝗼𝗴𝘀 (𝗚𝗿𝗮𝗳𝗮𝗻𝗮 𝗟𝗼𝗸𝗶) Captures each LLM interaction: - trace_id & span_id - Message sent & response received 𝟯. 𝗧𝗿𝗮𝗰𝗲𝘀 (𝗚𝗿𝗮𝗳𝗮𝗻𝗮 𝗧𝗲𝗺𝗽𝗼) Parent span connects all related calls: - "Analyze PDF" → Factor 1 (Profit) → Factor 10 - Each factor → LLM Call 1... Call 20 Shows serial vs parallel execution, timing 𝟰. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗨𝗜 (𝗚𝗿𝗮𝗳𝗮𝗻𝗮) Query by span_id → Complete journey with all logs 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀: - MTTR: 7 hours → 30 minutes - Timeline: 5 days(2 design + 2 dev + 1 infra) - Cost: $0 in new tools - Complexity: Zero new vendors 𝗞𝗲𝘆 𝗗𝗲𝘀𝗶𝗴𝗻 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀: 𝗪𝗵𝘆 𝗧𝗿𝗮𝗰𝗲𝘀 𝗢𝘃𝗲𝗿 𝗟𝗼𝗴𝘀? Logs show events. Traces show relationships. For 200+ interconnected calls, relationships matter. 𝗪𝗵𝘆 𝗘𝘅𝗶𝘀𝘁𝗶𝗻𝗴 𝗦𝘁𝗮𝗰𝗸 𝗢𝘃𝗲𝗿 𝗡𝗲𝘄 𝗧𝗼𝗼𝗹𝘀? Existing stack solved 80%. New tools solve 100% but cost 5x in time, money, and complexity. 𝗪𝗵𝘆 𝗣𝗮𝗿𝗲𝗻𝘁-𝗖𝗵𝗶𝗹𝗱 𝗦𝗽𝗮𝗻𝘀? Mirrors how our application maintains state. One request → Multiple factors → Multiple LLM calls. 𝗦𝘁𝗮𝗿𝘁𝘂𝗽 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗟𝗲𝘀𝘀𝗼𝗻𝘀: Don't: Add new tools before understanding the problem Do: Define what you need first (saved 3 weeks) Don't: Evaluate 5 tools when you don't know what you need Do: Use existing stack, learn what's missing, choose deliberately Don't: Wait for the "perfect" solution Do: Ship 80% solution in 5 days, iterate based on production issues We'll eventually need specialized AI observability tools. But not today. Tomorrow, we'll know exactly which tool solves which problem. That's 𝗦𝘁𝗮𝗿𝘁𝘂𝗽 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Choose simple over perfect solution. Solve for 80%. Iterate to 100%.

28

2 Comments

Why is Redis so fast? There are many design decisions that make Redis blisteringly quick. We’ll focus on two that carry most of the weight. - First is Redis’s in-memory design + single-threaded event loop. Keeping data in RAM and processing commands in one core loop avoids locks and random I/O. - Second is its purpose-built data structures and lean wire protocol. Most commands hit O(1) or O(log N) paths with tiny per-op overhead. What happens when you read from Redis👇 Step 1: A client sends a command using RESP (a simple, compact text/binary protocol). Minimal parsing and small payloads reduce CPU and network overhead. No big JSON/XML blobs. Step 2: Redis’s event loop accepts the socket and queues work. Single-threaded command execution means no lock contention or context-switch thrash. If you want to use more CPUs you need to run more Redis instances in a cluster. Step 3: The command is parsed and routed via a hash table lookup to the target keyspace entry.  Lookups are O(1) on average thanks to efficient dictionaries and cache-friendly memory layouts. Step 4: The operation runs entirely in memory using specialized structures:   Strings, lists (quicklist), hashes (compact encodings), sets/intsets, sorted sets (skiplist + hash), streams, etc. They're are engineered for predictable, fast operations with tight CPU caches. Step 5: The response is written back through the same event loop.   Pipelining and batching can amortize syscalls and round trips, pushing throughput even higher. Step 6: Persistence and replication are off the critical path.   AOF uses append-only, sequential writes with configurable fsync; RDB snapshots happen in a child process.  Basically: we can lose data! This is a tradeoff Redis makes. Replication is async by default. The slow stuff is handled in the background so the hot path stays hot. Because Redis keeps data in RAM, executes commands in a single, lock-free event loop, and uses highly optimized data structures and a lean protocol, it avoids the latency traps of disk I/O, heavy parsing, and lock contention. That’s why it’s fast. Reed more about Redis here! https://lnkd.in/g2TEVzvx

163

This is the next stage of AI. I've been coming to the same conclusion, albeit from a slightly different direction. Have you ever had a long conversation, and in the middle of things something's brought up that you think is really insightful, but then you lose the thread and you can only dimly remember what it was that you talked about? That's the current state of Transformers. Ideas are amorphous, often are highly contextual, and they shift and evolve over time. Contexts are both finite and bounded. Internally we abstract, but abstraction is almost invariably lossy. The next possible stage is persistent memory. RAG by itself isn't enough - it's primarily just a hack of LangChain to allow external services. What we need is a way of growing and holding named in-memory graphs so that they retain persistence. We need addressibility into that graph. Maybe Deepseek's architecture will be the one to do so, maybe someone else will figure out an alternative, but I think that has to happen for language models to get (mostly) past decaying coherence.

46

8 Comments

A lot of posts recently talking about the shift away from the physical act of writing code/syntax going around so I’ll weigh in with my personal experience. 2025 was a watershed moment for LLMs. With Claude Code it was finally “smart” enough for mass adoption. It now writes code better than most “programmers”. But here’s the important distinction. Engineering and Programming are not the same. As engineering leadership I see it every day. People who think writing code is their job, not requirment gathering, planning, designing and architecting systems, validating correctness, problem solving. People who have these skills, can make effective use of Claude. These are the successful early adopters and 10x engineers right now. Now here’s what that means for me personally. After a long time, I am enjoying being a software engineer again. There was a time I was deeply married to writing code (it was kind of the therapeutic part of software development for me). But the shift into people management, becoming a father, and other life commitments has slowly erroded any time I have to take on projects that require hours or continuous focus. Writing code became the moat around me inventing. That is no longer the case. LLMs have enabled me to enjoy software development again. I can work in more bite sized chunks that fit into my day. I can engineer solutions in idle time. I can invest in custom tooling quickly that would have been month long side quests before. I can dip into other development domains (backend, browser extensions, languages) that previously would have take months of learning to become effective. This is what it has meant to me, do you have any similar experiences?

40

40 Comments

Direct quote from Andrej Karpathy on X. https://lnkd.in/gqMxQ-is "I've never felt this much behind as a programmer. The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse and between. I have a sense that I could be 10X more powerful if I just properly string together what has become available over the last ~year and a failure to claim the boost feels decidedly like skill issue. There's a new programmable layer of abstraction to master (in addition to the usual layers below) involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations, and a need to build an all-encompassing mental model for strengths and pitfalls of fundamentally stochastic, fallible, unintelligible and changing entities suddenly intermingled with what used to be good old fashioned engineering. Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession. Roll up your sleeves to not fall behind." Key takeaways for me: 1) The playing field has partially reset. Nobody has decades of experience with this new “AI layer,” but strong fundamentals still compound faster than ever. Learn the new layer and keep sharpening the fundamentals. 2) The real skill gap is learning to think at the agent and orchestration layer. This is not just about writing prompts. It’s about designing systems that intentionally combine tools, context, memory, and workflows. 3) Everyone is figuring this out in public (including me). There are no true masters yet, but through hands-on building is how real expertise is forming. 4) Passive learning (or worse, ignoring that AI is here to stay) is the fastest way to fall behind. The only way forward is rolling up your sleeves and getting your hands dirty. This isn’t something to fear. It’s an inflection point. A paradigm shift. For everyone. If you engage, experiment, and keep learning, 2026 is going to be an incredible ride.

2