Paul Wyatt

LLMs are everywhere and I believe one of the biggest challenges right now is aligning them to real-world utility Some thoughts from recent work: - Fine-tuning is great, but prompt engineering still wins for many lightweight use cases. - Evaluating with user-facing metrics (not just accuracy or perplexity) changes how you build. - Bias & fairness issues pop up even with apparently clean data. If you’re working with LLMs, what evaluation metrics do you prioritize, and why?

2

1 Comment

Document-centric tasks sit at the core of many enterprise, business, and government workflows. Search is important, but the real challenge is going beyond retrieval, enabling systems to reason across documents, verify facts, and handle multi-step information tasks. Great to see a new model from Databricks moving in this direction. I expect we’ll see many more models, including open-weight ones, designed specifically for document-driven workflows, an area with huge potential across enterprise and government use cases.

23

1 Comment

There’s a lot of momentum around agentic AI right now. 📌 In practice, many systems labeled “agents” are really workflow automation with LLMs and tool calling. That’s often the right solution - workflows are predictable, testable, and deliver value quickly. Problems show up when we deploy these systems expecting agent-level behavior. Workflows are optimized for correctness on known paths. Agents exist to recover when the path is wrong. If a system can’t recognize failure, update state, and re-plan its approach, autonomy won’t magically appear - no matter how many tools or prompt chains we add. I wrote a deeper breakdown (with diagrams + concrete examples) on Substack 👉 Curious how others are drawing this line in production systems.

29

4 Comments

Just-in-Time UI: At Aligned Labs we frequently share reports about the data collection, fine-tuning and eval work that we do with clients.  They often need different views or refined analysis, which is why we built Whiz-bang Boom! Our new tool takes a CSV or JSON file and generates a shareable, interactive report. You can easily modify the report or focus the analysis with a simple prompt, and it creates a new revision. You can organize reports into spaces and customize those spaces with a simple prompt. Try it out. https://wizbangboom.com #businessIntelligence #AI

25

2 Comments

There is Bad AI. And then there is Chiron. Not all AI is created equal. Bad AI: Query time reasoning on raw telemetry. Latency compounds at scale. Black box inference with no causal trace engineers can inspect. Alert volume stays high. Correlation happens after the fact. Infrastructure costs scale linearly with data volume with no ceiling. Then there is Chiron: Causal rules compiled at build time. This delivers sub 500ms deterministic RCA for the fastest possible MTTR. Single actionable alert per incident. Causality is determined before impact. Every alert carries a full evidence chain back to the root cause. Query store free architecture. A streaming state store runs alongside existing infra with no rip and replace. Compressed state over raw logs. The cost profile is decoupled from data volume. SREs have been trained to react faster. The better engineering problem is asking why we are reacting at all. That is the architectural shift Chiron is built on.

12

2 Comments

Recently got asked about our data modeling approach - first piece of advice isn't novel to those in the space: read The Data Warehouse Toolkit by Ralph Kimball & Margy Ross! Beyond that, here's our quick framework for handling complex datasets: **1. Large source systems with 50+ tables** Map the business process first. At Uber, we had thousands of tables - but only 20-30 powered 80% of decisions. Start with core tables, add complexity only where needed. **2. Schema changes** • Version control everything in dbt or similar modeling framework tool • Test in staging / development environments first • Document breaking changes religiously **3. New business requirements** Ask three questions before adding "just one more column": - Does this fit our existing model? - Who else needs this in 6 months? - What's the maintenance cost? Building clean data models isn't about perfection - it's about building something that evolves without requiring yearly rebuilds. What's your biggest data modeling challenge? #DataEngineering #Analytics

17

2 Comments