Le Li

Third piece in the AAD series is up. This one is about acceptance criteria (AC) and why treating them as a formal contract changes what you can diagnose when autonomous development breaks.  AC serve as the bridge between the spec and the environment in which the spec is compiled into a product. This is implicit in the first article, but after marinating on this for a few weeks, I felt like I should make it explicit.                                                                                                    The short version: most teams write AC the way you'd write them for a human. "Handle edge cases gracefully." "Performance should be acceptable under normal load." In AAD, those are  specification failures. An agent will do something when it reads that. It might even do something reasonable. But you've handed control over what "reasonable" looks like to the LLM, and you'll only find out whether you agree after you've burned the iteration cycle. In my experience, the first pass is ok, but it can quickly diverge and it takes a lot of time to get back to where you intended to go in the first place. One thing I keep coming back to is the diagnostic value of keeping the spec and the AC separate. My currently line of thinking leads me to this framing: The spec describes intent. The AC defines done. You need both. The Spec is the what and the why. The AC are the falsifiable tests that the intention was met. Keeping them separate creates a framework so they each get the attention they deserve as part of this new world. Building on the notion of falsifiability, there's also a section on the right mindset for writing AC. Just like with the best scientist/experimentalist, you should be writing AC from a lens that lets you confirm intention was not met, not that intention was met. It's a subtle but meaningful nuance. If you assume the agent is going to get it correct, you're biased to see the final product and assume it's correct. When you train yourself to focus on the ways the agent might get it wrong, you open up a whole new world on writing great AC. I think this applies to traditional development as well as AAD. Claude made a fancy doc for this one! But also, the full read is in the comments.

15

1 Comment

We're hiring three AI Analytics Engineers on the Data Science & Analytics team at Airtable. These are early to mid-career roles for builders who are excited about working at the intersection of data science, analytics, and AI. One of these roles will report directly to me as we spin up a new AI & Analytics Platform function. Things are moving fast. The most impactful work in data right now isn't building models — it's fundamentally changing how a company gets answers from its data. That's where my focus is, and it's why Airtable is investing in a new team to own it. Over the past six months, our team stopped experimenting with AI on the side and restructured around it: - Agent-powered development workflows — I built shared AI systems for the data team embedded in how we write SQL, build pipelines, document business logic, and debug. Team members are regularly seeing 2x to 10x speed improvements — not from shortcuts, but from AI woven into every step of how we build. - Conversational analytics — Ad-hoc questions that used to fill our queue from leadership, product, GTM, and finance are increasingly answered by an AI agent — in natural language, where people already work. -A context layer that makes it trustworthy — We built a system that gives the agent real business knowledge — which table is correct for a given analysis, how metrics differ across teams, why something spiked last quarter. Context that usually lives in an analyst's head. - A learning loop that compounds — Every user correction becomes a permanent improvement, not a one-off fix. — We're doubling down. Airtable is standing up an AI & Analytics Platform function within DS&A to keep building at the intersection of data science, ML, analytics, and AI — and I'm leading it. We need more builders. AI Analytics Engineer, AI & Analytics Platform First hire on this team. You'll build the context layer that makes AI tools accurate, design evaluation frameworks that make them trustworthy, and develop agent systems that let the org self-serve with confidence. You'll work directly with me to shape what this function becomes. https://lnkd.in/gFstQqQb We're also hiring two roles embedded in other parts of the business: AI Analytics Engineer, Marketing Analytics Own data infrastructure, dashboards, and AI-native tooling for Marketing. Build the canonical data sources leadership relies on. https://lnkd.in/gYHS55ma AI Analytics Engineer, Business Analytics Own financial data models powering ACV, ARR, billings, and revenue. Partner with a Finance team that's building their own AI agents and workflows. You'll build the data foundation they run on. https://lnkd.in/gV5hnJjG — I'm looking for people who build before they're asked. Who have a side project that started as "I wonder if I could..." and turned into something real. Who pick up a new tool and immediately try to break it. If that sounds like you — reach out or apply. I'd love to talk.

79

8 Comments

AI is good at things analytics is bad at, and vice versa? I recanted a recent AI finding to Matt Regan today over lunch today. I passed Claude Code a delimited file and asked it to "tell me how column H differs when column J has a value or not." Column H was an extended email exchange between two parties. Column J was more or less a success flag. I more specifically asked it to tokenize the emails, n-gram the results, and find good positive prediction indicators. I was at least smart enough to tell it to exclude reply text (repetitive). The results were wholly uninteresting. I asked it to try some more filtering methods (TF-IDF, etc.). Still, the results were lame. It seemed to be too focused on success indication and the top 100 results were largely unusable (names, generic business jargon, etc.). I decided to change tack ... "Tell me what you find in the email exchanges where the success flag was on." Now, here we go. I got some clear and interesting findings, with stats to back them up. I'd actually studied this data myself beforehand and could verify (for the most part). I kept mulling it over the weekend, and come this morning I decided to push a bit further. "Is there anything particular about the first few emails in the chain that indicates a higher possibility of the success flag and vice versa?" All of a sudden I had a half dozen good points on each side. What led here? Getting out of the way. I'd probably still be sparring with Claude, trying to get decent lists of positively and negatively correlated n-grams, if I hadn't stepped back. I'm super happy I rolled with the AI tide rather than swimming against it.

10

Excited to share that our team’s paper on aspect-guided review summarization has been accepted to EMNLP 2025’s industry track! 📈🧠🤓 In this paper, we describe a live system at Wayfair that leverages LLMs alongside classical NLP techniques to summarize tens of millions of customer reviews, making it easier for shoppers to discover what matters most to them on site. To further support the research community, we’re also open-sourcing a dataset of ~12M anonymized reviews and their extracted aspects which we hope accelerates progress in aspect extraction, sentiment analysis, and summarization tasks. This milestone reflects the hard work of many folks across Wayfair, including our co-authors Ilya Boytsov, Misha Balyasin, Joe Walt, Caitlin Eusden, Marie-Claire Rochat, and Margaret Pierson. https://lnkd.in/ezJMi3g7 #NLP #EMNLP #EMNLP2025 #LLMs #Summarization #ABSA #OpenSource #Data #Dataset #ecommerce #wayfair #machinelearning #ai #publication

74

3 Comments

There’s one idea I keep finding useful, whether working on ML problems or making everyday decisions: second-order thinking. First-order thinking asks: “What happens next?” Second-order thinking asks: “And then what?” As the Farnam Street blog puts it, it’s about understanding “the consequences of the consequences”—looking beyond the immediate outcome to how a decision plays out over time. It’s the difference between optimizing a metric and understanding the system behind it. In ML, a first-order decision might be: “ This model is directly optimized for revenue—ship it.  ” Second-order thinking asks: “How does this change user trust, content diversity, content bias, or long-term retention?” A real-world parallel is quick-commerce platforms that optimized aggressively for short-term conversion and revenue using so-called “dark patterns.” While these tactics delivered immediate gains, they triggered user backlash and regulatory scrutiny, ultimately undermining trust.  In organizations, first-order thinking pushes for speed. Second-order thinking considers culture, talent compounding, and the behaviors we unintentionally reward. Second-order thinking is harder because: - It’s less obvious - It’s harder to measure - It often conflicts with short-term wins But as highlighted in Farnam Street’s writing on mental models, this is exactly where long-term, durable advantages are built. When designing systems or products or sometimes in everyday work, it helps to pause and ask one extra question: “If this works, what happens next—and after that?”

62

5 Comments

In my recent conversations, I've noticed a 2 concepts that seem to be causing some confusion in the DS community, and I'd like to take a moment to clarify them: 1. Difference in Bias of an estimator and Bias variance trade off. 2. Covariate shift in causal inferencing vs feature drift. Bias in the Estimator refers to a single, standalone property of a statistical model's prediction. An estimator is considered unbiased if, on average, it hits the true value of the parameter it's trying to estimate. In other words, if you were to repeat your experiment many times, the average of your predictions would equal the true value. Whereas, The Bias-Variance Trade-off is a concept that describes a fundamental tension in model building for choosing the complexity level. Covariate shift in causal inference and feature drift in machine learning are related concepts, but they refer to different problems and appear in different contexts. Covariate Shift in Causal Inference describes a situation where the distribution of the input variables (covariates) in the test data is different from the distribution in the training data, while the relationship between the covariates and the outcome variable remains the same. In a causal context, this is a big problem because causation is lost which Invalidates Causal Conclusions. Feature drift, or concept drift, is a broader term in machine learning. It refers to the problem where the statistical properties of the target variable or input features change over time in an unexpected way.  Hope this help to the readers.

43

1 Comment