Ryan Waite

Have you heard of the llm-d project? It improves LLM performance and drives down costs of inference serving on Kubernetes. It's based around a few key projects including vLLM and kgateway. If you want a deep dive into how it works with a detailed step-by-step "packet flow" through llmd, take a look at this blog I wrote recently. 👉 https://lnkd.in/g2mbEh3m

44

3 Comments

As LLMs continue to show up across the federal space, one thing matters more than model size or hype: control. AWS quietly made an important move by adding response streaming support to Amazon API Gateway, and it directly addresses a challenge many of us hit when deploying AI in the federal arena. Streaming responses mean APIs no longer have to wait for a full LLM output before responding. Users can see results as they are generated, which improves perceived performance and reduces impatience waiting for a response. With streaming, teams gain: - More control over how quickly responses are delivered, including intentionally slower or paced outputs when overuse is detected. - Better user experience without bypassing security, logging, or gateway controls that come from workarounds. - Support teams that want to stay serveless and keep their architecture simple. The ability to shape how an AI responds is just as important as what it responds with. Serveless infrastructure is finally catching up to how LLMs are actually being used in production. https://lnkd.in/gxe_tkrp

9

4 Comments

People often ask me about the differences in architecture between Amazon Dynamo (the 2007 SOSP paper), DynamoDB (the AWS serverless NoSQL database), and Aurora DSQL (the AWS serverless SQL databases). I captured the key points on durability, high availability, programming model, and trade-offs in my blog: https://lnkd.in/gkMG_tVD TL;DR: Despite the shared name, Dynamo and DynamoDB have little in common architecturally, and make very different trade-off decisions.

631

8 Comments

To add to that, Confluent Cloud Network (CCN) routing simplifies private networking for Flink, reusing existing Kafka infrastructure. It helps teams quickly connect secure pipelines across cloud environments. Security is further strengthened with IP Filtering—giving teams greater control over access in hybrid environments and enabling them to confidently deploy AI at scale. With these enhancements, Confluent is not just enabling smarter AI—it’s making it enterprise-ready. Organizations can now unlock: Seamless integration of historical and streaming data Secure, scalable infrastructure for AI agents Faster, context-aware decisions powered by trusted data Agentic AI is here. With Confluent Cloud, it’s ready for the enterprise. https://lnkd.in/gh-dA7qs #ConfluentCloud #ApacheFlink #SnapshotQueries #AgenticAI #RealTimeData #StreamProcessing #DataAnalytics #EnterpriseAI #BatchProcessing #DataStreaming #DigitalTransformation #Flink #AI #MachineLearning #DataSecurity #PrivateNetworking #IPFiltering Follow our LinkedIn page for more such interesting CX expert opinions/ Thought Leader Interviews/ CX Leadership news/ articles/ updates https://lnkd.in/g9eeFFgh

1

Want your platform teams to be engines of margin expansion and not line items of infrastructure cost? Developer Experience (DevEx) remains one of the most underleveraged levers for improving return on invested capital (ROIC) in large enterprises, especially those not born digital. In every CTO role I’ve held, modernizing DevEx has been critical to scaling innovation, shortening time-to-market, and reclaiming the 20–30% of engineering time lost to undifferentiated work. When I joined Amazon in 2014, the bar was clear: new hires shipped code to production by the end of Week 1. That kind of velocity was engineered: -Infrastructure as code for consistent setup -Opinionated tooling like Brazil, Apollo, Pipelines -Platform teams owning everything from build to release gating Centralizing undifferentiated effort into shared infrastructure made speed, scale, security, and quality the default across teams. Fast forward to this summer: my daughter Riya interned at Amazon. On Day 2, she called me: “The tools here are wild. I’m already doing code reviews.” I’m a proud mom of this bright and hardworking intern, and I also know those foundational systems are now even better. That’s what compounding platform investment looks like. Yet across many enterprises, DevEx initiatives still struggle to gain CEO sponsorship—largely because we’ve lacked a clear way to measure return. That’s why I’m excited about Amazon’s new framework: 📊 Cost to Serve Software (CTS-SW) A financial model for measuring and improving ROIC from DevEx investments. It answers: how much does it cost us to deliver one unit of working software to production? CTS-SW=Development Costs + Dev Infrastructure Costs/Software Delivery Units Lower CTS-SW signals higher platform efficiency and stronger ROIC from every dollar invested in DevEx—whether you're streamlining onboarding, accelerating deployments, integrating AI-assisted development, reducing tech debt, or scaling service ownership with better observability and tooling. CTS-SW also aligns central and federated teams to customer outcomes, breaks down silos, and frees engineers from undifferentiated work so they can focus on high-leverage innovation. In 2024, Amazon saw: -15.9% ↓ cost to build and operate software -18.3% ↑ deployments per developer (YoY) -For example, in a 1,000-dev org: that's a $20M annual efficiency gain For CTO/CIOs: CTS-SW bridges engineering productivity with financial performance For CEO/CFOs: it offers a data-backed lens to fund DevEx as a lever for margin expansion If you're building platform teams or modernizing software delivery, this is a framework worth your attention. Thanks Jim Haughwout Full article: https://lnkd.in/eiTrznxd #DeveloperExperience #ROIC #GenAI #CTO #CIO #CFO #CEO

23

With Finch (AWS's container development tool that provides an alternative to Docker), AWS SAM can provide a local environment similar to AWS Lambda as a container to build, test, and debug your serverless applications.

13

With AI adoption and deeper integration into enterprise products, the world moves toward being collectively intelligent. First movers will have a distinct advantage. Fully backward-integrated AI companies are best positioned to win on economics.

1