This article is a follow-up to my earlier post, AI Is Improving Software Engineering. But It’s Only One Piece of the System. In that post, I explored how AI is already helping engineering teams work faster and better, but also why those gains can be diminished if the rest of the delivery system lags.
Here, I take a deeper look at that system-wide perspective. Adopting AI is about strengthening the entire system. We need to think about AI not only within specific teams but across the organizational level, ensuring its impact is felt throughout the value stream.
AI has the potential to improve how work flows through every part of our delivery system: product, QA, architecture, platform, and even business functions like sales, marketing, legal, and finance.
If you already have robust delivery metrics, you can pinpoint exactly where AI will have the most impact, focusing its efforts on the actual constraints rather than “speeding up” work at random. But for leaders who don’t yet have a clear set of system metrics and are still under pressure to show AI’s return on investment, I strongly recommend starting with a platform or framework that captures system delivery performance.
In my previous articles, I’ve outlined the benefits of SEI (Software Engineering Intelligence) tools, DORA metrics (debatable), and, ideally, Value Stream Management (VSM) platforms. These solutions measure and visualize delivery performance across the system, tracking indicators like cycle time, throughput, quality, and stability. They help you understand your current performance and also enable you to attribute improvements, whether from AI adoption or other changes, to specific areas of your workflow. Selecting the right solution depends on your organizational context, team maturity, and goals, but the key is having a measurement foundation before you try to quantify AI’s impact.
The Current Backlash and Why We Shouldn’t Overreact
Recent research and commentary have sparked a wave of caution around AI in software engineering.
A controlled trial by METR (2025) found that experienced developers using AI tools on their repositories took 19% longer to complete tasks than without AI, despite believing they were 20% faster. The 2024 DORA report found similar patterns: a 25% increase in AI adoption correlated with a 1.5% drop in delivery throughput and a 7.2% decrease in delivery stability. Developers felt more productive, but the system-level metrics told another story.
Articles like AI Promised Efficiency. Instead, It’s Making Us Work Harder (Afterburnout, n.d.) point to increased cognitive load, context switching, and the need for constant oversight of AI-generated work. These findings have fed a narrative that AI “isn’t working” or is causing burnout.
But from my perspective, this moment is less about AI failing and more about a familiar pattern: new technology initially disrupts before it levels up those who learn to use it well. The early data reflects an adoption phase, not the end state.
Our Teams’ Approach
Our organization is embracing an AI-first culture, driven by senior technology leadership and, additionally, senior engineers who are leading the charge, innovating, experimenting, and mastering the latest tools and LLMs. However, many teams are earlier in their adoption journey and can feel intimidated by these pioneers. In our division, my focus is on encouraging, training, and supporting engineers to adopt AI tools, gain hands-on experience, explore use cases, and identify gaps. The goal isn’t immediate mastery but building the skills and confidence to use these tools effectively over time.
Only after sustained, intentional use, months down the line, will we have an informed, experienced team that can provide meaningful feedback on the actual outcomes of adoption. That’s when we’ll honestly know where AI is moving the needle, and where it isn’t.
How I Respond When Asked “Is AI Working?”
This approach is inspired by Laura Tacho, CTO at DX, and her recent presentation at LeadDev London, How to Cut Through the Hype and Measure AI’s Real Impact (Tacho, 2025). As a leader, when I face the “how effective is AI?” debate, I ground my answer in three points:
1. How are we performing
We measure our system performance with the same Flow Metrics we used before AI: quality, stability, time-to-value, and other delivery health indicators. We document any AI-related changes to the system, tools, or workflows so we can tie changes in metrics back to their potential causes.
2. How AI is helping (or not helping)
We track where AI is making measurable improvements, where it’s neutral, and where it may be introducing new friction. This is about gaining an honest understanding of where AI is adding value and where it needs refinement.
3. What will we do next
Based on that data and team feedback, we adjust. We expand AI use where it’s working, redesign where it’s struggling, and stay disciplined about aligning AI experiments to actual system constraints.
This framework keeps the conversation grounded in facts, not hype, and shows that our AI adoption strategy is deliberate, measurable, and responsive.
What System Are We Optimizing?
When I refer to “the system,” I mean the structure and process by which ideas flow through our organization, become working software, and deliver measurable value to customers and the business.
Using a Value Stream Management and Product Operating Model approach together gives us that view:
Value stream: the whole journey of work from ideation to delivery to customer realization, including requirements, design, build, test, deploy, operate, and measure.
Product operating model: persistent, cross-functional teams aligned to products that own outcomes across the lifecycle.
Together, these models reveal not just who is doing the work, but how it flows and where the friction is. That’s where AI belongs, improving flow, clarity, quality, alignment, and feedback across the system.
The Mistake Many Are Making
Too many organizations inject AI into the wrong parts of the system, often where the constraint isn’t. Steve Pereira’s It’s time for AI to meet Flow (Pereira, 2025) captures it well: more AI output can mean more AI-supported rework if you’re upstream or downstream of the actual bottleneck.
This is why I believe AI must be tied to flow improvement:
Make the work visible – Map how work moves, using both our existing metrics and AI to visualize queues, wait states, and handoffs.
Identify what’s slowing it down – Use flow metrics like cycle time, WIP, and throughput to find constraints before applying AI.
Align stakeholders – AI can synthesize input from OKRs, roadmaps, and feedback, so we’re solving the right problems.
Prototype solutions quickly – Targeted, small-scale AI experiments validate whether a constraint can be relieved before scaling.
Role-by-Role AI Adoption Across the Value Stream
AI isn’t just for software engineers, it benefits every role on your cross-functional team. Here are just a few examples of how it can make an impact. There are many more ways for each role than listed below.
Product Managers / Owners
Generate Product Requirements Documentation
Analyze customer, market, and outcome metrics
Groom backlogs, draft user stories, and acceptance criteria.
Summarize customer feedback and support tickets.
Use AI to prepare for refinement and planning.
QA Engineers
Generate test cases from acceptance criteria or code diffs.
Detect coverage gaps and patterns in flaky tests.
Summarize PR changes to focus testing.
Domain Architects
Visualize system interactions and generate diagrams.
Validate design patterns and translate business rules into architecture.
Platform Teams
Generate CI/CD configurations.
Enforce architecture and security standards with automation.
Identify automation opportunities from delivery metrics.
InfoSec Liaisons
Scan commits and pull requests (PRs) for risky changes.
Draft compliance evidence from logs and release data.
Don’t Forget the Extended Team
Sales, marketing, legal, and finance all influence the delivery flow. AI can help here, too:
Sales: Analyze and generate leads, summarize customer engagements, and highlight trends for PMs.
Marketing: Draft launch content from release notes.
Legal: Flag risky language, summarize new regulations.
Finance: Model ROI of roadmap options, forecast budget impact.
Risk and Resilience
What happens when AI hits limits or becomes unavailable? Inference isn’t free; costs will rise, subsidies will fade, and usage may be capped. Do you have fallback workflows, maintain manual expertise, and measure AI’s ROI beyond activity? Another reason for us to gain experience with these tools is to improve our efficiency and understand usage patterns.
The Opportunity
We already have the data to see how our system performs. The real opportunity is to aim AI at the constraints those metrics reveal, removing friction, aligning teams, and improving decision-making. If we take the time to learn the tools now, we’ll be ready to use them where they matter most.
What Now?
We already have the metrics to see how our system performs. The real opportunity is to apply AI purposefully across the full lifecycle, from ideation and design, through development, testing, deployment, and into operations and business alignment. By directing AI toward the right constraints, we eliminate friction, unify our teams around clear metrics, and elevate decision-making at every step.
Yes, AI adoption is a learning journey. We'll stumble, experiment, and iterate, but with intention, measurement, and collaboration, we can turn scattered experiments into a sustained competitive advantage. AI adoption is about transforming or improving the system itself.
AI isn't failing, it's maturing. We're on the rise of the adoption curve. Our challenge and opportunity is to build the muscle and culture to deploy AI across the lifecycle, turning today's experiments into tomorrow's engineered advantage.
For anyone still hesitant, know this: AI isn't going away. Whether it slows us down or speeds us up, we must learn to use it well, or we risk being left behind. Let's learn. Let's measure. Let's apply AI where it's most relevant and learn to understand its current benefits and limitations. There's no going back, only forward. Moving forward in our careers means gaining experience with these tools.
References
Afterburnout. (n.d.). AI promised efficiency. Instead, it’s making us work harder. Afterburnout. https://afterburnout.co/p/ai-promised-to-make-us-more-efficient
Clark, P. (2025, July). AI is improving software engineering. But it’s only one piece of the system. Substack. https://open.substack.com/pub/rethinkyourunderstanding/p/ai-is-improving-software-engineering?r=2gr71c&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false
METR. (2025, July 10). Measuring the impact of early-2025 AI on experienced open-source developer productivity. METR. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Pereira, S. (2025, August 8). It’s time for AI to meet flow: Flow engineering for AI. Steve Pereira. https://stevep.ca/its-time-for-ai-to-meet-flow/
State of DevOps Research Program. (2024). 2024 DORA report. Google Cloud / DORA. (Direct URL to the report as applicable)
Tacho, L. (2025, June). How to cut through the hype and measure AI’s real impact. Presentation at LeadDev London. https://youtu.be/qZv0YOoRLmg?si=aMes-VWyct_DEWz0
This article was originally published on August 09, 2025.
