Part 1 of 3
AI is changing how we build digital products
- Article series
- Product development
5 minutes
When AI is used to accelerate product development, traditional metrics are no longer enough. Counting the number of features delivered in a sprint doesn't tell the whole story. The focus needs to shift towards goals linked to impact, learning and decision support.
You need to understand how quickly a team can validate hypotheses, what business impact each experiment delivers, and how those insights are then turned into action.
Measuring the right things is always the foundation for creating real value.
New metrics for AI-driven development
KPIs can be used to measure activity, but also as a strategic tool for learning and prioritisation. AI makes it possible to analyse large amounts of data in real time and turn it into insights that influence everything from product development to business strategy, marketing and customer experience.
This opens up new ways of measuring real value: which initiatives create an impact, which hypotheses do we learn from, and how quickly can the organisation act on that knowledge? Metrics related to this include:
-
Validation speed: How quickly can the team test and validate a hypothesis? This can be measured as the time from an idea to the first data. Faster validation makes it possible to use resources more effectively.
-
Impact per experiment: What tangible value does each test or feature create? For example, increased conversions, reduced churn or higher revenue. AI can help isolate the impact of individual changes when several changes are made at the same time.
-
Learning per unit of resource: How much knowledge is generated relative to the time and investment required? By measuring learning, the organisation can identify which initiatives generate the most insight for the resources invested.
-
User response and behavioural change: How do changes affect user behaviour and engagement? AI can analyse large amounts of interaction data to provide valuable feedback.
These are examples of metrics that can complement traditional KPIs. They can provide a more nuanced picture of how the product is performing, helping the team make more informed decisions going forward.
Five questions for measuring AI-driven products
Before establishing new KPIs, it can be useful to reflect on the following:
-
Connection to business value
Which KPIs show directly whether our initiatives are creating tangible value? For example: increased revenue, reduced churn or higher user engagement.
-
Validation timeframes
How quickly can we measure and draw conclusions from new features? Shorter cycles enable faster learning and reduce costly investments in the wrong direction.
-
Data quality
Is our data reliable and structured enough for AI to generate meaningful insights? Poor-quality data can lead to incorrect conclusions and put the business at risk.
-
Acting on insights
Do we have processes in place for turning insights into concrete decisions? Identifying problems without acting on them creates no results.
-
Feedback loops
How do we ensure that learning from an experiment is shared and used across the organisation? Document processes, workshops and dashboards so that they are visible and accessible to teams.
From data to decisions
But collecting data is only the first step. For AI to create real value, insights need to be turned into concrete actions that influence the product's development.
Consider an e-commerce company testing a new feature in its checkout. Using AI to analyse user data in real time reveals that the feature increases conversion by 15% among new customers, while reducing it among returning customers.
Armed with this insight, the team can immediately adapt the flow for the different segments, without waiting for the next release.
Insights for digital leaders
For this to work, a few conditions need to be in place:
-
Prioritisation of experiments: You need to be able to determine which tests offer the greatest business value and run them first. In the example above, this could mean optimising the checkout flow for returning customers before testing other, less critical features.
-
Clear decision-making processes: Insights need a clear path to action. Who is responsible for acting on the data, and how are changes implemented and documented?
-
KPIs as decision-making tools: Metrics such as validation speed and business impact should actively inform decisions, rather than simply being reported in dashboards.
-
Transparency and communication: Everyone in the organisation needs to understand what is being measured and why, so that insights can be shared and used across the team.
This turns measurement from a retrospective analysis into an active tool for driving faster, better and more value-creating decisions throughout the development process.
It also enables teams to measure more than traditional delivery metrics and focus instead on impact and faster paths to decision-making. The right KPIs, clear processes and real-time insights make it possible to create value while building an organisation that continuously improves its products.
Last updated:
2026-10-01
Andreas Ronder
Digital strategist and business developer with experience in digital initiatives and product development. He connects business, users, and technology to create solutions with measurable impact.
