How do we know how satisfied our customers are with the results of our work? Can we assess—or even know—at any given time whether an investment (such as a new feature) will be worthwhile? How quickly can we, as a company, or our teams learn from the customer feedback we receive and adjust our work accordingly?
That is exactly what our Xperts from the Project Management team discuss in the following blog post.
These questions are particularly important for us as product owners and project managers, as they provide genuine insight into a product’s value. To answer them, we must look beyond the project itself and take a holistic view of our customers’ problems. For example, by measuring user satisfaction and incorporating it into our decisions, we ensure the long-term success of our product. By regularly analyzing these metrics, we quickly receive feedback on the value of our service or product.
In agile software development, we can then incorporate opportunities for improvement into future product releases and achieve systematic improvement in our performance. An additional positive effect of this approach is the reduction of waste—that is, wasted effort. Instead of clinging to implementations that add no value, we can focus more quickly on the useful measures. This not only has a positive impact on users but also on the profit our company can generate as a result.
Behind this approach lies a concept for measuring success that we would like to explain in more detail in this article: Evidence-Based Management.
Whether you already have experience as a product owner or are responsible for making decisions in another capacity within the company, this approach helps you make well-reasoned decisions at work and measure their success.
Evidence-Based Management (EBM)
Evidence-Based Management (EBM) is a school of thought based on two principles: first, on experience with the Scrum Framework and, second, on scientific methods from the field of medicine, particularly the approach based on empirical evidence. Let's first take a closer look at the meaning of the term "evidence-based management.".
- Management
- Management is leadership and guidance.
- This is achieved through decisions.
- Every decision has an impact on a company's success—it can be either positive or negative.
- Evidence-Based
- Decisions are based on an evidence-based approach.
- The better the information on which a decision is based, the easier it is to avoid decisions that have a negative impact and to promote those that have a positive impact.
- Measuring performance metrics helps establish a foundation of information for sound decision-making as quickly as possible.
As an agile organization, we base our success on delivering value (referred to as „Value“ in the EBM Guide) and continuously improving customer satisfaction. Typically, we operate in an environment where there is a certain degree of uncertainty regarding the conditions for success. But how can we still ensure that we’re on the right track?
Let's say, for example, that we already distribute an app called „Hello World“ through the Play Store that offers simultaneous translations in all languages. Our goal now is to become the app with the most satisfied users and also the market leader.
With EBM, we have a methodology that helps us gain insights for precisely these kinds of situations and use them to make well-informed decisions. This means we no longer have to rely solely on the opinions of stakeholders and our gut feelings. Instead, we use metrics, key performance indicators, and customer feedback to chart our course.
EBM Workflow
In this process, we follow the well-known agile „inspect and adapt“ approach. Even with a concept like EBM, we move toward our goal through iterations. We use what are known as experiment loops: First, we determine which hypothesis to test next. Then we plan an experiment of manageable scope and carry it out. An experiment can be anything from simple text adjustments in an online store to an A/B test. After implementation, we measure the results (e.g., tracking the traffic to variants A and B) and thereby test our hypothesis. Depending on the outcome, the assumed solution may be pursued further, modified, or possibly even discarded after evaluating the results. The knowledge gained is then incorporated into the next set of hypotheses. Through these recurring feedback loops, we steadily move closer to our goal.
A brief summary of the experimental loop:
- Formulation of Hypotheses
- Experiment & Measurement
- Rating
- Adjustment
To return to our example:
- How can we get potential new customers excited about our „Hello World“ app?
- What added value does the app offer to existing customers, and what growth potential does it hold?
To do this, we first need to understand the status quo. In our case, we assume that customer satisfaction, the number of new registrations, and the number of active users are the most important metrics for assessing whether we’ve achieved our goals. Of course, these metrics must also be measured—for example, by implementing a tracking tool or providing interfaces to analyze existing data.
In this first round of experiments, we now want to improve new user registrations. According to our analysis, this is where the greatest potential appears to lie. The goal is to test the hypothesis that an accessible landing page leads to an increase in new user registrations. During the implementation phase, this assumption can already be supported by initial tests. After the new page is launched, we will analyze metrics such as registration numbers and the bounce rate on the landing page to confirm or refute whether this implementation achieves the desired results.
Key Value Areas
Now that we’ve explained the basic approach, let’s take a closer look at what should be included in the measurements and how they should be conducted so that they truly support our decision-making. EBM is primarily about identifying what value has been created. It is therefore not about achieving the highest possible throughput in software releases (output), but rather about the impact these releases have on users (outcome). The following questions play an important role in this context:
- What results were achieved among users, and what benefits did they gain from it?
- What new product features, for example, are now available to you?
However, making outcomes measurable is significantly more difficult than measuring outputs. Yet the only way to determine whether we have been successful with our activities and outputs is through such measurements. The EBM Guide offers various perspectives here for examining value. These are the Key Value Areas (abbreviated as KVAs):
- Current Value
- Unrealized Value
- Time to Market
- Ability to Innovate
Current Value
Current Value, or CV for short, focuses on the value a product or service delivers to customers at this very moment. To determine the current value, the following metrics, among others, should be considered:
- How satisfied are customers with the product right now?
- How do certain features work?
- What is the ratio of costs to revenue?
- But also: How satisfied and engaged are employees right now?
If we have a good understanding of a product's current value, we can quickly identify areas for improvement.
For our sample app „Hello World,“ one of the goals is to become the app with the highest number of satisfied customers. Customer satisfaction falls under the category of Current Value and should therefore definitely be measured. These measurements might also reveal that further improvements aren’t necessary because, for example, satisfaction levels are already very high.
Unrealized Value
Unrealized Value, or UV for short, focuses primarily on the value behind future revenue that has not yet been realized—that is, the potential that would exist if the needs of all potential customers were met. The following questions help us in this regard:
- Is there additional value we can create with our product and in our market?
- What is our market share?
- What is the satisfaction gap among customers?
- How much effort does it take to reach potential customers?
The second goal for the further development of our „Hello World“ app is to become the market leader. To achieve this, measuring unrealized value is particularly important. If we know which customer needs must be met to increase market share, the app can be further developed in these areas, and the success of these implementations can then be measured. However, it is equally conceivable that further investment to increase market share would no longer be worthwhile, as the costs would be too high.
Time to Market
Time to Market, or T2M for short, measures how quickly a company can deliver new capabilities, services, or products. This is primarily because this key value factor influences the time it takes to generate value.
- How quickly can we learn from new information and adapt ourselves or our product?
- How quickly can we test new ideas?
If we reduce the time to customer, we can also more quickly influence the current value (CV) of our products. Possible metrics here include release frequency, the lead time from formulating a hypothesis to delivering actual value to the customer, or the time it takes to fix a defect.
To increase customer satisfaction with our app, it may be helpful to also improve the turnaround time or release frequency. Specifically, this would mean, for example, reducing the number of features in product releases in order to deliver value more quickly to a subset of our customers.
Ability to Innovate
Last but not least, it is also advisable to consider an organization’s ability to deliver innovative solutions. EBM refers to this as the “Ability to Innovate,” or A2I for short. The priority should be to enhance this ability in order to effectively improve the value of products.
Here's an example to illustrate this:
- If a company has flawed decision-making processes, struggles to improve product quality, or if customers find it very difficult to install its products, this reduces the company’s capacity for innovation.
- The focus is then on maintaining our products or removing obstacles. There is no budget or time for new initiatives.
- The goal should therefore be to establish processes and capabilities that foster innovation and reduce barriers to customer adoption.
- Possible metrics here include technical debt, the innovation rate, or the installed version index.
For the „Hello World“ app, it can therefore also be helpful to consider this range of values. Reducing systemic barriers or eliminating features that add little value increases the ability to respond more quickly to customer needs regarding CV or UV and, above all, frees up more capacity to do so.
Why is EBM important to us?
As product owners and project managers, the evidence-based approach is our method of choice. Through continuous measurement, we gain insights that increase the value of every further development. This means we know we’re working on the right things—or we quickly receive feedback if that’s not the case. We do not rely on assumptions; instead, we verify them through measurements and data analysis. Especially when starting new projects, this approach helps us quickly understand customer needs and identify potential opportunities. In projects where we’ve been involved for some time, the use of EBM and the associated measurement of success is just as much of a motivator for us as it is for the team. At the same time, our customers also benefit from this focus on value creation.
