How do we know how satisfied our clients are with the results of our work? Can we assess or even know at any time whether an investment (e.g. a new feature) will be worthwhile? How quickly can we as a company or our teams learn from the customer feedback we receive and adapt our work accordingly?

Our Xperts from the project management team address this topic in this blog post.

These questions are particularly important for us as product owners and project managers because they provide real information about the value of a product. To answer them, we have to look beyond the actual project and consider the problems of our customers holistically. As an example, by measuring user satisfaction and including it in our decisions, we ensure the long-term success of our product. By regularly evaluating such metrics, we quickly get feedback on the value of our service or product.

Subsequently, in agile software development, we can introduce improvements into the future deliveries of the product and achieve a systematic performance improvement. An additional positive effect of this approach is the reduction of waste, i.e. wasted work. Instead of holding on to implementations that do not add value, we can focus more quickly on the useful measures. This not only has a positive impact on the beneficiaries, but also on the profit that our company can achieve.

Behind this approach is a concept for measuring success, which we would like to explain in more detail in this article: Evidence-Based Management.

Regardless of whether you already have experience as a product owner or are responsible for decisions elsewhere in the company: this approach supports you in making transparent decisions at work and making their success measurable.

Evidence-Based Management (EBM)

Evidence-Based Management (EBM) is a way of thinking based on two foundations: experience with the scrum framework and, scientific methods from medicine, particularly the approach of empirical evidence. Let us first take a closer look at the word Evidence-Based Management in its meaning.

1. Management

   • Management is leadership and guidance.
   • This is done through decisions.
   • Every decision has an impact on the success of the company - which can be positive or negative.

2. Evidence-Based

   • An evidence-based approach forms the basis for decisions.
   • The better the information basis of a decision, the easier decisions with a negative influence can be avoided and those with a positive influence can be promoted.
   • Measuring KPIs (key performance indicators) helps to attain an information basis for good decisions as quickly as possible.

As an agile organisation, we base our success on delivering value and continuously increasing customer satisfaction. Typically, we operate here in an environment of some uncertainty about the conditions for success. But how can we still ensure that we are on the right track?

Let's say, for example, that we already distribute an app "Hello World" via the Play Store, which offers simultaneous translations in all languages. Our goal is to become the app with the most satisfied users and also the market leader.

With EBM, we have an approach that helps us gain knowledge for exactly such cases and make informed decisions based on it. This means we no longer rely solely on stakeholder opinion and gut feeling. Instead, we use metrics, KPIs and customer feedback to determine our path.

EBM Workflow

Here we follow the well-known agile approach of "inspect and adapt". Even with a concept like EBM, we move towards our goal with the help of iterations. We use experiment loops: First we determine which hypothesis we will pursue next. Then we design an experiment of manageable size and carry it out. An experiment can be anything from simple text adjustments in an online shop to an A/B test. After implementation, we measure the results (e.g. measuring the number of hits of variant A and B) and thus test our hypothesis. After evaluating the results, it may be that the assumed solution is pursued further, changed or possibly even rejected, depending on the results. The knowledge gained in turn flows into the next hypotheses. Through these recurring feedback loops, we are constantly approaching our goal.

The experiment loop in brief:

• Hypothesising
• Experiment & measurement
• Evaluation
• Adaptation

To come back to our example:

• How can we inspire potential new customers for our app "Hello World"?
• What added value does the app offer existing customers and what development potential is there?

To do this, we first need to know the status quo. In our case, we assume that customer satisfaction, the number of new registrations and the number of active users are the most important parameters to be able to make statements about the achievement of our goals. Of course, these values must also be measured. For example, by implementing a tracking tool or providing interfaces for the evaluation of existing data.

In a first experimental loop, we now want to improve the new registrations. According to our evaluation, this seems to be where the most potential is hidden. The hypothesis that an accessible entry page leads to an increase in new registrations will be tested. During the implementation phase, this assumption can already be substantiated with initial tests. After the release of the new page, the registration figures and the bounce rate on the entry page are evaluated, among other things, to confirm or refute that this implementation leads to the desired success.

Key Value Areas

Now that the basic procedure has been explained, let's take a closer look at how and what should be part of the measurements so that they really support our decisions. EBM is primarily about recognising what added value has been created. It is not about the highest possible throughput of software releases (output), but about the impact that these releases have on the users (outcome). The following questions play an important role:

• What result was achieved for the users and what added value do they get from it?
• For example, what new product features are now available to them?

However, making outcomes measurable is much more difficult than measuring outputs. Whether we have been successful with our activities and outputs, however, can only be answered by such measurements. Here, the EBM Guide offers different perspectives to look at value. These are the Key Value Areas (abbreviation KVAs):

• Current Value
• Unrealized Value
• Time To Market
• Ability To Innovate

Current Value

Current Value, or CV for short, focuses on what value a product or service is delivering to customers at that moment. To recognise the current value, the following parameters, among others, should be considered:

• How satisfied are the current customers of the product?
• How is the use of certain features?
• What is the cost-to-sales ratio?
• But also: How satisfied and committed are the employees right now?

If we have a good picture of the current value of a product, we can also quickly recognise where there is room for optimisation.

For our example app "Hello World", one of the goals is to become the app with the highest number of satisfied customers. Customer satisfaction falls into the area of current value and should therefore be measured. These measurements could also show that further improvements are not necessary because, for example, a very high level of satisfaction already exists.

Unrealized Value

Unrealised Value, or UV for short, is primarily concerned with the value behind future implementations that have not yet been realised. In other words, what the potential would be if the needs of all possible customers were met. The following questions help us here:

• Is there additional value we can create with our product and in our market?
• What is our market share?
• What is the customer satisfaction gap?
• How big is the effort to reach the potential customers?

The second goal for the further development of our app "Hello World" is to become the market leader. Measurements of the unrealised value are particularly important for this. If it is known which customer needs would have to be met in order to increase the market share, the app can be further developed in these areas and the success of the implementations can then be measured. However, it is just as conceivable that further investment to increase market share is no longer worthwhile because the costs would be too high.

Time To Market

Time To Market, or T2M, looks at how quickly a company can deliver new capabilities, services or products. This is mainly because this key value has an impact on the time we need to generate value.

• How quickly can we learn from new information and adapt ourselves or our product?
• How quickly can we test new ideas?

If we reduce the time to customer, then we can also influence the current value (CV) of our products more quickly. Possible measurements are the release frequency, the lead time from the creation of a hypothesis to the actual benefit for the customer, or the time until a defect is fixed.

In order to increase customer satisfaction for our app, it can be helpful to also improve the lead time or release frequency. In concrete terms, this would mean, for example, reducing the number of features in product releases in order to deliver added value more quickly to some of our customers.

Ability To Innovate

Last but not least, it is also advisable to look at an organisation's ability to deliver innovative solutions. EBM calls this the Ability To Innovate, or A2I. The priority should be to increase this ability, thereby effectively improving the work on the value of products.

An example to illustrate this further:

• If poor decision-making processes exist within a company, if product quality can only be improved with difficulty or if customers can only install the products with great effort, this reduces the capacity for innovation.
• The focus is then on maintaining our products or removing obstacles. Budget and time for new features is not available.
• The goal should therefore be to set up processes and capabilities that favour of innovation and to reduce hurdles for use by customers.
• Possible metrics here are the technical debt, the innovation rate or the index of installed versions.

For the "Hello World" app, it can therefore also be beneficial to look at this value area. Reducing systemic barriers or removing low-value features increases the ability to respond more quickly to customer needs in terms of CV or UV and, most importantly, to create more capacity to do so.

Why is EBM important for us?

As product owners and project leads, the evidence-based approach is the tool of choice in our work. Through continuous measurement, we gain insights that increase the value of any further development. So we know we are working on the right things or get feedback quickly if that is not the case. We don't rely on assumptions, but check them by measuring and evaluating the data. Especially when starting in new projects, this procedure helps us to quickly understand the customer's needs and also to identify possible potential. In projects we have been involved in for a long time, the use of EBM and the associated measurement of success is just as great a motivator for us as it is for the team. Our clients additionally benefit from this focus on generating value.