Hypothesis-driven Development 

An IT project typically involves a few definite stages. A team has an idea of what would be valuable to the end-user. The same team designs proposed solutions for the product implements the ideas, tests the ideas, deploys them, and eventually, the final product reaches the customer. Ideally, the customer likes the product, uses it, and benefits from it. However, the product team does not immediately have a clear view of the customer experience. 

This is where Hypothesis-driven Development (HDD) comes into the picture. 

Hypothesis-driven development is about solving the right problem at the right time. It ensures that understanding of the problem is verified before actual work on the solution begins.  

What is Hypothesis-driven Development? 

Hypothesis-driven Development (HDD) involves the development of new products and services through iterative experiments on hypotheses. The results of the experiments help to decide whether an expected outcome will be achieved. The steps are repeated till a desirable outcome is reached or the idea is deemed to not be viable anymore. 

HDD advocates experimentation over detailed planning, client feedback over instinct, and iterative design over conventional “big design up front” deliveries.  

With HDD, solutions are viewed as hypotheses, which can be theories about an area relevant to the problem statement. Hypotheses can be developed about areas like the market being targeted, the performance of a business model, the performance of code or the customer’s preferred way of interacting with the system. 

As the software development industry has matured, teams now have the opportunity to leverage capabilities like Continuous Design and Delivery to maximize their potential to learn quickly what works and what does not. By taking an experimental approach to information discovery, teams can more rapidly test their solutions against the problems they have identified in the products or services they are attempting to build. 

What does Hypothesis-driven Development Hope to Achieve?

The goal of HDD is to improve the efficacy of teams, by ensuring they solve correctly-identified problems, rather than continually building low-impact solutions. In HDD, teams focus on user outcomes, rather than their outputs. 

What led to the Popularity of Hypothesis-driven Development?

Since every software company began competing to deliver better software, faster, development teams faced two major challenges: 

1. Delivering functionality fast, before the features became outdated 

2. Developing features for a more selective user-base, that demanded a better experience in every possible way. 

3. Multiple competing options for a user to switch loyalties to 

In today’s fast-paced world, teams need to be ready to adapt continuously and quickly, even if it may require deviating far from the originally chosen path.  

The conventional approach of researching and documenting user requirements, developing a solution “to spec” and deploying it months or years later can no longer be considered suitable. 

Requirements add value when teams are executing a well-known or understood phase of an initiative and can leverage previously used practices to achieve the outcome. However, when teams are in an exploratory phase of the product’s development they should use hypotheses to validate ideas first. 

The adoption of this ‘deliver early and fail fast’ approach has become so commonplace, that the word ‘pivot’ is commonly recognized to mean the act of rapidly changing from one plan to another. 

Why Is HDD Important? 

Handing teams a set of business requirements does not allow the developers the freedom of innovation. It implies that the business team is in charge of the problem as well as the solution. The development team merely implements what they are told.  

However, when developing a new product or feature, the entire team should be encouraged to share their insights on the problem and potential solutions. A development team that merely takes orders from a business owner is not utilizing the full potential, experience, and competency that a cross-functional multi-disciplined team offers. 

Key Steps in Product Development 

It is crucial to lay down the foundational steps in HDD product development. The following four steps are integral to the process: 

1. Finding the Right Problem 

Teams can make use of the ‘persona hypothesis’ and ‘JTBD/Job To Be Done hypothesis’ to ensure they have identified the right problem. The ‘persona hypothesis’ focuses on the persona, which is a humanized and detailed view of who the user is, and what motivates their actions. To aid with creating the persona, the team usually follows an ‘interview guide’ to ensure they can gather sufficient information about the users they are solving problems for. 

The second type of hypothesis that can aid in identifying the right problem is the ‘JTBD hypothesis’. This hypothesis tries to understand the tasks (or jobs) that customers are trying to achieve in their lives. This framework is foundational for understanding what motivates customers and why customers behave the way they do. 

2. Identifying the Demand for the Solution

This area is pivotal and easy to assess: have you validated that you have something for your audience that they prefer over the alternatives. Teams use the ‘demand value hypothesis’ to aid them with identifying demand. The ‘demand/value hypothesis’ is a hypothesis that contains the exact value that would be given to potential clients. 

3. Finding the Right Solution to the Problem

By making use of the ‘usability hypothesis’, teams can assess whether they have found the right solution to the problem. The ‘usability hypothesis’ helps to determine how easy-to-use the designed solutions are. Simpler solutions are more likely to be adopted by more users.

4. Achieving Continuous Delivery while Deploying the Solution 

By delivering enhanced products to users quickly, teams have the opportunity to learn faster. Teams make use of the ‘functional hypothesis’ in continuous delivery pipelines, to make sure the delivered products provide the expected results. 

How Hypothesis-driven Development Works

HDD is a scientific approach to product development. In HDD, teams make observations about customers, to come up with hypotheses or explanations, which they believe align with their customers’ views. Each hypothesis is then tested by predicting an outcome based on it – if the outcome is aligned with the prediction, the hypothesis can be assumed to be correct. 

The key outcome of this experiment-based approach is a better understanding of the system and desired outcomes. If the results of the hypothesis are not as expected, deductions can be made about how to refine the hypothesis further. 

The experiment at the heart of every iteration of HDD must have a quantifiable conclusion and must contribute to gaining more information about the end-users usage of the product. For each experiment, the following steps must take place: 

1. Make observations about the user 

2. Define a hypothesis 

3. Define an experiment to assess the hypothesis 

4. Decide upon success criteria (e.g., a 30% increase in successful transactions) 

5. Conduct the experiment 

6. Evaluate the results of the experiment 

7. Accept or reject the hypothesis 

8. If required, design and assess a new hypothesis 

Product development teams can leverage Continuous Design and Delivery to deliver changes related to their hypothesis for real-time testing of their theories.  

How is a Good Hypothesis Framed? 

There is a framework that should be followed to define a clear hypothesis. 

The framework supports communication and collaboration between developers, testers as well as non-technical stakeholders in the product. 

It is of the following format: 

We believe <this capability> 

‘This capability defines the functionality that is being developed. The success of the feature is used to test the hypothesis 

Will result in <this outcome> 

‘This outcome’ defines the clear objective for the development of the capability/feature. Teams identify a tangible goal to measure the usefulness of the feature being developed. 

We will have the confidence to proceed when <we see a measurable signal> 

The measurable signal refers to the key metrics (which can be qualitative or quantitative) that can be measured to provide evidence that the experiment has succeeded. 

An effective hypothesis is fundamental to data-driven optimization. Hypotheses are used to convert data collected about customers into actionable insights. Every hypothesis is a theory that must be assessed; each idea that is proven to either hold true or false, confirms notions about customer expectations and behaviors, and drives the iterative optimization process.  

For example, an e-retail website could have a high rate of abandonment in the purchase flow. The hypothesis could be that links in the flow are distracting potential customers. The experiment would be to remove them. Should there be an improvement in the number of completed purchases, it would confirm the hypothesis. This would give a confirmed improved understanding of the retail website’s customers and their behavioral trends. This improved insight would help decide what could be optimized next, why, and how results could be measured. 

The following is how the same hypothesis could be defined in an ideal user story: 

 We believe that removing irrelevant links from the purchase page 

This Will result in improved customer conversion 

We will have the confidence to proceed when we see a 20% increase in customers who checkout their shopping cart with successful payment. 

Following the framework is an easy way to ensure you have thought of every aspect of the problem as well as the proposed solution before starting actual work on the project. The framework also ensures that only meaningful features are developed, by quantifying the benefits of these features. 

Best Practices in Hypothesis-driven Development 

The following are the best standards that will help teams ensure they implement HDD well: 

1. Gather Sufficient Data 

Data is what marks the difference between a well-formed hypothesis and a guess. To create a meaningful hypothesis, a business intelligence report is greatly helpful. By monitoring customer behavioral patterns using techniques like web analytics, and indirect sources like competitor overviews, a valuable profile about customers can be created. 

2. Create a Testable Hypothesis 

To adequately test the hypothesis, there need to be well-defined metrics with clear criteria for success and failure. For example, the hypothesis that removing unnecessary navigation links from the checkout page will ensure customers’ complete transactions can easily be assessed for correctness. The change in the company’s revenue will indicate whether or not the hypothesis was correctly defined. 

3. Measure the Experiment Results as per the Need 

The threshold used for determining success depends on the business and context. Not every company has the user sample size of Amazon or Google to run statistically significant experiments in a short period. Limits need to be defined by the organization to determine acceptable evidence thresholds that will allow the team to advance to the next step. 

For example, while building a vehicle, the experiments will have a high threshold for statistical significance. However, if the experiment is to decide between two different flows intended to help increase user sign-up, a lower significance threshold can be acceptable. 

4. State the Assumptions 

Clearly and visibly state any assumptions made about the hypothesis, to create a feedback loop for the team. This allows the team to provide further input, debate, and understand the circumstance of the test. The assumptions must make sense from a technical and business perspective. 

5. Use Insights to Drive Learning 

Hypothesis-driven experimentation gives comprehensive insights into customer behavior. These insights lead to additional questions about customers and the product experience and lead to an iterative learning process. 

The learning process follows this iterative pattern: 

• Collect data about the product domain, customers, and use the knowledge gained to formulate questions 
• Design a hypothesis based on the insights gained 
• Create and implement a campaign derived from the hypothesis 
• Inspect the results to determine whether the hypothesis is valid or not 
• Document the conclusions  

• Use the conclusions to formulate additional questions 
• Connect the results to the problem being solved 

To ensure optimum learning, begin the entire process with a problem statement, and not a preferred solution. Should the solution fail to deliver the expected result, use the problem statement to analyze new potential solutions and repeat the process? This practice ensures focus on the problems being solved. 

What HDD has in Common With Strategies Like Design Thinking and Lean Startup 

Within large companies, innovation teams commonly use strategies like Design Thinking, Lean Startup, or Agile. Each of these strategies defines only one aspect of the product development process. However, each of these strategies does have certain key principles in common with HDD, which are highlighted below: 

1. Observe Humans to Learn 

2. The cost of development can be kept low if solutions are designed after observing client behaviour, and then iterating. HDD reinforces the “problem-first” mentality by first observing the target audience for the problem being solved. 

3. Focus on Client Actions 

4. This helps to prioritise which areas of the problem need to be targeted first. HDD focuses on the prioritisation of the problems being solved. 

5. Work Fast, but Keep the Blast Radius Minimal 

Even though continuous delivery is a crucial tactic, it cannot be at the expense of correctness. HDD does not promote reduced standards of work, even during the experimentation phase. 

    6.  Minimise Waste 

By focusing on the core of the problem being solved, product development teams ensure they do not waste time/money/resources on features that would not be used by clients.  

Teams can refine the framework they follow as per their needs, but HDD provides them with a foundation for the best practices used across each of these popular strategies.  

Identify How Your Team Has Successfully Implemented Hypothesis-driven Development 

It is necessary to have a capable monitoring and evaluation framework set up when using an experimental approach to software development to quantify the impact of efforts. These results are then used to generate feedback for the team. The learning that is gained during HDD can be the primary measure of progress for work done. 

Ideally, an iteration of HDD is not considered complete till there is a measurable value of what was delivered, that is, data to validate the hypothesis. 

Teams can ensure they have succeeded at implementing HDD by measuring what makes a difference. 

• Vanity metrics are statistics that look good on the surface but do not necessarily provide meaningful business insights. These measurements are usually superficial, easily quantifiable, and sometimes, easily manipulated. Examples could include metrics on the number of social media followers or the number of visits to a promotional advertisement for a product. These metrics do not provide insights about what led to these numbers. 

They also do not reflect how these numbers can be achieved again or how they can be improved. 

• Actionable metrics are the metrics that have real significance. They can provide insights that help make decisions about what can be done next to achieve specific business goals. 

Distinguishing Between Actionable and Vanity Metrics: 

Metrics that run deeper than vanity metrics do not necessarily have to be actionable. For example, revenue measures a business’s performance, rather than the vanity metric of how many visitors the business website has. However, merely knowing the statistical changes does not indicate what causes these changes. If, for example, revenue increased but the reason for the increase was not identified, the business cannot repeat the actions made!  

Not just that, they do not have the means to further amplify the improvement! If, however, the revenue was measured before and after a noted change that affected a target set of users, then the business has an actionable metric at hand. They can then go ahead and employ the approach of hypothesis-driven development to try experiments and understand the best way to add value. 

Vanity metrics and actionable metrics can be measured for several actions.  

The first, and in today’s world, most significant, is website behaviour. Apart from this, teams can measure developer or team productivity, alongside bugs reported, and customer usage metrics. Knowing the count of lines of code written does not add value if the code is problematic and requires rework in the next development cycle. It also does not matter if the team is working fast but on a solution that no one will use. 

The relevant point is to work with the team to identify statistics that will provide value to the product development. The metrics should be shared with the team before, during and after any significant changes done to the product. 

Benefits of Hypothesis-driven Development

HDD is being adopted rapidly because of the several benefits it offers. 

1. HDD helps product teams prioritise the development of features, 

Teams can understand how the features are connected to the business goals. By tracking metrics before and after product deliveries, clear insights can be learned and worked upon for further growth. As long as teams keep in mind the end-users pain points, which can only be assessed by experimentation, they will deliver incrementally improved products. 

2. HDD enables tracking the desired and real outcomes of the development process.  

Each experiment is formulated to define the expected outcomes. These can be used to understand how the team’s development strategy can be revised for maximum gain. 

3. HDD is Cost-Effective 

It is more cost-effective to test the prototype before delivering the completed product to production.  

By constantly inquiring about the customer’s needs during the development process, the team will benefit from a feedback loop about the product’s performance, during the development phase itself. This will lead to minimal rework on the released product. 

4. Discover and Prioritize the Potential Benefits of Hypothesis-Driven Experiments 

Teams can easily understand the business benefits of changes they make. They can use these numbers to refine a company’s roadmap 

5. Establish a Common Language for Ideation and Research 

By following a framework for designing hypotheses, teams benefit from a standard way to define potential ideas. This enables development and research teams to collaborate and communicate more transparently. 

6. Choose Problems that are Aligned with the Company’s Challenges 

By reviewing the impact of experiments, teams ensure that while working on smaller goals, they stay aligned with the company’s end-state vision. 

7. Gain from Planning and Budgeting insights 

Other than measuring the outcome of experiments, teams can also measure the number of hypotheses being tested, the cost of these experiments and the time taken for each experiment. Advanced analysis can also help teams understand their development velocity. The measurable nature of the hypothesis-driven approach makes it simple for an organization to understand how to plan, budget for and undertake a hypothesis-driven approach. 

8. Quantify and Manage Risk 

Teams can understand how many hypotheses they need to validate or invalidate to determine whether they should make further investments in their product. Stakeholders can monitor the previously opaque process of risk evaluation, by evaluating the quality and quantity of hypotheses tested. 

9. Explicitly Documented Learning 

The hypothesis-driven approach for product development has an added benefit in that it explicitly captures lessons learned about the organization’s target market, customers, competitors, and products. A hypothesis-driven approach requires the thorough documentation and capture of each hypothesis, the details of the experiment as well as the results. This data becomes an invaluable store of information for the organization. 

10. HDD leads to better Technical Debt Management 

By reworking the product during the development phase, there are likely to be no surprises in customer reviews of the final product. This will ensure the technical debt is minimal and reduces overall development costs. 

Success Story of Hypothesis-driven Development 

A great example of a company that successfully used HDD while developing its main offering is Dropbox, the popular remote file-sharing service. Today, Dropbox is used by over 45 million users. However, when they initially launched, they had several competitors in the same domain. The key point was that each of these options was not well made. 

Dropbox’s initial hypothesis was that if they offered a well-executed file system, then people would be willing to pay to use it. 

When they began their experimentation, they used the persona hypothesis to define their ideal target user base. The persona they devised was of a technically-aware person, who would work in the software world. 

The solution they designed was centered to be ideal for the person they had devised. 

The Job To Be Done was sharing files between groups of people. The existing solutions at the time were manual, like file systems that needed to be backed up by an engineer. 

Dropbox’s value hypothesis was that a transparent remote file system would be adopted by several users, provided it was well-made. Dropbox needed to identify the demand for their solution. However, they did not have the resources at the time to create the product and have it validated by multiple users for the first round of experimentation of their hypothesis. They circumvented this blocker by releasing a video, which detailed their idea. The video was published online and advertised on development forums. 

The interest in their proposal was significant, enough to help them validate their proposed solution design, 

Conclusion 

Hypothesis-driven development draws its strength from the fact that the real world is complex, in a state of constant flux, and can sometimes be confusing. Consistent hypothesis-driven experimentation helps programs make a significant and beneficial impact on a company’s company objectives. Using data that is strongly coupled to the company’s vision ensures that focus is given to areas of significance for customers, rather than points that seem significant to a specific group of product managers.  

Remember, in the scientific world of development, data and facts will always trump intuition.