Superpower 5: Experimentation and testing

Back to PMF diagnostic

DOWNLOAD PDF

Why is this valuable?

Whether you have launched your product or you are starting to explore new ideas or features for a longstanding product, you are usually in a space of maximum uncertainty. You have many questions about what elements of the solution are working and why. In these instances, it is important to divide your solution, and acquisition and retention efforts, into what is known and what is unknown, to ensure ongoing improvement towards product-market fit. Rather than guess at the answer to these questions, we suggest that you exploit experiments. Experiments are a powerful tool for startups. They encourage teams to pause and test assumptions before barreling forward with a complete solution that might not work.

Experiments help startups articulate what hypotheses need to be proven and to approach them in a rigorous, cost-effective way. The team decides what questions need to be answered about how users experience the product, and then designs a low-fidelity model (or two) to share with users for observation and feedback. For instance, you could test customers’ perceived value of your product based on various taglines, or their willingness to buy and/or use your product depending on how the offer is articulated. You could also test the right time and place to encourage users to refer others. Such tests allow you to make informed decisions so that you invest your time and effort where it makes sense to grow the business. Experimentation is not just a tool, but a mindset. All team members need to buy into the method of experimentation. 

How do you test ideas?

The first step is to identify your key hypotheses (our chapter on value propositions can help), and decide which ones to test. Next, decide which experiment method to deploy. A minimum viable product (MVP), for example, is a type of experiment that allows you to observe how potential users might respond to your initial, early-stage offering. It is important to identify your riskiest assumption/hypothesis (i.e., what users want, how the design should work, what messaging to use), and then find the easiest, fastest possible way to test those assumptions. Then, use the results of the experiment to correct the course and further progress to product-market fit. 

The list below can help you decide what type of experiments to conduct, and how to create a framework to collect valuable results. Costs vary depending on the type of experiment you select.

When should you do it?

You can start testing and measuring your solution as soon as it’s launched, and provided you’re testing just one variable at a time (to ensure you can attribute any differences in results to your experiment), this process can occur on an ongoing basis. This helps ensure continual product improvement. 

We believe you should make experimentation part of your startup culture. Instead of relying on assumptions, use experiments to generate data (see our chapter on data to learn how to collect and analyze it), to guide your decision-making when it comes to understanding what to do, when and how. 

That said, adopting an experimental mindset and embedding it into the company’s processes won’t be easy. It requires buy-in from the entire team, and realization that your great ideas are just hypotheses that can be validated (or, more likely, invalidated). Experimentation requires a lot of rigor, and time to properly design the experiments, run them, and extract and document learnings over time.

What kind of expertise is required?

Designing lean experiments requires no additional expertise beyond that which is already on your team. Your CTO, product person or anyone on your tech team who has basic data analysis skills can get cracking. If no one in your team can do this, it indicates an opportunity for someone on your team to upskill.

Getting started: Designing lean experiments 

Experiments are most often used to test either marketing or product concepts. For example: marketing experiments help ensure that your messaging resonates with customers and that you’re using the right channels. It can also aid in quantifying the expected conversion rates against cost, while product experiments will help you confirm that customers will see value in your proposed additions or changes, and that they understand how to use your product.

Such experiments can either be qualitative or quantitative. For instance, qualitative marketing tests could include showing customers your marketing materials and asking for feedback re: what is appealing to them and why. For quantitative marketing tests, such as A/B testing, you could test two marketing campaigns targeted toward the same users, to see which one converts customers more reliably using statistical tests as a guide. 

Here are some common steps you can take to ensure your experiments are well-designed:

1. Describe the hypothesis that you want to test. Indicate how critical it is for your business model.
2. Define the test you are going to run to verify whether your hypothesis is valid or need to be revised. Indicate how costly and reliable this test would be.
3. Define what data you are going to measure. Indicate how long it would take.
4. And finally, define a target threshold that will indicate whether you can validate or invalidate your hypothesis.

When deciding how to execute your experiment, your approach will depend on factors like the level of detail you need, the resources you have available, how far along in your product development process you are, etc. Approaches will vary based on how closely your test represents the real product (known as fidelity), and how much the customer can interact with the MVP relative to a live product (interactivity). A few product experiments approaches, in order from low to high fidelity/interactivity, include: 

• Sketches: Sketches have the lowest fidelity and lowest interactivity, but are extremely low-cost and easy to rapidly iterate. They can be a useful way to test marketing concepts or user experience features. Pen & paper is a helpful tool for sketches. 
• Wireframes: A wireframe is a visual guide that illustrates the skeletal framework of a product, typically a website or mobile app. Wireframes offer improved fidelity over sketches as you can give a sense of the product components and how they might be arranged. They can help startups make decisions about how to organize a home page and iterate quickly without spending time on a full design.
• Mockups: Mockups are the next level up in fidelity, and look much more like the final product than a wireframe. Mockups often include visual design details, images, etc., instead of placeholders as in the wireframe. For example, they can help designers decide which icons most resonate with users. InVision and Figma are helpful tools for mockups.
• Interactive prototypes: Prototypes are not quite fully functional, but they provide a level of interaction beyond a clickable mockup. They give users a fairly complete experience of the product so that the team can observe it and receive feedback.
• Concierge MVPs: These allow you to test your product or service live using manual workarounds or hacks, instead of a full-blown backend. Using a “concierge” MVP involves doing everything for early customers by hand, not automated by tech, and works best for services that require a lot of interaction and input from customers. Zapier is a helpful tool to enable some automations.
• Live product: A live product has the highest fidelity possible, so it can help you generate feedback on things like how the product looks and behaves in different contexts that you would not get from lower fidelity options.

A qualitative experiment in practice: Interviews for product MVPs

Once you have an MVP, take a structured approach to testing it with users. We suggest the following:

1. Develop a test script for the entire conversation, including the steps you want the user to take. Test it on your team members first. An MVP test will likely take 1-2 hours per person.
2. Start by spending a few minutes getting to know the user in question, and setting expectations for the experience. Make sure they know you want them to give honest feedback, even if it is negative. Ask how they feel about what they are using now, how it works, and their current frustrations.
3. Ask questions and observe, but don’t lead. If a customer does something curious, ask them why they took that action: “I see you did this, can you tell me why?”
4. Ask open questions, which begin with “why”, “how”, and “what”, rather than questions with yes/no responses. Write your questions in advance so that you are not distracted.
5. If the user has challenges understanding or using the product, your job is to understand the issue – not to help them – so you can keep the test as realistic as possible. After the test wraps up, you can answer questions or respond to the challenges they had, but don’t interrupt to problem solve or guide (to the extent possible).
6. Don’t forget to ask if they would like to be notified when the product comes out!
7. Finally, as a thank you – consider giving them a token gift for their time.