Testing hypotheses and running experiments can get complicated, especially if you don’t have an organized plan. There are many steps you can take to get started, but how do you know if you’re making the right decisions to produce accurate and trustworthy data? Well, you’ve come to the right place! In this post, we’ll help you get the ball rolling by listing some of the top things you should and shouldn’t be doing with A/B testing. If you find this post helpful and want to go a level deeper, we break down the process for you step-by-step in our eBook: How to Build a Strong Optimization Practice. It’s free to download here.
First, let’s jump into the DO’S.
Do include a strong, data-driven hypothesis.
Hypotheses are the engines of every experiment. They are foundational for all the next steps that lead into finalizing a testing plan, running a test, and measuring the impact. Make sure to spend time refining and sustaining your hypothesis with data.
Do have a clear objective.
When creating a testing plan, figure out how you will measure success. Pick one metric as your primary goal that clearly quantifies the impact your variations have on user behavior.
Do target a specific audience.
When targeting visitors, be as specific as possible. There is no such thing as an average user, and if your variations are defined for a specific persona, use the targeting capabilities in your testing tool to only include the relevant segments of your traffic.
Do identify the dedicated testing team.
Define the owners of every step in each test, and the responsibilities that are involved in those roles. This way, it is easier for every role to be aware of where they’re involved and to allocate time necessary in advance.
Do keep your testing plans in a centralized repository.
Have a place where you store all your experiments, from testing plans to insights. You can use a dedicated optimization program management tool, a wiki, or a spreadsheet. The purpose is to work in a more organized manner and to be able to have a bird’s eye view of your optimization practice.
Next, let’s take a look at some of the DON’TS when testing your data.
Don’t overburden your variations.
When creating the variations, try to have them as granular as possible. If you include multiple changes in one variation, then it can be difficult to identify causation from within all the changes. So if you want to test multiple items, consider running multivariate experiments or several stages of testing.
Don’t add a myriad of metrics as primary goals.
We know that with every test you want all your KPIs to go up. It is important to monitor all impact derived from your experiments, but only focus on one primary metric.
Don’t keep your testing plan behind closed doors.
Make sure to share your testing plans with all your stakeholders. If it’s a more strategic test, don’t forget to also get executive buy-in. This will help spread out the concepts of testing, make approvals a straightforward process in the long run, and nurture a culture of experimentation within your organization.
Don’t forget to estimate your sample size.
Hopefully, you don’t just focus on one experiment at a time, and you have multiple hypotheses you want to test on at the same time. Estimating your sample size needed for reaching significance is an important step for understanding how long your experiments will run for. This also adjusts stakeholders’ expectations on how quickly you will be able to get results.
Don’t rush into starting an experiment.
Lastly, don’t rush into running a test before you have clearly defined all the steps of building a testing plan. Leaving blank spots might have unexpected consequences and will lead to a more difficult decision-making process.
Now that you know where to focus your energy in A/B testing, you may be asking yourself:
How do I build a testing plan?
Well, there’s good news. We break down the process for you step-by-step in our eBook: How to Build a Strong Optimization Practice. In this free eBook, you’ll learn how to build testing plans that deliver results, shift your company’s culture, and make your organization more data-driven. Check out our eBook today to get started.