A/B testing is not a process run just by analysts. This is a process that requires several different team members for every experiment. In this post, I’m going to break down what the specific steps are to building an A/B testing process and who is involved in each step.
First, for those less familiar with this topic, A/B testing is when you conduct a randomized experiment with two or more variations of a digital experience to determine which version is more effective. Building a strong testing process is something that’s very important to do because every business needs to understand how to gain beneficial insights from their data.
From my perspective, with a testing process, there are 7 key steps. Each step contains a main contributor and has specific deliverables. Let’s do a quick overview of what’s involved in each one.
Step 1: Build a strong hypothesis.
First, let’s start with what a hypothesis is in digital experimentation. A hypothesis is a prediction supported by evidence and is created before a testing run. The main contributor to this step is the product/marketing management team, and deliverables include testing an idea with a pre-analysis. Structure-wise, a hypothesis is broken down into three parts.
If [variation], then [outcome] because [argument].
The variation portion is an element in the experience that will be changed to improve the performance of the digital channel. For the outcome, it is the expected result of the updated experienced element. And finally, the argument is the data (qualitative or quantitative) that supports the claim and makes the hypothesis stronger. All of these elements, when incorporated together, make up the hypothesis structure.
Here’s a working example:
Step 2: Craft test variations.
This is the next step in creating a testing process. After the hypothesis is built, the design team steps in as the main contributor, and they deliver the mock-ups.
In experimentation, variations are different versions of an experience that are tested against each other to determine which performs better for a specific goal.
There are two basic types of variations:
Step 3: Architect the experiment.
After you craft test variations, you can now build a blueprint of online experiments. These are known as testing plans that are delivered by the main contributor in this step: your analytics team. Some things to measure or questions you may be asking at this step include:
Step 4: Implement the actual variations.
For this step, the main contributor is the development team, and they deliver variations on tracking. The implementation process can be broken down into two different ways:
All of these steps may seem a little overwhelming at first for any organization to take on. Many different teams are involved in this process, and it takes time, but keep in mind that what we do is simplify the A/B testing process. It’s also important to remember that the entire process of experimentation is a cycle. All new learnings from a deeper analysis are the ideas going through a prioritization process for the next hypotheses to test.
This overview is just the beginning of how we can help you simplify the process of testing your hypotheses. Stay tuned for future posts as we dive into each step more in depth.