A/B Testing Methodology

a/b testing
A/B Testing Methodology

by: Emily Allred, RMI Art Director

The popularity of A/B testing has increased dramatically over the past decade, and that rising tide of awareness has carried along with it countless webinars, conference segments, clickbait-y numbered list articles, infographics, ebooks, A/B testing platforms, and on and on. Most people now have an idea of what A/B testing is on the surface, and many people do test, but fewer people have a deep understanding of the whys, whens, and hows of A/B testing methodology.

What is A/B Testing?

Put most concisely, A/B testing on the web is a randomized experiment wherein one or more treatments are pitted against the control. Visitor behavior is tracked during the test and the results are analyzed based on pre-determined key performance indicators.

Put more simply, it is comparing two distinct designs or variations on a base design against one another with normal (random) traffic, while tracking the outcomes and analyzing the resulting data to determine which version affects the measurements of success more positively. If the goal of a test is to increase sales, and the data show that the variant produces better email list sign-ups but lower sales, you won’t keep that design. The test is a loss, and you will analyze the data to try to determine what went wrong and what you can do better next time.

As you learn what does and does not work, eventually you will start to see some wins and positive optimizations will result in incremental lifts in the page’s conversion rate (or whatever other key performance indicator is desired). This process is called continuous optimization, and it is a practice that has increased in popularity over time as a method of improving sites over time without the chaos and risk of wholesale redesigns.

However, it’s possible to damage your long-term revenue with poor testing practices and while proper testing practices can seem very complicated, some basic knowledge will go a long way to helping you successfully practice continuous optimization.

What A/B Testing is Not

A/B testing is not a panacea. There is no guarantee of results, profits, promotions, etc. The purpose of A/B testing is to learn – with the obvious intent of learning something to implement in a way that improves business – and during this learning process as often as not you will learn that your hypothesis was incorrect.

A/B testing is not for everyone. (More on that in a minute.)

A/B testing does not establish universal truths. First, results are not always clear for a variety of reasons (seasonality, external factors, unknown unknowns, etc.). More importantly, results are not necessarily translatable across sites/clients/verticals/time. The layout, content, branding, category, state of mind of the visitor, and overall design and cultural trends will all influence results, and a test that gave you a mega lift in one case may not work on another site, in another geo, or after a certain period of time when your cutting-edge tactics have become familiar.

A/B testing is not exciting most of the time. A/B testing is not exciting on a day-to-day or sometimes even week-to-week basis. You’re playing the long game here. Find something else to put in daily or weekly reports.

A/B testing is not immune to garbage in, garbage out. Here are some common reasons tests don’t perform well:

  1. Test ideas are not based on real insights and not validated with analytics or other sources.
  2. The tests selected to run are not prioritized by potential.
  3. Other changes were made or conflicting tests run during the test.
  4. The test had technical issues.
  5. Traffic sources changed.

A/B testing is not always valid – even when you’ve done everything right.

Why A/B Test?

Ideally, A/B testing is statistical hypothesis testing as part of a cycle of continuous experimentation. Quality data are used to determine a potential test (the hypothesis), the test is run, data are collected and analyzed, knowledge is gained, and the cycle repeats itself. For many, this is a common sense decision that’s good for business and good for you.
Need more convincing? Here are some good reasons to test:

Doing Nothing Is Dumb
Pretty simple. Every visitor to your site could be part of a data set that could tell you more about what you should or should not focus on, what does or does not work, how different segments behave, etc. Letting those people pass without gathering that data from them is like tossing pennies in a well. Each one isn’t worth much, but if you keep on doing it all day every day, you’ll eventually waste a lot of money that you can’t get back.

Measurable Results
If you understand and minimize threats to validity and follow basic procedure, you can be fairly assured that you are measuring some kind of effect and gathering some knowledge with each test. You can show results and act with confidence. This may not win over an “actions speak louder than words” audience, but it’s great for data-driven individuals.

Targeted Effort
The measured results you get can help you focus your energies in the right areas. For example, if you see that certain types of tests seem to wash out for your site, then you will spend your efforts in the future focusing on different tests that will move the needle.

Incremental Gains
You can risk a lot of time and money on full redesigns – and maybe your complete overhaul will be a smashing success. But you can incrementally increase your revenue (or sign-ups, or shares, or whatever you’re optimizing for) by A/B testing. Individual tests spread out time, effort and costs, and you learn specifics about what does and doesn’t work along the way.

Reduced Risk
If you engender a data-driven, learning-oriented approach to optimization, you not only get all of the above benefits, but you can reduce volatility. People start to learn that tests are rarely predictable, and that anyone can have ideas that pop or that drop. Example:

So your HIPPO wants to overhaul the site because he thinks it’s stale. Maybe it is stale, or maybe someone with a lot at stake spent too much time staring at it. Will the overhaul do better or worse? You may never know. Maybe everything seems fine at first, but over time it seems the same or worse. If you’re not measuring, you’ll certainly never know. If you’re not testing, it will be hard to prove in any case

If you are a persuasive A/B testing proponent, you’ll identify some of your HIPPO’s specific concerns, and convince them to let you slowly but confidently optimize your way to a refined version of your site. Or, in the more likely event that you go for the complete overhaul, it will be tested in rather than blindly implemented. Some of your users will be getting the best-performing experience either way, reducing your risk. It’s important to note that with careful monitoring, a true bomb can be stopped at any point (though it’s actually not recommended…. more on that later on).

In the end, the experience you gain through testing allows you to avoid pitfalls and move forward armed with more knowledge.

How to Approach Testing

Your approach to A/B testing should not be to come up with a bunch of haphazard ideas and test them. Furthermore, the goal of testing is not to simply have tests running so you can say you’re an A/B tester. Remember what A/B testing is not.

Tests should have a reason – a hypothesis – to justify them. The reason can come from reading a best practices article, coming across a cool feature on some website, tracking site performance data, or analyzing the demographics and behavior of your site’s users, etc. But you should have an idea of what you’re testing, what you expect to see, and why you expect that result.

Recall that A/B testing is a cycle of continuous experimentation. The major phases are research, hypothesis, test, measure… repeat. Research is both the first and last step of every test. Some ways to perform the research that provides the insights to generate effective testing:

  1. Web analytics (i.e. Google Analytics)
  2. On-page analytics (heat/scroll maps, etc.)
  3. User screen recordings
  4. Online polls, surveys, and feedback forms
  5. User testing
  6. Heuristic methods (problem-solving through educated guesses – walk through the funnel with the
    mindset of the user/segment you are optimizing for and look for problems or areas of opportunity)


Want more in-depth information into A/B Testing Methodology? Download our white paper on the topic!
You’ll learn more on the above information as well as:

  1. Can you A/B test?
  2. What Should you A/B Test?
  3. The Nitty Gritty on Proper Testing Methodology.
  4. Ending a Test.
  5. The RMI Way to testing!

Get me that white paper!


At Response Mine Interactive, testing is part of our core.  We are continually examining and testing our clients’ goals. We don’t take your advertising to market with just one idea. We come with a plan to test, optimize and repeat, which creates the best opportunity for more customers for you. See how RMI can help your marketing initiatives by contacting us or call, 404.233.0370 today!

Response Mine Interactive