A/B Testing Maturity

A/B testing is an incredible tool to have in your marketing arsenal. By definition, A/B testing allows you to introduce a change in one variable of your site. This post elaborates on what that variable can be, this also dictates the maturity of your A/B testing efforts.

Versions of Content

Basically wherever you have versions of content (Audio, Images, Text, Video etc.) you can use A/B testing. Now the underlying versions might be different in one way or the other. Some of the different versions can be :

  • Anything related to fonts : Font size, Font Face etc.
  • Anything related to color
  • Anything related to size
  • Anything related to look & feel / Completely different content

These form the most basic of A/B tests and are widely used. However, in no way are these A/B tests inferior or yield less business results. These tests are especially helpful when you are discussing which banner/CTA/Headline/Promotional Offer/Size of product pack /product description is better.

Location of content

In this type of A/B test – you want to change just the location of the content.

Moving that “contact us” from the left side to right side is an example of this test.

Moving a menu item from one place on the header to another

Basically, you want to play with the location of your content. The results can be surprising.

Number of fields in a content object

This is applicable when you have a content object. Survey & Registration Forms are good examples of content objects. Here we want to try the ideal number of individual units in a content object. For example , a registration form with 3 fields may be better than 5 fields. This is a simple scenario & answer is self explanatory. However, there are complex scenarios – the answers to them are only obtained through A/B testing.

These tests are in no way limited to forms – you can use these to experiment whatever you want to experiment on. Ex: Identify the ideal number of reviews to be put on a page during checkout site. We frame hypothesis and test it using A/B testing.

Page Layout

Here we just experiment with the page layout. Examples of these tests are :

  • Number of content columns / tiles
  • Size of the content columns etc.

User Flow

In this case, our variable is the user flow. Let me give you a quick example. An eCommerce site has below journey for check out

Check out – > Enter Shipping Information – > Enter Payment Details -> Order Confirmation

There can be so many ways to build this journey:

Check out -> Enter Payment Details with check box for using earlier used Shipping Address -> Order Confirmation

I guess this explains what we are trying to do here.


If there are different algorithms to determine the price on site and you want to see which algorithm fares better – then we run a A/B test to find the winner.

New Features

If there is a new feature going live – we test it to see how it resonates with the users. We build a hypothesis and that dictates the test. Example:

  • Does auto play of video on product display lead to more conversions ?
  • Does auto populating the payment information with a placeholder to enter CVV lead to more conversions?
  • Does adding a visual indicator of shopping cart progress lead to more conversions ?
  • Does adding new sections on pages (reviews, number of purchases etc.) lead to more conversions
  • Will prompting for registration later on lead to more conversions?

So, in this way – we start with basic A/B testing and the move on to use A/B testing for tactical decision making. A/B testing is both art and science. Art is designing the test while implementing it is science.

Adobe Target , Optimizely & VWO helps in accomplishing few of these scenarios quite easily, but it requires a lot of commitment to implement such A/B tests for continuous decision making.

Note : There are some nuances of A/B testing that one should be aware of, these are often ignored. The below article captures these very well.

Dark Side of A/B testing

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.