Creating an A/B test for a site becomes so easy using Adobe Target. However, one case easily overlook certain aspects of the test. This can lead to a bad experience for the site visitors. This post aims to share few tips to ensure a smooth implementation end to end.
Strategize the test
This means at a minimum – think and document the below aspects of the test :
- Objective of the test : Most of the times this will be an improvement on metrics like Clicks, PageViews etc.
- Site Element you want to test : You may not want to test a element which is dynamic and will be taken off from the site during the next content update
- Finalizing the duration of the test : You can use the Adobe Target sample size calculator to get an idea of this. This is to ensure mistakes like -Declaring a winner before the test is complete or continuing a test endlessly are avoided.
- Finalizing the Audience for the test: It is common for companies to do a test on 20% of population and have the rest of the population experience the winner on a big day/promotion
While creating a test is very easy. One should be careful on the below aspects:
- Check for flickers which is control version getting displayed before the experience B. I will write a separate post on how to solve the flicker issue on Target
- Check whether all pages ensure the same experience. This is very much needed when you are performing a test on items which can be seen across pages like headers and footers. All you have to do is go to Configure – > Page Delivery and add the domain/Url/ Path
- Check the site UI elements : The UI elements can easily fall out of layout or grid. So you have to ensure things like grids or image containers are resized. This ensure a good experience for the visitors
Reporting & using the results:
- Ensure that the test is deactivated once the duration is passed. Check the lift and the confidence. A lift of >10% with a confidence of 90% is good to implement (opinion). This actually depends on the organization as to what they want the confidence level to be . Some can even peg it 95 or 98% . Implement what you have learned from the A/B test and optimize the site.
Let me know your thoughts as well