How to Improve UX Design Through A/B Testing

By Andrii Horiachko / 01 August 2019

Making decisions on design changes often involves discussion, but is there another way to discover what design elements are the best?

You must involve your users in the UX design process, make data-driven decisions, and create a product design they will truly love. A/B and multivariate testing will help you with these all.

Your ability to achieve business goals depends on the amount of effort that you put into the improvement of your product’s user experience (UX). According to research from Adobe, 38% of people will stop engaging with a website if the layout or design is unattractive.

UX improvements happen through testing product design elements and their effectiveness. Here UX testing comes in to help make data-driven decisions on what design elements impact users the most and encourage them to complete desired actions.

According to a Forrester study, for example, an improved UX design can raise your conversion rate up to 400%, making cost and time investments in UX testing reasonable.

Today more companies understand the importance of UX design testing for Conversion Rate Optimization (CRO). Many factor testing expenses into their budgets because testing is more cost-effective than full product redesigns.

Defining A/B and Multivariate UX Design Tests

A/B testing and multivariate testing are two complementary parts of a whole. A/B testing – otherwise called split testing – and multivariate testing (MVT) are both techniques that eliminate guesswork and use risk-free design modifications.

In A/B tests, you test one variant against another variant of the same element. For multivariate testing, you test a few variants with different elements. The differences are illustrated in the graphic below.

a/b and multivariate testing

Both types of tests break down complex website data on user visits, engagement, conversions, interactions with design elements, and users’ entire behavior.

When Humana used A/B testing for its banner design, for example, it ran a test on the two options of homepage banner design based on the banner image, color scheme, banner copy, and button form. Both options are shown in the image below.

humana a/b testing

After running the test the company discovered that a simpler design coupled with a stronger call-to-action was more effective. This led to a 433% higher click-through-rate and more application form submissions.

A/B and multivariate testing don’t need to be complicated. This article outlines the following steps for an effective implication of both testing techniques:

  1. Identify elements to test
  2. Map out steps of the testing process
  3. Choose tools to implement the test
  4. Avoid testing pitfalls

Follow these steps and you'll have run the most successful A/B testing possible.

1. Identify Elements to Test

To get insightful test results, you should base your testing process on some important differentials, including analytics and usability tests that show users’ pain points or specific problematic interactions.

Create your design improvement hypothesis based on these specific insights for best testing results.

Not all design elements are worthy of testing. Design elements that have a direct, instant impact on users’ actions or on your overall conversion rate are the most important.

From this comes two approaches to testing elements. They include testing:

  1. Elements close to the point of macro conversion
  2. Elements close to point of micro conversions.

A micro conversion is any step a user takes in your marketing and sales funnel where they show some interest in your product. This includes clicks on CTA or social media buttons, visits on different pages, or downloads of your resources.

Micro-conversions can be smaller steps toward primary macro conversions, which are the users’ main steps to complete your website’s primary conversion action. If your website’s main conversion goal, for example, is selling a product, then the macro conversions might include users adding the product to their cart and proceeding to purchase.

Once you define primary and secondary actions in your customers’ journey and the elements users interact with, you are ready to begin your UX testing of micro and macro-conversions to optimize their conversion rate.

2. Map Out Steps of the Testing Process

When you are ready to run your UX test, first identify data on your website or app’s traffic. This will help you understand how much time the test will require.

The graphic below illustrates the main steps required to conduct your testing, beginning with determining conversion metric to improve and ending with measuring results.

a/b testing map

For an A/B test, decide which single variable you’d like to test for reliable results. Conducting an A/B test of a CTA button, for example, should only test either the button’s color or form. If you want to test more than two variables, even within a single element, you should conduct a multivariate test.

Here are the key steps to conducting your test:

  • Establish metrics to define your results after choosing elements to test. To identify the key performance indicators for your test results, think of the specific metrics you’re looking to improve with A/B or multivariate testing.
  • Create a hypothesis and variation option based on the hypothesis. This could be as simple as creating an alternative version or variant B that includes changes to test.
  • Test the variation option against the existing version.
  • Analyze and implement the results of the winning variant.

One example of a successful test comes from broadcast platform Ustream. Their team hypothesized that a clear CTA on their main page would increase the number of broadcast sessions – the desired action of their user.

Ustream tested 2 design variants of just one button – one with a CTA and one without it. You can see the original and the variant in the image below.

ustream a/b testing

The B variant, tested against the original, resulted in a 12% conversion increase for Ustream. This small design change was effective for their business because of testing.

3. Choose Tools to Implement the Test

Digital tools can simplify your tests during the set-up, implementation, and analysis.

Some useful tools for UX design testing include:

  • Google Analytics: Google Analytics provides real-time digital marketing reports and website analytics. Use this tool for insights on users’ interactions and how you can improve them.
  • Google Optimize: This Google tool, shown below, sets up personalized A/B and multivariate tests while and displays results visually. Google optimize also implements the winning variant into your design.

google optimize

Google Optimize is a user-friendly, intuitive tool that walks users through each phase of the testing process.

  • Visual Website Optimizer: The Visual Website Optimizer allows users lacking prior technical knowledge to run A/B tests and geo-behavioral targeting campaigns. It is an all-in-one platform that helps users to conduct visitor research, build an optimization roadmap, and run continuous experiments.
  • Adobe Target: Adobe Target is a rule-based testing and targeting tool. It helps decipher which offers, experiences, and messages are truly engaging customers.
  • Maxymiser: Maxymiser optimizes every part of the online and mobile app customer experience. It optimizes website homepages, campaign landing pages, and multistep checkout funnels.

Not every tool will be the right fit for your testing. Try different tools and then choose one that best suits your needs.

4. Avoid Pitfalls of Testing

Results of your testing, while useful knowledge, should not be the only research recommendation you use in your UX.

Keep in mind that A/B testing will not provide an exact reason one particular design element works better than another. Qualitative user research can help identify these factors.

UX testing also doesn’t work in isolation. Consider how your tests will impact metrics of other projects.

Finally, always double-check the results your A/B or multivariate test provides. Supplement that quantitative data with qualitative testing. If the qualitative testing correlates with your original results, you can implement your improved elements in your UX design.

A/B and Multivariate Testing Are Key to Improving UX

A/B and multivariate testing are important steps to achieving your business' goals. Testing improves your UX and directly impacts lead generation and customer retention.

If you need professionals to assist in UX design improvement, you can leverage our Softermii design team’s tech expertise in UX design testing and development.

If you decide to do testing on your own, however, iterate and work on your UX strategy for improved metrics across your website or app.

Lead capture icon


Based on your budget, timeline, and specifications we can help you build a shortlist of companies that perfectly matches your project needs. Get a free shortlist of best-fit companies from a Manifest Analyst.