A/B testing done right: a beginner's guide

A/B testing is the most powerful tool in conversion optimisation. Instead of guessing what works, you test two variants against each other and let the data decide. Sounds simple — but in practice, alarmingly often, it's done wrong.

In this guide we'll show you how A/B testing actually works, which mistakes to avoid, and how to get meaningful results even with low traffic.

What is A/B testing?

In an A/B test you show two different versions of a page or element to different visitor groups simultaneously. Version A (the control) is your current page. Version B (the variant) contains a change you want to test.

Traffic is split randomly: 50% of visitors see version A, 50% see version B. After a defined period, you compare the results and know which version converts better.

The key rule: only ever change one element at a time. If you change headline, image, and button colour at once, you'll never know which change made the difference.

What should you test?

Not every test is equally valuable. The biggest levers for conversion rate usually sit at these elements:

1. Headlines (highest impact)

The main headline is the first thing a visitor sees. A different phrasing can shift conversion rate by 20-50%. Test:

• Benefit-oriented vs. feature-oriented ("Win more customers" vs. "Professional landing pages")
• Question vs. statement ("Are you losing 97% of visitors?" vs. "97% of visitors don't convert")
• Specific vs. general ("67 new customers per month" vs. "More customers")
• With number vs. without ("More inquiries in 30 days" vs. "More inquiries fast")

2. Call-to-action (high impact)

The CTA button is the moment of truth. Small changes can have outsized impact:

• Button text: "Get a free consultation" vs. "Start now" vs. "Book a slot"
• Button colour: contrast colour vs. brand colour
• Button size: bigger is often better, but not always
• Position: above the fold vs. after the social proof
• Microcopy below the button: "No credit card required" or "Takes 2 minutes" can lift click-through

3. Social proof (medium impact)

Test different formats and placements:

• Testimonials with photo vs. without photo
• Video testimonials vs. text testimonials
• Customer counts vs. individual reviews
• Placement: directly under the hero vs. before the CTA

4. Forms (medium impact)

Form design directly affects inquiry volume:

• Number of fields: 3 fields vs. 5 vs. 7
• Single-page vs. multi-step form
• With required-field markers vs. without
• Inline form vs. pop-up after button click

5. Images and video (variable impact)

Visual elements influence the visitor's emotional reaction:

• Real team photos vs. illustrations
• With video vs. without
• Product-in-action vs. product alone

How long does an A/B test need to run?

This is where most people make the biggest mistake: they end the test too early. An A/B test is only meaningful once it has reached statistical significance.

What is statistical significance?

Statistical significance means the observed difference between version A and B is very likely real and not due to chance. The industry standard is a 95% confidence level — meaning the probability that the difference is random sits below 5%.

Why ending early is dangerous

Imagine: after 3 days version B has a conversion rate of 5.2% and version A 3.8%. That looks convincing — but with only 200 visitors per variant, the result is statistically meaningless. If you switch to version B now, the long-term result may be worse than the original.

Rules of thumb for test duration

• At least 100 conversions per variant — anything less isn't reliable.
• At least 2 full weeks — to balance day-of-week variation.
• Never start or end mid-week — Monday traffic differs from Friday traffic.
• Use a significance calculator — tools like Optimizely's calculator show whether your result is reliable.

Common A/B testing mistakes

Mistake 1: too many changes at once

If you change headline, image, button colour, and layout at the same time, that's not an A/B test — it's a redesign. You can't know which change made the difference. Always test one element per test.

Mistake 2: testing changes that are too small

The other extreme: testing whether a button in green vs. dark green converts better. On most websites the difference is so small you'd need months for a statistically significant result. Test big, meaningful changes — different headlines, different page structures, different audience messaging.

Mistake 3: ending the test as soon as one side is winning

Our brains love confirmation. When version B is leading after 3 days, it's tempting to stop the test and crown the "winner." Wait until statistical significance is reached — even if it requires patience.

Mistake 4: no hypothesis

A good A/B test starts with a clear hypothesis: "If we change the headline from feature-oriented to benefit-oriented, conversion rate will lift by at least 15%, because visitors recognise the personal benefit faster." Without a hypothesis you test blindly — and learn nothing from the results.

Mistake 5: not documenting results

Every test generates knowledge about your audience. Document what you tested, why, what the result was, and what you concluded. That knowledge is more valuable long-term than any single test win.

The best tools for A/B testing

You don't need expensive enterprise software to get started. Here are the best options by budget:

Free / low-cost

• Google Optimize (now: Google Analytics 4 Experiments): free, native integration with Google Analytics. Great for getting started.
• Clarity by Microsoft: free heatmap and session-recording tool. Not A/B testing, but ideal for analysis before the test.

Mid-range

• VWO (Visual Website Optimizer): from ~€99/month. Visual editor, no coding required. Solid value.
• AB Tasty: European provider, GDPR-compliant. From ~€149/month.

Enterprise

• Optimizely: the market leader. Powerful, but expensive. From ~€500/month.
• Kameleoon: European enterprise provider with a strong privacy focus.

A/B testing on low traffic

A common problem for local businesses: "We only get 500 visitors a month — can we even run A/B tests?" The honest answer: classic A/B tests are difficult at very low traffic. But there are alternatives:

• Sequential testing: show version A for 2 weeks, then version B for 2 weeks. Less reliable than simultaneous testing, but better than nothing.
• Qualitative tests: ask 5-10 people from your target audience about both versions. Often a few qualitative interviews reveal clear preferences.
• Heatmap analysis: tools like Hotjar or Clarity show where visitors click, scroll, and bounce — meaningful even on low traffic.
• Test bigger changes: the larger the difference between A and B, the less traffic you need for a significant result.

A practical example

A tax advisor with a landing page for "Have your tax return prepared" tests the headline:

• Version A (control): "Professional tax advice in Munich"
• Version B (variant): "Average €1,200 refund — tax returns from €149"

Version B converts 34% better. Why? It communicates a concrete, quantifiable benefit (€1,200 refund) and names the price, removing uncertainty. Version A is generic and interchangeable.

From this single test we learn: this audience responds to concrete numbers and pricing transparency. That insight feeds into all future tests and optimisations.

Bottom line: data over gut feeling

A/B testing takes the guessing out of conversion optimisation. Instead of debating whether the headline should be green or blue, you let your visitors decide. The key is discipline: formulate clear hypotheses, test one variable at a time, wait for statistical significance, and document results.

Start with the element that has the biggest impact — usually the main headline or the call-to-action. A single well-run test can lift your conversion rate sustainably and pay off for months and years.