←  All terms
A
Analytics & Conversion

A/B Test

An A/B test compares two versions of a page or element to identify the better-performing option using measurable user behavior.

An A/B test compares two versions of a page or element to identify the better-performing option using measurable user behavior. Reliable use requires a clearly defined metric, sound data, and privacy-conscious analysis. The concept is closely related to Conversion Rate Optimization and Personalization.

Read more

Conversion Rate Optimization

Personalization

Landing Page

FAQ

A/B Test

What are typical mistakes in A/B testing?
Common mistakes include ending tests too early, bundling several changes into one variant, and starting without a reasoned hypothesis. Tracking gaps also distort results. A clean test starts with a hypothesis, a defined metric, and a check that data collection works correctly before launch.
How does an A/B test differ from a multivariate test?
An A/B test compares complete variants, while a multivariate test evaluates combinations of several elements at once. Multivariate tests need far more traffic because every combination requires its own sample. For pages with limited traffic, running A/B tests one after another is usually more practical.
Is an A/B test worthwhile with low traffic?
With low traffic, an A/B test rarely pays off because small effects cannot be detected statistically. Bigger changes, qualitative user testing, or session analysis produce insights faster in that situation. A test becomes reasonable once traffic is sufficient to detect the expected effect size within an acceptable timeframe.
How long should an A/B test run?
A test runs until enough visits have accumulated for a reliable conclusion and at least full weekly cycles are covered. Stopping early when one variant seems to win often locks in random noise. The required duration depends on traffic, baseline conversion rate, and expected effect size, and should be estimated before launch.
Which metric is suitable for an A/B test?
A suitable metric maps directly to the goal of the change, such as completed forms or clicks on a call to action. Raw click counts with no link to a business goal are misleading. A sound setup defines one primary metric before launch and adds a few guardrail metrics such as bounce rate or load time.