Q: How many visitors do I need?

A common rule of thumb is at least 100 conversions per variant before reading results. For low-traffic sites, run tests for at least 2 full business weeks.

Q: What does relative uplift mean?

Relative uplift shows how much better (or worse) the variant performs compared to control. A 30% uplift means the variant converts 30% more visitors — not that the conversion rate increased by 30 percentage points.

Q: Should I use one-tailed or two-tailed tests?

VizTools uses a two-tailed test, which is the safer default. It tests whether the variant is different in either direction, not just better.

Q: How long should I run an A/B test?

Run your test for at least one full business cycle — typically 1–2 weeks — to account for day-of-week traffic patterns. Never stop a test early just because it looks significant; early significance is often a statistical artifact called the "peeking problem." Decide your minimum runtime and minimum sample size before the test starts, and stick to both. For low-traffic sites, two full weeks is a reasonable minimum regardless of sample size.

Q: What sample size do I need for an A/B test?

The required sample size depends on your baseline conversion rate, the minimum detectable effect (MDE) you care about, your desired confidence level (95%), and statistical power (80% is standard). As a rough guide: detecting a 10% relative improvement on a 5% baseline conversion rate requires approximately 5,000–6,000 visitors per variant. Smaller effects require larger samples. Use a dedicated sample size calculator before starting a test if traffic is limited.

Q: What does a p-value of 0.05 actually mean?

A p-value of 0.05 means there is a 5% probability of observing a difference this large (or larger) by random chance alone, assuming the null hypothesis (no real difference) is true. It does not mean there is a 95% probability the variant is better — it is a statement about the data given the assumption of no effect. This is why replication and practical significance (the size of the uplift) matter as much as statistical significance.

Question 1

What is statistical significance?

Accepted Answer

A result is statistically significant when there's less than a 5% probability the difference occurred by chance. The industry standard is 95% confidence (p < 0.05).

Question 2

How many visitors do I need?

Accepted Answer

A common rule of thumb is at least 100 conversions per variant before reading results. For low-traffic sites, run tests for at least 2 full business weeks.

Question 3

What does relative uplift mean?

Accepted Answer

Relative uplift shows how much better (or worse) the variant performs compared to control. A 30% uplift means the variant converts 30% more visitors — not that the conversion rate increased by 30 percentage points.

Question 4

Should I use one-tailed or two-tailed tests?

Accepted Answer

VizTools uses a two-tailed test, which is the safer default. It tests whether the variant is different in either direction, not just better.

Question 5

How long should I run an A/B test?

Accepted Answer

Run your test for at least one full business cycle — typically 1–2 weeks — to account for day-of-week traffic patterns. Never stop a test early just because it looks significant; early significance is often a statistical artifact called the "peeking problem." Decide your minimum runtime and minimum sample size before the test starts, and stick to both. For low-traffic sites, two full weeks is a reasonable minimum regardless of sample size.

Question 6

What sample size do I need for an A/B test?

Accepted Answer

The required sample size depends on your baseline conversion rate, the minimum detectable effect (MDE) you care about, your desired confidence level (95%), and statistical power (80% is standard). As a rough guide: detecting a 10% relative improvement on a 5% baseline conversion rate requires approximately 5,000–6,000 visitors per variant. Smaller effects require larger samples. Use a dedicated sample size calculator before starting a test if traffic is limited.

Question 7

What does a p-value of 0.05 actually mean?

Accepted Answer

A p-value of 0.05 means there is a 5% probability of observing a difference this large (or larger) by random chance alone, assuming the null hypothesis (no real difference) is true. It does not mean there is a 95% probability the variant is better — it is a statement about the data given the assumption of no effect. This is why replication and practical significance (the size of the uplift) matter as much as statistical significance.

A/B Test Significance Calculator

Test data

Control

Variant

Not Yet Significant

Conversion rate comparison

How it works

Frequently Asked Questions

More from VizTools

UTM Campaign Link Builder

Email Campaign Revenue Calculator

Ad Budget Calculator