A/B test significance calculator

Did your new version win, or was it chance? Get the confidence level and p-value.

Control Group

Variant Group

Statistical significance

Not yet
Below 90% confidence, keep the test running
Control rate5.00%
Variant rate6.00%
P-value0.3267

What your result looks like

Each pile shows the results this test could have produced by chance. The less the piles overlap, the more confident you can be. Move left to see what the same rates would have told you with less data.

A: controlB: variantEach ball is one possible result of this test. Hopper A drops at the control rate (5.00%), hopper B at the variant rate (6.00%); the pegs scatter them as much as a sample this size really would. The less the two piles overlap, the more confident the result. Move left to see what the same rates would have told you with less data.

How significance works

z = (rate B − rate A) ÷ standard error

The calculator runs a two-proportion z-test on your two conversion rates. The p-value is the chance of seeing a difference this large if the change did nothing.

How do I calculate statistical significance for an A/B test?

Enter visitors and conversions for the control and the variant. The calculator compares the two conversion rates with a two-proportion z-test and returns the p-value and confidence level.

Is it a 5% or a 95% significance level?

They are the same threshold seen from two sides. A 5% significance level (p below 0.05) is 95% confidence.

Is a p-value of 0.001 statistically significant?

Yes. It is far below the usual 0.05 cutoff, which is 99.9% confidence.

What confidence level should I use?

95% (p below 0.05) is the standard for A/B testing: a 5% chance that the result is random variation.

How many visitors do I need for 95% confidence?

It depends on your baseline conversion rate and the lift you want to detect. The sample size calculator gives the exact number per version.

What if my test is not significant?

You need more data. Check the sample size calculator, then let the test run its full duration.

Can I stop a test early if it is significant?

Stopping early raises false positives. Run the test for the duration the test duration calculator gives you.

More for a/b testing

All 18 free tools
Sample sizeVisitors you need for a valid testTest durationDays until you can call the testConversion rateYour rate against benchmarks

Get real-time significance, automatically.

Humblytics calculates statistical significance continuously as your test runs, no manual number crunching. See live confidence levels, see when a winner emerges, and let your agent explain the results.

Start free trialBook a demo

14-day trial · Card required to start · Cancel anytime

/pricing · headline testWinner found
Control$1.12
Variant B$1.79

+59.4%revenue per visitor · 95% confidence