A/B testing calculator for statistical significance

Statistical significance is important when running A/B tests because it ensures your results are certain and didn’t happen by chance. 

Gather quick answers using the SurveyMonkey A/B testing calculator above.

A/B testing, or split testing, compares the performance of two versions—such as a product concept or ad creative—to identify which variant is more appealing to your target audience.

Researchers, CX professionals, and marketing experts use A/B testing to test small changes, like a new website button or homepage design. It provides direct feedback and data to guide decisions on which variant to choose. 

In A/B tests, statistical significance measures the likelihood that the difference between the control and test versions is genuine and not due to error or random chance.

For example, if you run a test with a 95% significance level, you can be 95% confident that the differences are authentic.

Statistical significance is used to observe how experiments affect your business’s conversion rates. In surveys, statistical significance is used to ensure confidence in results. 

For example, if you asked whether people preferred ad concept A or ad concept B in a survey, you’d want to ensure the difference in their results was statistically significant before deciding which ad concept to use.

First, you must form a hypothesis. For any experiment, there is a null hypothesis, which states there’s no relationship between the two things you’re comparing, and an alternative hypothesis.

An alternative hypothesis typically tries to prove that a relationship exists and supports the statement you’re trying to make. 

For example, if you are doing conversion rate A/B testing, your hypotheses may be:

After formulating null and alternative hypotheses, statisticians sometimes do tests to ensure their hypotheses are sound.

A z-score represents confidence level and evaluates the validity of your null hypothesis, which can tell you if there is, in fact, no relationship between the things you’re comparing. A p-value tells you whether the evidence you have to prove your alternative hypothesis is strong.

Next, decide whether your test will be one-sided or two-sided (sometimes called one-tailed or two-tailed). A one-sided test assumes that your alternative hypothesis will have a directional effect, while a two-sided test accounts for if your hypothesis could have a negative effect on your results.

For instance, in the conversion rate A/B testing example, your test could be:

Next, you will collect the results from your A/B test, including the relevant metrics for both the control (A) and test (B) versions. 

In our example, the results of the A/B test could be: 

Then, you will calculate the z-score which measures how far the observed results are from the null hypothesis to determine if the difference between A and B is statistically significant. 

Additionally, you will calculate the p-value, which indicates the probability that the observed difference is due to random chance. A smaller p-value suggests stronger evidence against the null hypothesis. 

In our example:

To determine statistical significance, set a significance level (alpha). This is commonly set at 0.05 (5%), representing the acceptable risk level for incorrectly rejecting the null hypothesis.

Next, compare your p-value to the alpha level. If the p-value is less than the alpha level, reject the null hypothesis and conclude the difference is statistically significant. 

In our example, the p-value is less than the alpha level, meaning the difference of 14% is statistically significant.

Now, it’s time to interpret the results. If you receive significant results, it indicates that the observed difference is unlikely due to chance, providing evidence supporting the alternative hypothesis. Non-significant results indicate insufficient evidence to reject the null hypothesis, meaning the observed difference could be due to random variations.

For the most efficient process, use tools for calculation such as:

In summary, statistical significance validates your A/B testing results. Using statistical significance is important in making informed decisions based on A/B tests.

Check out the calculator at the top of the page to automatically calculate the significance of your survey results.