Welcome to our A/B Test Calculator! This tool helps you compare the performance of two variants—commonly known as the control (A) and the variant (B) to determine which one performs better. By leveraging Bayesian statistics, our calculator estimates a probability that one variant performs better than a control, along with conversion rates and the percentage uplift between the two.

We believe strongly in the value of a Test & Learn approach as part of a broader data driven marketing strategy. If you would like to learn more about our more advanced and faster approaches to AB testing why not speak to us.

How to Perform a Basic A/B Test

To get started, simply run your experiment by splitting your audience into two groups—one exposed to the control (A) and the other to the variant (B). Input the total number of events and conversions for both groups into our calculator. For reliable results, aim for a probability threshold of at least 95%, indicating a high confidence level that the variant truly performs better than the control. This threshold helps minimize the risk of false positives and ensures that your decisions are backed by robust data.

Inputs

• Total Events (Control A): Total number of users (or events) exposed to Control A.
• Conversions (Control A): Number of successful actions (e.g., signups) from Control A.
• Total Events (Variation B): Total number of users (or events) exposed to Variation B.
• Conversions (Variation B): Number of successful actions (e.g., signups) from Variation B.

Methodology Overview: Bayesian A/B Testing with Fixed Prior

What is Bayesian A/B Testing? Bayesian A/B testing is a statistical approach that calculates the probability that one variant (A or B) performs better than the other based on observed data. Unlike traditional methods that rely on fixed hypothesis tests, Bayesian methods update beliefs as new data is collected, making the results more intuitive and flexible.

How Does the Tool Work?

• Fixed Prior Assumption: The calculator starts with a fixed, non-informative prior, which assumes that both the control (A) and the variant (B) are equally likely to perform well before any data is observed. This prior does not change over time, ensuring a fair and unbiased comparison from the outset.
• Posterior Distribution: The observed data is used to update the initial belief, resulting in a posterior distribution that reflects the updated probability of each variant’s performance.
• Monte Carlo Simulation: The tool uses Monte Carlo simulation to estimate the probability that one variant is better than the other by generating random samples from the posterior distribution and comparing the outcomes.

Benefits of the Fixed Prior Bayesian Approach

1. Fair Starting Point: By using the same prior for both the control and the variant, the tool ensures that the comparison is unbiased. Any observed differences in performance are driven purely by the data collected.
2. Intuitive Interpretation: The Bayesian approach provides results in the form of probabilities, such as the probability that one variant is better than the other, making it easier to make decisions based on the results.
3. Simplicity: The use of a fixed prior simplifies the calculations and provides a consistent baseline for comparison across multiple tests.

Potential for Dynamic Priors

While the current tool uses a fixed prior, more advanced Bayesian models allow for dynamic updating of the prior as new data is collected:

• Sequential Updating: In more sophisticated models, the prior can be updated after each round of data collection, using the posterior distribution from the previous round as the new prior. This allows the model to learn and adapt over time.
• Advanced Flexibility: Dynamic priors can incorporate historical data or adapt to changes in user behaviour, leading to faster convergence and potentially more accurate results.

Why Results Might Vary Slightly

Understanding the Results:

While the A/B Test Calculator provides a probability estimate for whether Variant B performs better than Control A, it’s important to understand what this probability represents:

• 100.00% Probability: Sometimes, the calculator may show a result of 100.00% probability. This does not imply absolute certainty. Rather, it reflects that, given the current data and assumptions, Variant B appears to outperform Control A across all simulated scenarios. However, this result could also be due to rounding; the actual probability might be very close to, but not exactly, 100%. Always remember that statistical estimates, even when very high, are not guarantees.
• Random Sampling Variability: The Monte Carlo simulation uses random sampling to estimate probabilities. Each time the simulation runs, the results can vary slightly due to the inherent randomness in the process.
• Fixed Prior Assumption: The tool uses a fixed prior as a consistent starting point for each test. While this helps standardize results, it doesn’t adapt to incoming data, which could slightly influence outcomes, especially if actual performance deviates significantly from the prior assumption.

Disclaimer: We have made efforts to ensure the accuracy of this tool, but please note that the results are estimates. Users should exercise their own discretion when making decisions based on this data. We cannot be held liable for any actions taken as a result of using this tool.