Blame view

docs/blog/refer/dynamic-yield/free-ab-testing-calculators.md 2.87 KB
62b7972c   tangwang   docs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
  # Free Bayesian A/B Testing Calculators
  
  ## We’re excited to announce the official release of two free and useful tools: Our Bayesian A/B Testing Calculator and Test Duration/Sample Size Calculator.
  
  A while back, we explored a less restrictive and more reliable approach to A/B testing in the form of a newer, Bayesian testing method. With its simplicity, reliability, and intuitiveness, the Bayesian framework is a superior A/B testing methodology which will provide marketers a quicker and more robust statistical engine.
  
  Therefore, we’ve been working long and hard to make it easier for marketers and conversion rate optimizers to utilize this new approach. And for us, that’s meant taking the complex math out of the equation.
  
  ## Bayesian A/B Testing Calculator
  
  Our Bayesian-powered A/B testing calculator will help you find out if your test results are statistically significant. For each variation you test, all you have to do is input the total sample size and number of conversions. Then, based on statistical significance, the statistical engine will declare a winning variation.
  
  Here’s a quick breakdown of the terms and metrics we run:
  
  *   **Sample Size** – The number of users, sessions, or impressions depending on your KPI.
  *   **Conversion** – The number of clicks, even purchases or goal completions (e.g. purchases or video views).
  *   **Conversion Rate** – The number of completed actions (i.e. conversions) divided by the sample size.
  *   **Probability to be Best** – Each variation’s long-term probability to outperform all other live variations.
  *   **Expected Loss** – The percent you are expected to lose in the long term if you declare the wrong variation as a winner.
  *   **Posterior Simulation of Difference** – The distribution of conversion rates given the sample size collected so far.
  
  ## Bayesian A/B Test Duration & Sample Size Calculator
  
  Not sure how long you will have to run your experiments in order to get statistically significant results?
  
  Our free online Bayesian-powered A/B test duration and sample size calculator will help you avoid false positives and increase the validity of your A/B testing. There’s no hard limit on how many variations you can test against the control.
  
  The calculated output provides range estimations of the time required to run the test in order to get statistically significant results, and the minimum required sample size to support that.
  
  Here’s a quick breakdown of the terms and metrics we run:
  
  *   **Baseline conversion rate** – The current conversion rate for the experience you’re testing.
  *   **Expected uplift in conversion rate** – The X% change in conversion rate you are aiming for from your baseline rate.
  *   **Number of variations** – The number of variations compared in a single test.
  *   **Average sample size per day** – The number of visitors to be served in the experiment over the course of one day.