I wrote about Emily Robinson and her A/B testing activities at Etsy before, but now she’s back with a great new blog full of practical advice: Emily provides 12 guidelines for A/B testing that help to setup effective experiments and mitigate data-driven but erroneous conclusions:
- Have one key metric for your experiment.
- Use that key metric do a power calculation.
- Run your experiment for the length you’ve planned on.
- Pay more attention to confidence intervals than p-values.
- Don’t run tons of variants.
- Don’t try to look for differences for every possible segment.
- Check that there’s not bucketing skew.
- Don’t overcomplicate your methods.
- Be careful of launching things because they “don’t hurt”.
- Have a data scientist/analyst involved in the whole process.
- Only include people in your analysis who could have been affected by the change.
- Focus on smaller, incremental tests that change one thing at a time.
More details regarding each guideline you can read in Emily’s original blogpost.
In her blog, Emily also refers to a great article by Stephen Holiday discussing five online experiments that had (almost) gone wrong and a presentation by Dan McKinley on continuous experimentation.