Tag: experiment
-
Does It Matter That Online Experiments Interact?
Does It Matter That Online Experiments Interact? What interactions do, why they are just like any other change in the environment post-experiment, and some reassurance Photo by Uriel Soberanes on Unsplash Experiments do not run one at a time. At any moment, hundreds to thousands of experiments run on a mature website. The question comes up:…
-
Scale Experiment Decision-Making with Programmatic Decision Rules
Scale Experiment Decision-Making with Programmatic Decision Rules Decide what to do with experiment results in code Photo by Cytonn Photography on Unsplash The experiment lifecycle is like the human lifecycle. First, a person or idea is born, then it develops, then it is tested, then its test ends, and then the Gods (or Product Managers) decide its worth.…
-
Stop the Count! Why Putting A Time Limit on Metrics is Critical for Fast and Accurate Experiments
Stop the Count! Why Putting A Time Limit on Metrics is Critical for Fast and Accurate Experiments Why your experiments might never reach significance Photo by Andrik Langfield on Unsplash Introduction Experiments usually compare the frequency of an event (or some other sum metric) after either exposure (treatment) or non-exposure (control) to some intervention. For example:…