-
4. Having no intention of course-correcting based on experimental evidence
Product teams are often guilty of this. Roadmaps are committed to and things are promised. No one wants to be the one that disappoints so they focus on delivery rather than performance. Experimentation is set aside as a distraction or as unnecessary. You’ll often hear the phrase, “We know it will work. There’s no need to test this. We have a lot of customer feedback.” -
3. Not establishing decision-criteria before you run a test
One should be using evidence to make decisions, not to only confirm one’s assumptions. Our Confirmation Bias has a terrible way of biasing how we interpret data. We test to lower our standards if it means confirming our beliefs. -
2. Blindly following Best Practices
Starting with a Best Practice is not wrong. But never challenging them or validating them on your site absolutely is. Every site is different and serves different audiences. There is no such thing as a one-size-fits-all. -
1. Peeking
Stopping a test once Statistical Significance is reached. This is a grave sin as it invalidates your data. Under no circumstances should you do this.
A list of sins that Conversion Rate Optimizers (CROs) make – and how to atone for them.
