CRO Sins Logo

A list of sins that Conversion Rate Optimizers (CROs) make – and how to atone for them.

3. Not establishing decision-criteria before you run a test

3. Not establishing decision-criteria before you run a test
Home » Sins » 3. Not establishing decision-criteria before you run a test

About this sin

One should be using evidence to make decisions, not to only confirm one’s assumptions. Our Confirmation Bias has a terrible way of biasing how we interpret data. We test to lower our standards if it means confirming our beliefs.

How to atone

Decide what you will do before a test is run based on the potential outcomes of a test. Decide on your test’s thresholds and what you will do if your test shows positive lift, negative lift, neutral lift, or is inconclusive ahead of time. Be honest. Involve your stakeholders to avoid trouble down the line.

Useful script




Because we want to move quickly, I recommend deciding our next steps based on the results of your test ahead of time. This way we can start planning for potential outcomes and save time. To remove barriers down the road, let's involve all your major stakeholders now - otherwise, based on my experience, we will likely run into challenges later. It's not unusual for organizations to spend months going back and forth trying to figure out what to do based on test results. Here is a useful article that I feel highlights the importance of establishing decision-making criteria up-front:

Leave a Reply

Your email address will not be published.

CRO Sins