About this sin
One should be using evidence to make decisions, not to only confirm one’s assumptions. Our Confirmation Bias has a terrible way of biasing how we interpret data. We test to lower our standards if it means confirming our beliefs.
How to atone
Decide what you will do before a test is run based on the potential outcomes of a test. Decide on your test’s thresholds and what you will do if your test shows positive lift, negative lift, neutral lift, or is inconclusive ahead of time. Be honest. Involve your stakeholders to avoid trouble down the line.
Useful script
Because we want to move quickly, I recommend deciding our next steps based on the results of your test ahead of time. This way we can start planning for potential outcomes and save time. To remove barriers down the road, let's involve all your major stakeholders now - otherwise, based on my experience, we will likely run into challenges later. It's not unusual for organizations to spend months going back and forth trying to figure out what to do based on test results. Here is a useful article that I feel highlights the importance of establishing decision-making criteria up-front:
