Why Your Design of Experiments Is Probably Wrong

We love to run experiments. We love to throw bold ideas at the wall to see if they stick, tinkering with different factors, and seeing how what happens can be incorporated into what we do every day. To us, it’s a very hot topic — we’re writing about it whenever we can, and trying to lift the curtain on what, behind the scenes, we’re cooking up on our own marketing team.

But we’re going to let you in on yet another secret: Experiments are not designed to improve metrics.

What Is an Experiment?
In a forward-thinking marketing environment, it’s easy to forget why we run experiments in the first place, and what they fundamentally are. That’s why we like referring to the term design of experiments, which refers to “a systematic method to determine the relationship between factors affecting a process and the output of that process.” So, much like the overall point of conducting experiments in the first place, this method is used to discover cause-and-effect relationships.

To us, that informs a lot of the decision-making process behind experiments — especially whether or not to conduct it in the first place. In other words, what are we trying to learn, and why?

Experiments Are Quantitative User Research
From what I’ve seen around the web, there seems to be a bit of a misconception around experimentation — so please, allow me to set the record straight. As marketers, we do not run experiments to improve metrics. That type of thinking actually demonstrates a fundamental misunderstanding of what experiments are, and how the scientific method works.




for more you can check:


Product Reveal Video Example

%d bloggers like this: