Customer Scientific Method
BY BROOKE NIEMIEC – VP, MARKETING STRATEGY
I think it’s time we give the phrase “test and learn” its proper recognition. Too often today, I hear it being said—and implemented—with reckless abandon. Tests are conducted on an ad hoc basis, sometimes with controls and sometimes without, often prioritized by daily whims of the business or the latest testing fad. And that’s only if testing is actually attempted.
It’s hard to decide what’s worse: the casual and ambiguous nature of testing or the fear it induces in the hearts and minds of marketing teams. When asked, most would still say it’s important to test, suggesting that testing programs are not challenged by lack of perceived value, but rather by the difficulty of the process itself.
So why don’t we clear up the process and help testing make a better name for itself? I thought you’d never ask.
One of the best ways to eliminate testing pandemonium is to formalize the process. Not in a “here are 1,000 forms that should keep you out of our hair for awhile” way, but in a “let’s simplify the testing approach so we don’t actually hate it” way. Queue the Customer Scientific Method. Using the same basic concepts of the scientific method, the “test and learn” steps are straightforward and will remove the debate about results.
STEP 1: ASK A QUESTION
The Customer Scientific Method starts with an observation about customer behavior – behaviors we want to replicate or behaviors we want to change. A well-designed question should be:
- Insight-Driven – any source of customer insight will count!
- Relevant – questions must drive meaningful changes to the business.
- Actionable – answers must be able—and likely—to drive change.
- Divergent – allow curiosity and speculation to drive innovation.
STEP 2: DEVELOP A HYPOTHESIS
While this step is likely to induce eye rolling, it’s critical and frequently taken for granted or skipped altogether. Failure to capture a measurable hypothesis can lead to misleading results or wasted tests. Hypotheses should therefore be structured to include both an action and a result. Quite simply:
STEP 3: CHALLENGE THE HYPOTHESIS
This is the part of testing that should be the most fun. Who doesn’t like to try to prove something wrong? Of course, to really win an argument, you need undisputable facts. That means designing a course of action that will yield objective, unbiased data designed to validate or invalidate the hypothesis. Beyond traditional A/B testing, don’t forget to consider customer analytics, data studies, surveys, ethnography and academic research—it all counts!
STEP 4: ANALYZE THE DATA, DRAW A CONCLUSION AND VALIDATE THE RESULTS
This part of testing is where marketing should consider partnering with a data science or analytics group. Be critical! Enlist additional reviewers, especially those who may be skeptics, and respond to their challenges. The best conclusions are those that have been vetted by supporters and detractors alike.
If your experiment validated your hypothesis, congratulations! Take a moment and reflect on your success. After action reviews are often scheduled after projects that fail, but they rarely are scheduled when things go well. In the event that the hypothesis is not validated, a “failed” test can still provide valuable insight (even if the insight is to avoid doing something like that again). As Thomas Edison once said, “I have not failed 700 times. I’ve succeeded in proving 700 ways how not to build a light bulb.”
STEP 5: COMMUNICATE THE RESULTS
As a last step, it is critical that results are shared. Findings cannot influence future decisions if they are not known. This means more than emailing a spreadsheet of test and control data. In fact, it usually means NOT emailing that spreadsheet. Rather, results should be compiled in an easy to understand, concise template that is used from one test to the next.
Implementing the Customer Scientific Method within an organization can be an experiment itself. Try it with one hypothesis, summarize the results, and refine the process for the next experiment. The key is to start doing something sooner, rather than later, as successful tests will begin to generate support for the process. That is the beauty of this method—the act of using it at all will add to its value over time.
Customer Scientific Method
BY BROOKE NIEMIEC – VP, MARKETING STRATEGY
I think it’s time we give the phrase “test and learn” its proper recognition. Too often today, I hear it being said—and implemented—with reckless abandon. Tests are conducted on an ad hoc basis, sometimes with controls and sometimes without, often prioritized by daily whims of the business or the latest testing fad. And that’s only if testing is actually attempted.
It’s hard to decide what’s worse: the casual and ambiguous nature of testing or the fear it induces in the hearts and minds of marketing teams. When asked, most would still say it’s important to test, suggesting that testing programs are not challenged by lack of perceived value, but rather by the difficulty of the process itself.
So why don’t we clear up the process and help testing make a better name for itself? I thought you’d never ask.
One of the best ways to eliminate testing pandemonium is to formalize the process. Not in a “here are 1,000 forms that should keep you out of our hair for awhile” way, but in a “let’s simplify the testing approach so we don’t actually hate it” way. Queue the Customer Scientific Method. Using the same basic concepts of the scientific method, the “test and learn” steps are straightforward and will remove the debate about results.
STEP 1: ASK A QUESTION
The Customer Scientific Method starts with an observation about customer behavior – behaviors we want to replicate or behaviors we want to change. A well-designed question should be:
STEP 2: DEVELOP A HYPOTHESIS
While this step is likely to induce eye rolling, it’s critical and frequently taken for granted or skipped altogether. Failure to capture a measurable hypothesis can lead to misleading results or wasted tests. Hypotheses should therefore be structured to include both an action and a result. Quite simply:
STEP 3: CHALLENGE THE HYPOTHESIS
This is the part of testing that should be the most fun. Who doesn’t like to try to prove something wrong? Of course, to really win an argument, you need undisputable facts. That means designing a course of action that will yield objective, unbiased data designed to validate or invalidate the hypothesis. Beyond traditional A/B testing, don’t forget to consider customer analytics, data studies, surveys, ethnography and academic research—it all counts!
STEP 4: ANALYZE THE DATA, DRAW A CONCLUSION AND VALIDATE THE RESULTS
This part of testing is where marketing should consider partnering with a data science or analytics group. Be critical! Enlist additional reviewers, especially those who may be skeptics, and respond to their challenges. The best conclusions are those that have been vetted by supporters and detractors alike.
If your experiment validated your hypothesis, congratulations! Take a moment and reflect on your success. After action reviews are often scheduled after projects that fail, but they rarely are scheduled when things go well. In the event that the hypothesis is not validated, a “failed” test can still provide valuable insight (even if the insight is to avoid doing something like that again). As Thomas Edison once said, “I have not failed 700 times. I’ve succeeded in proving 700 ways how not to build a light bulb.”
STEP 5: COMMUNICATE THE RESULTS
As a last step, it is critical that results are shared. Findings cannot influence future decisions if they are not known. This means more than emailing a spreadsheet of test and control data. In fact, it usually means NOT emailing that spreadsheet. Rather, results should be compiled in an easy to understand, concise template that is used from one test to the next.
Implementing the Customer Scientific Method within an organization can be an experiment itself. Try it with one hypothesis, summarize the results, and refine the process for the next experiment. The key is to start doing something sooner, rather than later, as successful tests will begin to generate support for the process. That is the beauty of this method—the act of using it at all will add to its value over time.
Tags: