BY BROOKE NIEMIEC – CHIEF MARKETING OFFICER
In a previous blog post, I tackled the concept of testing through the Customer Scientific Method. While the process itself might seem fairly straightforward, I continue to see the execution of the process fail because the wrong people are involved. It’s not enough to have bodies thrown at a testing initiative—they have to be the right bodies.
In many cases, leaders simply throw any person with excess capacity to help solve the problem of testing. In others, leaders try to create project teams based on team members’ ability to get along with each other. Or, individuals may be assigned to testing because of some perceived strength in science, or math, or email design (take your pick).
In any of these scenarios, one of the biggest challenges is that the testing team ends up being comprised of individuals who are all like each other. This is because companies and leaders tend to hire people like themselves. Although this is not a bad thing (and we encourage consideration of cultural fit in hiring decisions), it tends to result in a very singular skillset. For testing specifically, many teams end up with an organizational skew toward either ideation or execution. In practice, this either results in lots of ideas that never get done, or a few well-executed tests that lack strategic vision or boundary-stretching ideas.
When reflecting on the Customer Scientific Method, it seems that different types of people are likely to be better at different types of activities across a testing program. The conclusion, then, is that the best test and learn teams consist of a mix of personality types and skillsets, each of which are emphasized at different points in the process.
Let’s revisit the steps of the Customer Scientific Method from a team participation and skillset perspective.
STEP 1: ASK A QUESTION
During the ideation stage, test and learn team members will be responsible for reviewing various sources of data for inspiration, compiling results, and ultimately defining the question to be asked. These people must be naturally curious, but must also possess an ability to synthesize various inputs into a simple and articulate question to be answered. Although many people are good at observing things about their customers, narrowing down a robust list of questions into a single question that gets to the core goal of explaining customer behavior is a more refined skill. While ideation can be fun, it is imperative that this group of individuals can whittle things down to a question that matters—one that, when answered, can be acted upon to drive the business forward.
STEP 2: DEVELOP A HYPOTHESIS
Hypothesis generation is best conducted by the type of person who is very good at speculation and making educated guesses from available data. If you’ve ever known someone who can project a trend when given a single data point, that’s the person you need at this stage. This role requires a level of comfort with ambiguity, an ability to think innovatively about what might be driving behavior, and a sense for what might be able to change that behavior. A solid understanding of testing concepts is also valuable here, as the designed test must be structurally valid, actionable, measurable, and able to directly validate or refute the proposed hypothesis. Attention to detail is important, as a small error in definition of the control group, for example, can invalidate the entire test. Make sure these people do, in fact, sweat the small stuff.
STEP 3: CHALLENGE THE HYPOTHESIS
This is the step where the thinkers turn the reigns over to the doers. A good working partnership between these two groups is critical for a successful handoff, as those responsible for Step 3 will need to execute the test exactly as designed. This is a group who is also detail oriented, who excels at project management, and, quite simply, is the type of person who can just get stuff done. Because the design and execution to plan is essential to the Customer Scientific Method, the team supporting this step must also be comfortable playing by the rules and working within clearly defined boundaries.
STEP 4: ANALYZE THE DATA AND DRAW A CONCLUSION
To state the obvious, those responsible for analysis should be analytical by nature. Their ability to understand and interpret results must be unquestionable, and they must have a strong familiarity with the business rules that should be used to determine whether the hypothesis was validated by the test.
Not perhaps quite as obvious is the need for this role to have excellent visual communication skills. Typically, decision-makers aren’t interested in diving deep into Excel workbooks loaded with supporting data, so the visualization of data in a way that quickly supports the primary conclusion is an important asset for a test and learn program.
Further, this team must not only be able to analyze data, but they must also be able to draw conclusions with implications. One of the significant challenges often faced by leaders of analytical people is convincing them to make a recommendation based on the findings. If that skillset is not available on the analyst team, then it might be important to pair them up with someone who is good at interpreting data and driving to a decision or proposed course of action.
STEP 5: COMMUNICATE THE RESULTS
It is perhaps also obvious to suggest that the team responsible for communicating results should be good communicators. This includes an ability to document findings in a way that clearly explains the results and an ability to socialize or present the results with key stakeholders. This is more about identifying and maintaining channels of communication across the organization than it is about sending an email to a pre-defined distribution list. Relationship management, and consistency in communication will ensure that this process yields meaningful results and changes the performance of the business overall.
Given the number of different skills that it takes to staff an ideal test and learn team, the goal is to structure the team so there is at least one individual who can effectively take the lead at each stage in the process. If certain individuals possess more than one strength, it is more than acceptable to have them participate in or lead more than a single step. However, to avoid potential conflicts arising from role clarity issues, leaders for each stage should be defined and communicated across the entire test and learn program from its inception. The ultimate goal is a balanced team that can work in partnership to ensure a solid flow of ideas, well-designed tests, and thorough analysis that identified key opportunities to improve ROI.
The Test And Learn Team
BY BROOKE NIEMIEC – CHIEF MARKETING OFFICER
In a previous blog post, I tackled the concept of testing through the Customer Scientific Method. While the process itself might seem fairly straightforward, I continue to see the execution of the process fail because the wrong people are involved. It’s not enough to have bodies thrown at a testing initiative—they have to be the right bodies.
In many cases, leaders simply throw any person with excess capacity to help solve the problem of testing. In others, leaders try to create project teams based on team members’ ability to get along with each other. Or, individuals may be assigned to testing because of some perceived strength in science, or math, or email design (take your pick).
In any of these scenarios, one of the biggest challenges is that the testing team ends up being comprised of individuals who are all like each other. This is because companies and leaders tend to hire people like themselves. Although this is not a bad thing (and we encourage consideration of cultural fit in hiring decisions), it tends to result in a very singular skillset. For testing specifically, many teams end up with an organizational skew toward either ideation or execution. In practice, this either results in lots of ideas that never get done, or a few well-executed tests that lack strategic vision or boundary-stretching ideas.
When reflecting on the Customer Scientific Method, it seems that different types of people are likely to be better at different types of activities across a testing program. The conclusion, then, is that the best test and learn teams consist of a mix of personality types and skillsets, each of which are emphasized at different points in the process.
Let’s revisit the steps of the Customer Scientific Method from a team participation and skillset perspective.
STEP 1: ASK A QUESTION
During the ideation stage, test and learn team members will be responsible for reviewing various sources of data for inspiration, compiling results, and ultimately defining the question to be asked. These people must be naturally curious, but must also possess an ability to synthesize various inputs into a simple and articulate question to be answered. Although many people are good at observing things about their customers, narrowing down a robust list of questions into a single question that gets to the core goal of explaining customer behavior is a more refined skill. While ideation can be fun, it is imperative that this group of individuals can whittle things down to a question that matters—one that, when answered, can be acted upon to drive the business forward.
STEP 2: DEVELOP A HYPOTHESIS
Hypothesis generation is best conducted by the type of person who is very good at speculation and making educated guesses from available data. If you’ve ever known someone who can project a trend when given a single data point, that’s the person you need at this stage. This role requires a level of comfort with ambiguity, an ability to think innovatively about what might be driving behavior, and a sense for what might be able to change that behavior. A solid understanding of testing concepts is also valuable here, as the designed test must be structurally valid, actionable, measurable, and able to directly validate or refute the proposed hypothesis. Attention to detail is important, as a small error in definition of the control group, for example, can invalidate the entire test. Make sure these people do, in fact, sweat the small stuff.
STEP 3: CHALLENGE THE HYPOTHESIS
This is the step where the thinkers turn the reigns over to the doers. A good working partnership between these two groups is critical for a successful handoff, as those responsible for Step 3 will need to execute the test exactly as designed. This is a group who is also detail oriented, who excels at project management, and, quite simply, is the type of person who can just get stuff done. Because the design and execution to plan is essential to the Customer Scientific Method, the team supporting this step must also be comfortable playing by the rules and working within clearly defined boundaries.
STEP 4: ANALYZE THE DATA AND DRAW A CONCLUSION
To state the obvious, those responsible for analysis should be analytical by nature. Their ability to understand and interpret results must be unquestionable, and they must have a strong familiarity with the business rules that should be used to determine whether the hypothesis was validated by the test.
Not perhaps quite as obvious is the need for this role to have excellent visual communication skills. Typically, decision-makers aren’t interested in diving deep into Excel workbooks loaded with supporting data, so the visualization of data in a way that quickly supports the primary conclusion is an important asset for a test and learn program.
Further, this team must not only be able to analyze data, but they must also be able to draw conclusions with implications. One of the significant challenges often faced by leaders of analytical people is convincing them to make a recommendation based on the findings. If that skillset is not available on the analyst team, then it might be important to pair them up with someone who is good at interpreting data and driving to a decision or proposed course of action.
STEP 5: COMMUNICATE THE RESULTS
It is perhaps also obvious to suggest that the team responsible for communicating results should be good communicators. This includes an ability to document findings in a way that clearly explains the results and an ability to socialize or present the results with key stakeholders. This is more about identifying and maintaining channels of communication across the organization than it is about sending an email to a pre-defined distribution list. Relationship management, and consistency in communication will ensure that this process yields meaningful results and changes the performance of the business overall.
Given the number of different skills that it takes to staff an ideal test and learn team, the goal is to structure the team so there is at least one individual who can effectively take the lead at each stage in the process. If certain individuals possess more than one strength, it is more than acceptable to have them participate in or lead more than a single step. However, to avoid potential conflicts arising from role clarity issues, leaders for each stage should be defined and communicated across the entire test and learn program from its inception. The ultimate goal is a balanced team that can work in partnership to ensure a solid flow of ideas, well-designed tests, and thorough analysis that identified key opportunities to improve ROI.
Tags: