Intercept Studies Can Still Be Useful If…

The lack of fully understanding intercept studies has often resulted in worthless findings. This piece addresses the feasibility of salvaging insights from these studies.

This article is the concluding part of an earlier piece titled, Should You Conduct That Intercept Study? That article addressed the key challenges and limitations of intercept studies: internal and external validity issues. This final piece argues that despite these limitations, a cautionary interpretation, along with a few safeguards, can ensure a useful intercept study. This final piece also takes the position that the continuous execution of intercept studies will pave the way for new approaches to be created and for methodology evolution to occur.

If you have ever conducted an intercept study, or are planning to, you should be commended for striving to ensure the accountability and performance of your digital marketing assets. Intercept studies promise empirical insights on how digital assets impact brand perception and audience conversion. Unfortunately, these studies cannot easily deliver on that promise. This is because the results can neither confidently state that the lift in the metrics resulted from your digital assets (internal validity issues), nor confirm that the respondents in the studies represent your typical audience (external validity issues). Both of these limitations were addressed extensively in the preceding article. If you read the seemingly insurmountable challenges in that article, you may wonder whether the studies are worth conducting at all. Thus, our final question…

Question 3: Is there a useful application of intercept studies, given the limitations of both the internal and external validity?

Yes, intercept studies can still be useful, if users understand their limitations and interpret the findings cautiously. Users must understand the limitations of intercept studies, interpret the findings with care, and commit to investing in the appropriate approaches, rather than finding a cheaper option that may not provide the required results.

Many senior researchers at the organizations that offer these studies (comScore, InsightExpress, Dynamic Logic, Nielsen) are aware of the limitations of intercept studies, but they are also aware of the approaches that can deliver stronger results. The cost of conducting studies that could deliver stronger confirmatory results seems unattractive compared to the supposed directional finding from the quasi-experiment based version. Ensuring that intercepts studies deliver valuable insights is therefore the responsibility of both the executing team and end users. These users—marketers and advertisers (specifically their analytics group)—must be immersed in the study’s details and limitations in order to deploy and interpret the findings correctly. A few suggestions on how to manage the study’s limitations in order to obtain useful results are explained in the following sections.

Managing internal validity challenges
Internal validity pertains to the ability of a study to measure what it was set up to measure, which for experiments such as intercept study, pertains to the ability of the study to confirm that the digital asset caused the observed lift on awareness, consideration, likeability, or purchase lift. The closer your approach mimics experimentation, the better its internal validity.
1. The most important action you can take is to invest in a true experimental study. This is done by randomizing the sampled audience in the test and control groups and using server side rules . All issues relating to a low response, population skew, or motivational difference should theoretically balance out, and the results should be internally valid (but not necessarily externally valid).
2. Where randomization is not feasible, ensure you recruit both the test and control groups at the same time and from the same location (or context). Recruiting a test group from a website and a control from a panel will diminish audience similarity, and thus, the test must rely heavily on weighting, which as we mentioned earlier, is an imperfect solution.

Managing external validity challenges
External validity relates to the ability of a study to be generalize-able to the population being studied (recall the study only evaluates a sample, a small subset of the population). In other words, how can you ensure that your sampling frame covers your intended population?
1. Ensure that a population of at least 250 people are used for your test and control groups, in order to get a precision range of a plus/minus percentage point of 4, with 95% internal confidence.
2. Spread out the duration of your study over at least 2 weeks to ensure that both the test and control groups sample a broad range of your population.
3. Ensure that your control groups are also sampled to reflect the population, or else the behavior of the non-exposed audience (control) will not be representative of the population’s baseline; hence, curtailing external validity.
4. If sampling from a panel, ensure the use of a stratified sampling approach, as this will help minimize sampling errors.
5. Consider implementing helpful tactics, such as incentives and re-contacts (where applicable), to help reduce non-response.

Analysis and interpretation of findings
The most important point raised thus far, is that marketers and advertiser must carefully interpret and apply the findings of non-experimental design intercept studies. You will minimize wastage of marketing funds resulting from unsubstantiated insights if you have a better grasp of the intercept studies and their limitations. The underlying theme is caution.
1. You should familiarize yourself with the general limitations of quasi-experimental designs, and the specific limitations of your intercept study. Interpret causality in these quasi-experimental studies with caution, since there are plausible alternative hypotheses that may explain the observed results.
2. Understand the weighting schemes applied in the study to adjust for imbalances between the test and control groups. Also, know which weighting schemes can be applied to both the test and control groups to bring them to par with the expected distribution in the “true” audience population . In general, be wary of weighting schemes expected to reflect true population when as there is no confirmatory basis for the weights applied.
3. Where possible, compare findings with other studies, including brand tracking studies, though not without its own limitations too–a topic for another day and empirical data, such as sales.

For further ideas on executing your next study appropriately, Interactive Advertising Bureau (IAB) also provides suggestions for executing intercept studies effectively here.

Other alternatives and future promise

Self-reported studies may never depart completely from the analytical marketing tool kit. Marketers and the online research industry will continue to explore alternatives approaches to improving the results of these studies, and to estimate the expected ranges of bias relative to standard experimental designs so that results of intercept studies could be calibrated effectively in the future.

In the future, the core business questions that intercept studies address may be inferable from other data sources. These alternate insights will evolve from the increasingly abundant and aggregate-able user level data sets that have been created by the acceleration of technology. For instance, search and social media behavior could serve as credible proxies of consumer awareness and purchase intent.  Another promising alternative may reside in large aggregation of online user behaviour via persistent cookies (rather than opt-in online panels, which also suffer from self selection bias).  Compiling the cookies into longitudinal online panels will enable the use of longitudinal behavioral tracking with multiple pre-and post-study opportunities.

All said, intercept studies will be around for a while, and the survey firms will continue to find different approaches to increase the credibility and validity of these studies. Marketers, advertisers, and analysts can continue to deploy these studies, but they must understand their limitations, interpret them with caution, and opt for experimental designs rather than quasi-experiments where possible. Treat intercept studies as another research tool kit, and where possible, seek validation through other empirical studies or data sets.