This Research Insight examines statistical methods, better suited to demonstrating that any observed confusion is negligible and therefore lacks material significance.
In a trademark confusion dispute, the defense occupies an uncomfortable statistical position. The plaintiff need only show that its own survey detected an appreciable rate of confusion. However, the defense, if it wishes to do more than criticize the plaintiff’s methodology, must show that confusion is not occurring (or at least below some threshold), a claim that requires demonstrating the absence of an effect rather than the presence of one. This task is more difficult than it first appears, because the tool people tend to reach for, a conventional null-hypothesis significance test comparing a survey’s confusion rate to a control condition, was not designed to support the conclusion the defense actually needs. A non-significant p-value indicates only that the data were insufficiently informative to reject the hypothesis of no difference. It does not indicate that no difference exists, and it does not rule out that the survey did a poor job capturing a real effect, either because of flaws in the survey itself (e.g., poor stimuli or dependent variables) or because the survey was underpowered to detect it due to sample size limitations. Again, to stress, failing to reject the null is not equivalent to evidence for the null.
This paper reviews four statistical tools that are better suited to the defense’s actual burden: (1) confidence intervals, (2) effect size, (3) equivalence testing, and (4) power analysis. The goal is not to replace substantive criticism of a plaintiff’s survey methodology. Indeed, this may still be the primary line of defense, especially if bias is present that inflates the net confusion level. Rather, it is to give the defense an affirmative statistical case, either with the statistics presented by the plaintiff, or in support of any research conducted by the defense.
Confidence Intervals: Moving Beyond the Mean
The single most easily obtained statistic in a defense-side analysis is the confidence interval (CI) surrounding the observed net confusion rate. A survey reporting “18% net confusion” (i.e., the difference between the test and control conditions) conveys information about the mean, but no information about the possible variability surrounding it. For example, two experts may report an identical net confusion rate of 18%. However, the variance around the mean might vary as a function of sample size (a larger sample size will result in a narrower confidence interval) and/or the values of the test and control (e.g., if the test is 25% and the control is 7%, this will have a wider CI than if the test were 19% and the control were 1%). Suppose we set the threshold for material confusion at 15%. Treating these two results as equivalent, solely because both cross this threshold, discards precisely the information needed to assess the evidence appropriately. For example, using the situation where the test is 25% and the control is 7%, if there are 250 people in each condition, the mean would be 18%, but the CI would range from 11.8% to 24.2%. Hence, from a conservative perspective, the lower bound of the CI is below the stated threshold.
Equivalence Testing: Testing the Range
From the perspective of the defense, one wants to establish that the true confusion rate is sufficiently low to be legally immaterial. That hypothesis can be tested directly using equivalence testing, most commonly implemented as the Two One-Sided Tests (TOST) procedure originally developed to demonstrate bioequivalence between pharmaceutical formulations (Schuirmann, 1987) as well as within psychology as a mechanism for demonstrating that an effect is contained within a pre-specifiedrange (Lakens, 2017).
The logic underlying TOST is easiest to understand through the context in which it originated. A generic drug manufacturer seeking approval is not required to demonstrate that its product has the same effectiveness as the brand-name formulation, but rather that it is within an acceptable window (Schuirmann, 1987). This is a fundamentally different exercise than a standard significance test, and it maps directly onto the task of the defense.
TOST asks whether the data are inconsistent with an effect at least as large as a pre-specified equivalence bound. This is represented by the upper bound (e.g., 15%) and the lower bound (0%). Two one-sided tests are conducted, one testing whether the observed effect is significantly less than the upper bound and the other testing whether it is significantly greater than the corresponding lower bound (a one-sided test is appropriate when the direction of the effect is specified a priori). From the defense’s perspective, ideally one would demonstrate that the mean is both lower than 15% and equal to zero. Given that even the most conservative courts have not used zero as the lower bound, the former is of greater importance.
Effect Size: Distinguishing Statistical Significance from Practical Importance
Consider two weight-loss programs, each with 15,000 participants. At the end of a year, the average weight of the first group is 167.3 lbs. versus 167.9 lbs. for the second (i.e., a little over half a pound). Given that the sample size is so large, it is very possible that this difference might be statistically significant, even though the finding is trivial (i.e., the effect size would be small). To this end, it is always important to keep in mind that statistical significance does not necessarily imply a large effect size. Cohen’s (1988, 1994) consideration of effect size (e.g., Cohen’s h) was developed specifically to address this problem and underscore that statistical significance, in and of itself, does not indicate the size or importance of an effect. A very small effect will reach significance in a sufficiently large sample, hence the phrase “significant but trivial” when referring to a finding that seems important but really is not. There are a number of different effect size estimates available, such as Cohen’s h, d, and f, as well as other metrics that have been developed. These have been designed around different types of data and assumptions in place.
Power Analysis: Could an Effect Be Detected?
Power analysis evaluates whether a survey’s sample size was sufficient to detect an effect of a specified magnitude (e.g., 15% confusion or greater) with high probability, had such an effect existed (Cohen, 1988). A related sensitivity analysis begins with the sample size obtained and determines the magnitude of effect the study had sufficient statistical power to detect. These analyses may be reported alongside a null result to demonstrate that the absence of a statistically significant effect is not simply due to an inadequate sample size. A null result from a study that never possessed adequate power to detect a legally meaningful effect is not informative, regardless of which analytic technique is subsequently applied to it. An opposing expert’s obvious rebuttal to any survey demonstrating confusion that is significantly different from a threshold (e.g., 15%) is likely to be a challenge to whether the sample was large enough to support the stated conclusion. Reporting the study’s power preempts that rebuttal.
Conclusion
None of the tools discussed in this paper (confidence intervals, effect sizes, equivalence testing, and power analysis) are new to statistics. What remains comparatively underused is their systematic application to the defense side of trademark confusion litigation, where the burden of demonstrating a null effect has commonly been through methodological criticism of the plaintiff’s survey, rather than through an affirmative statistical case of relying on data produced by the defense and/or that presented by the plaintiff.
Used together, these tools change the question the defense is able to answer. A confidence interval shows how precisely confusion has been estimated and whether the plausible range of values extends beyond a threshold of legal concern. Effect size expresses that estimate in terms of practical rather than merely statistical importance. Equivalence testing converts the defense’s position into a hypothesis that can actually be tested, namely that confusion falls below a pre-specified bound. Power analysis establishes that the study was capable of detecting meaningful confusion had it been present.
This shift does require discipline. The equivalence bound must be specified in advance and justified by reference to the legal standard rather than selected after the data are in hand. The sample must be large enough for the resulting interval to be informative. Finally, the survey itself must still satisfy the methodological standards on which any confusion evidence depends, because no statistical technique can rescue a study whose universe, stimuli, or questions fail to reflect the marketplace. Within those constraints, however, the potential payoff is present.