Evaluating the validity of clinical outcome assessments: avoiding jingle-jangle fallacies!

07-06-17Andrew Trigg

When selecting a suitable clinical outcome assessment for an upcoming trial, such as a disease-specific patient-reported outcome (PRO) measure, it is important to assess the construct validity of each scale. One way of achieving this is through assessing the convergence of each scale with others hypothesised to measure similar latent constructs. As a practical example, if a new PRO, designed to measure depressive symptoms, highly correlates with an existing PRO putatively measuring depressive symptoms, you would have greater confidence that scores of the new PRO provide a valid reflection of the hypothesised latent construct.

When evaluating convergent validity in the above manner, particularly when reviewing evidence in the literature, a common error to avoid is committing jingle-jangle fallacies (1):

  • The jingle fallacy is the assumption that since two scales are named the same (or highly similar) they must be measuring the same construct
  • The jangle fallacy is the assumption that since two scales are named differently they must be measuring different constructs

Although seemingly obvious, jingle-jangle fallacies are commonly observed pitfalls and important to keep in mind.

For example, the following table shows the correlations between two scales as observed in two studies (2,3):

FACT-G Social/Family Well-being

EORTC QLQ-C30 Social Functioning

0.21 (Silveira et al, 2010)

0.08 (Conroy, 2004)

Since these correlations seem low, you may want to conclude that the EORTC QLQ-C30 Social Functioning scale is not a valid measure of a latent social functioning or well-being construct. If so, you have just committed the jingle fallacy! When comparing each scale at the item (question) level, it becomes clear that the EORTC QLQ-C30 Social Functioning scale is focussed on the interference with social activities or family life due to a persons’ physical condition or treatment, whilst the FACT-G Social/Family Well-being scale contains items assessing perceived support, communication and closeness to family and friends in addition to satisfaction with sex life. Consequently, the two similarly named scales actually assess different underlying conceptual experiences.

The jangle fallacy is observed less frequently, and is prone to occur when an unexpectedly high correlation between two seemingly unrelated scales is observed. It is possible that, although the name of each scale is distinct, the items within these scales have considerable conceptual overlap.

Although first recognised more than 100 years ago (4), it is important to remain mindful of this common pitfall. How can we plan to avoid it? A recent paper (5) in information systems research applied natural language processing algorithms to help identify such fallacies; an interesting approach but the utility of such an approach in patient-centred outcomes research remains to be seen. The utility of the following statement, however, seems intuitive: do not interpret scores simply according to the scale name; take it one step further and look at the individual items and conceptual meaning which contribute to the score before drawing any conclusions.

Here at DRG Abacus we understand the importance of robust patient-centred measurement. To find out more email us at Access@TeamDRG.com

References

  1. Kline RB. Principles and Practice of Structural Equation Modeling (4th ed.). 2015, New York: Guilford Press.
  2. Silveira AP, Goncalves J, Sequeira T, et al. Patient reported outcomes in head and neck cancer: selecting instruments for quality of life integration in clinical protocols. Head and Neck Oncology, 2010 2:32.
  3. Conroy T, Mercier M, Bonneterre J, et al. French version of FACT-G: validation and comparison with other cancer-specific instruments. European Journal of Cancer, 2004 40(15):2243-52.
  4. Thorndike EL. An introduction to the theory of mental and social measurements. 1904. New York: Teachers College, Columbia University.
  5. Larsen KR & Bong CH. A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 2016 40(3):529-51.

© DRG Abacus 2020