When selecting a suitable clinical outcome assessment for an upcoming trial, such as a disease-specific patient-reported outcome (PRO) measure, it is important to assess the construct validity of each scale. One way of achieving this is through assessing the convergence of each scale with others hypothesised to measure similar latent constructs. As a practical example, if a new PRO, designed to measure depressive symptoms, highly correlates with an existing PRO putatively measuring depressive symptoms, you would have greater confidence that scores of the new PRO provide a valid reflection of the hypothesised latent construct.
When evaluating convergent validity in the above manner, particularly when reviewing evidence in the literature, a common error to avoid is committing jingle-jangle fallacies (1):
Although seemingly obvious, jingle-jangle fallacies are commonly observed pitfalls and important to keep in mind.
FACT-G Social/Family Well-being
EORTC QLQ-C30 Social Functioning
0.21 (Silveira et al, 2010)
0.08 (Conroy, 2004)
Since these correlations seem low, you may want to conclude that the EORTC QLQ-C30 Social Functioning scale is not a valid measure of a latent social functioning or well-being construct. If so, you have just committed the jingle fallacy! When comparing each scale at the item (question) level, it becomes clear that the EORTC QLQ-C30 Social Functioning scale is focussed on the interference with social activities or family life due to a persons’ physical condition or treatment, whilst the FACT-G Social/Family Well-being scale contains items assessing perceived support, communication and closeness to family and friends in addition to satisfaction with sex life. Consequently, the two similarly named scales actually assess different underlying conceptual experiences.
The jangle fallacy is observed less frequently, and is prone to occur when an unexpectedly high correlation between two seemingly unrelated scales is observed. It is possible that, although the name of each scale is distinct, the items within these scales have considerable conceptual overlap.
Although first recognised more than 100 years ago (4), it is important to remain mindful of this common pitfall. How can we plan to avoid it? A recent paper (5) in information systems research applied natural language processing algorithms to help identify such fallacies; an interesting approach but the utility of such an approach in patient-centred outcomes research remains to be seen. The utility of the following statement, however, seems intuitive: do not interpret scores simply according to the scale name; take it one step further and look at the individual items and conceptual meaning which contribute to the score before drawing any conclusions.
Here at DRG Abacus we understand the importance of robust patient-centred measurement. To find out more email us at Access@TeamDRG.com