The Danger of Face Validity

Everyone (of my generation, anyway) knows the story of the Van Halen M&M Rider: this was a provision in Van Halen’s touring contract that required each venue to provide the band a large bowl of M&M candies with all the brown ones removed. One reason everyone knows the story is that it so clearly exemplifies what was wrong with rock ’n’ roll in the late 1970s: arrogant rock stars had become used to getting whatever they wanted in whatever amounts they wanted, their most absurd whims catered to by a support system of promoters and managers who were willing to do whatever it took in order to get their cut of the obscenely huge pie. The story was perfect, and it was all too easy to imagine the members of Van Halen, swacked on whiskey and cocaine, howling with laughter as they made their manager add increasingly-ridiculous items to the band’s contracts.

In other words, the standard explanation for Van Halen’s M&M rider — that it was a classic expression of bloated rock privilege — is a hypothesis with a great deal of face validity: it simply makes good intuitive sense, and is therefore easy to accept as true.

Like many hypotheses with a great deal of face validity, however, it turns out to be wrong. Not just imprecise or lacking in nuance, but simply wrong. The reason that the members of Van Halen put the M&M rider into their contract had nothing to do with exploiting their privilege or with an irrational aversion to a particular color of M&M. It had to do with the band’s onstage safety. As it turns out, other provisions of the band’s contract required the venue to meet certain safety standards and provide certain detailed preparations in terms of stage equipment; without these preparations, the nature of the band’s show was such that there would have been significantly increased danger to life and limb. The M&M rider was buried in the contract in such a way that it would easily be missed if the venue’s staff failed to read the document carefully. If the band arrived at a venue and found that there was a bowl of M&M’s in the dressing room with all the brown ones removed, they could feel confident that the entire contract had been read carefully and its provisions followed scrupulously — much more confident than they would have been if they had simply asked the crew “You followed the precise rigging instructions in 12.5.3a, right?” and been told “Yes, we did.”

What does this have to do with scholarly communication? Everything. In scholarly communication (as in just about every other sphere of intellectual life), we are regularly presented with propositions that are easy to accept because they make obvious sense. Sometimes these are accompanied by rigorous data; too often they are supported by sloppy data or anecdotes. Sometimes they aren’t supported at all, but are simply presented as self-evidently true because their face validity is so strong.

In scholarly communication, we are regularly presented with propositions that are easy to accept because they make obvious sense.

A classic example is the citation advantage of open access (OA) publishing. It seems intuitively obvious that making a journal article freely available to all would increase both its readership and (therefore) the number of citations to it, relative to articles that aren’t free. By this reasoning, authors who want not only broad readership but also academic prestige should urgently desire their articles to be as freely available as possible. But the actual data demonstrating the citation impact of OA is mixed at best, and the reality and significance of any OA citation advantage remains fiercely contested (for example, here, here, here, here, here, here, here, and here).

Another example is the impact of Green OA on library subscriptions. It makes obvious sense that as more and more subscription content becomes available for free in OA repositories, subscription cancellations would rise. This is a hypothesis with obvious face validity, and yet despite the steady growth of Green OA over the past couple of decades, there is not yet any data to indicate that library subscriptions are being significantly affected. In my most recent posting in the Kitchen, I proposed that the reason we haven’t seen significant cancellations is that Green OA has not yet been successful enough to provide a feasible alternative to subscription access; others have argued that there is little reason to believe that Green OA will ever harm subscriptions no matter how widespread it becomes. There probably won’t be sufficient data either to prove or to disprove the hypothesis definitively for some time.

Another example of a scholarly communication hypothesis with strong face validity is the proposition that if funders make OA deposit mandatory, there will be a high level of compliance among authors whose work is supported by those funders. Do the available data bear out this hypothesis? Eh, sort of. The most recent analysis of compliance with the Wellcome Trust’s OA requirement found 61% of funded articles in full compliance — not exactly a barnburning rate. In 2012, Richard Poynder determined that the compliance with the National Institutes of Health’s OA mandate was a slightly more impressive (but still not stellar) 75%. As far as I can tell, compliance data are not available from the Gates Foundation or the Ford Foundation, both of which are major private funders of research in the United States and are of course under no obligation to provide such figures publicly. (If anyone has access to compliance data for these or other funder mandates, please provide them in the comments.)

The danger of a false but valid-looking hypothesis increases with the importance of the decisions it informs.

What these three examples suggest is that the face validity of any hypothesis is a poor guide to its actual validity. Some hypotheses with high face validity (like the OA citation advantage) start to buckle under rigorous examination; some (like the impact of Green OA on library subscriptions) may turn out to be valid and may not, but there’s no way to know for certain based on currently-available evidence; for others (like the impact of funder and institutional mandates on authors’ rates of article and data deposit) the supporting data is somewhat mixed. This suggests that deep caution is called for when one encounters a hypothesis that sounds really good — and even more caution is indicated if the hypothesis happens to flatter one’s own biases and preferences.

The current political landscape in the U.S. and Europe has many of us feeling an increasing level of concern about whether important decisions are being made by individuals, by government agencies, and by political leaders in the face of solid and reliable evidence or based simply on what sounds good. Face validity is seductive, which makes it dangerous — and the danger increases with the import of the decision, and with the degree to which the decision-maker is truly relying upon face validity rather than on actual data, carefully gathered and rigorously analyzed.

Autor: Rick Anderson
Twitter: <@LOOPTOPPER>
Fuente: <https://scholarlykitchen.sspnet.org/>