New, Buzzy Cochrane Study Sets the “Fools Gold” Standard for Anti-Maskers | naked capitalism

By Lambert Strether of Corrente.

“One does not consider the brand to be the guarantee of quality. For us, the quality guarantees the brand.” –Lord Peter Wimsey in Dorothy Sayers, Murder Must Advertise

I didn’t expect to have to drag out my yellow waders for a study from a prestigious brand like Cochrane, but here we are. What is a Cochrane study? A publication of the Cochrane Library, an important institutional player in the field of Evidence-Based Medicine (EBM), whose methods are, in the main, standardized and rigorously enforced,(For EBM, see KLG’s comprehensive takedown here). Vox explains:

With something as complex as masks and respiratory disease, the right tool for the job isn’t one study but many: a meta-analysis of a number of different smaller studies to determine what effect is detectable in aggregate across the body of research.

For medical questions, a major source of high-quality, reputable meta-analyses is Cochrane, a UK-based nonprofit that publishes long and comprehensive meta-analyses of current evidence on medical and therapeutic interventions. It’s a good place to go if you’re wondering if antidepressants work, if blood pressure medications help, if therapy does anything, and lots more. Cochrane reviews are frequently called the “gold standard” for evidence-based medicine.

(“Gold standard,” oft-repeated in the punditry. Hence the headline.) What then is a meta-analysis? From the Cochrane Library itself:

If the results of the individual studies are combined to produce an overall statistic, this is usually called a meta-analysis. Many Cochrane Reviews measure benefits and harms by collecting data from more than one trial, and combining them to generate an average result. This aims to provide a more precise estimate of the effects of an intervention and to reduce uncertainty.

Sadly, in the study we are about to consider — “Physical interventions to interrupt or reduce the spread of respiratory viruses” (January 30, 2023), the selection of trials for the meta-analysis is dubious (all Randomized Controlled Trials, or RCTs), the epidemiology is bad, and various other Cochrane standards have not been met. (Hereinafter, I will call this study “Conly’s Cochrane Study” or “Conly” after the corresponding author, who in my view is the driving force because of his institutional power. We will consider authorship criteria for Cochrane studies below.) While “Conly” itself is cautious in its conclusions — especially if you know how to read Cochrane studies — the uses to which “Conly” has been put by polemicists — uses which some of the authors, being polemicists themselves, must surely have known — are not at all cautious. Bret Stephens, opinionating in The New York Times, was a typical anti-mask voice: “Opinion: The mask mandates did nothing. Will any lessons be learned?”

I did say that “Conly” was “buzzy.” For your delectation, here is a spectrum of opinion, from right to left: Brownnosestone Institute (Robert Malone) (February 2), Reason (February 8), Daily Mail Peter Hitchens (February 18), Spectator (February 10), Bret Stephens, New York Times (February 21), Becker’s Hospital Review (February 10), The Atlantic (February 16), Slate (February 12), The Conversation (February 6), Vox (February 22), Los Angeles Times (February 24), @trishgreenhalgh (February 26), and WSWS (February 22). All this pieces have varying degrees of ferocity ranging from anti-mask triumphalism through colorable fair-minded-ness to — being fair — conforming to my priors starting with The Conversation (masking should be part of a model of “layered protection” against airborne viruses — SARS-CoV2 and whatever comes next — implemented at both the personal and societal levels). The Greenhalgh thread is masterful; consider reading it in full.

Here is the money quote from the first (lead) author, Tom Jefferson, interviewed here, and propagated in the Times by Bret Stephens:

There is just no evidence that [masks] make any difference. Full stop. My job, our job as a review team, was to look at the evidence, we have done that.

The more tendentious studies and triumphalist reaction pieces (Brownnosestone Institute through Slate) either agree with Jefferson, or go into “Well, he has a point” mush-mouth mode. But Jefferson doesn’t have a point. Remarkably, his oft-propagated talking point is contradicted by his own study. From Conly’s Cochrane Study:

“In summary, more high-quality RCTs are needed to evaluate the most effective strategies to implement successful physical interventions in practice, both on a small scale and at a population level.”

“More studies needed” ≠ “no evidence.” (We will address the Cochrane Library’s RCT fetish — there’s no other word for it — below.) More from Conly:

The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions>.

“Hampers drawing firm conclusions” ≠ “no evidence.” (There are many other quotations floating around about “confidence,” which we will address below.)

So, even if we waded no further into Conly, you, the critically-thinking reader, have at least one litmus test when following the discourse: Anybody who quotes Jefferson approvingly on “no evidence,” “full stop” hasn’t read the study, and probably doesn’t know the field.

That said, and with regret, I must pull on my yellow waders and go through Conly in detail. I will show that:

1. Conly’s Cochrane Study Fails as Scientific Communication
2. Conly’s Cochrane Study Fails as Science
3. Conly’s Cochrane Study Fails as Epidemiology
4. Conly’s Cochrane Study is of Dubious Provenance

Section 4 is long, detailed, and unique, but if you, dear readers, have ever participated in a brutal game of office politics, whether corporate or academic, you will enjoy the twist and turns. It’s gonna be fun, OK?

Conly’s Cochrane Study Fails as Scientific Communication

This issue is not particular to Conly, but a failure to translate Conly’s jargon on “certainty” into terms easily understood by lay persons. A usage example from Conly’s Conclusion:

There is uncertainty about the effects of face masks. The low to moderate certainty of evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect.

But what does Cochrane (hence Conly) mean by “low to moderate certainty”? IM Doc, via email, explains:

When people, even very educated people, outside medicine hear “low-level confidence” or “low-level certainty” they naturally think that it is actually low-level. But that is not at all what these meta scientists mean.

A much much better way of doing it that everyone would understand – is doing it on a 1 to 100 scale. It would not be that hard to do.

Gravity is 100 It is certain.

A flat earth is 0 – It is not certain.

Both have overwhelming and enduring evidence of their position as 0 or 100.

Place your calculations of your findings in between.

What I am saying is if you did something like the masking study on a scale — my guess (and I am guessing — it was a bit of a very sloppy paper) — would be they would call their conclusions about an 85. They are thinking it is 85. The man on the street who does not understand the verbiage would hear “low level confidence” and think 25. Total confusion all the time. They even confuse themselves in conferences quite frequently.

I restated this as follows:

So on “low level certainty,” Cochrane types are mentally measuring downward from 100, and lay people up from zero.

So to a professional, 85 would be “low level,” which a lay person would consider “high level” (which it would be, since an 85 would be good to incorporate into one’s personal protocol, especially for a strategy of layered protection, and even more so for a low cost/no risk measure like masking). However, the lay person does not in fact do this, because when they hear “low level” they think 10 or 20, not 85.

This is a generic failure in scientific communication for Cochrane, not a particular failure for Conly. Nevertheless, a failure it is.

Conly’s Cochrane Study Fails as Science

Conly fails because it considers RCTs only, excluding “mechanistic” evidence, which is often superior. From BMJ, “Adapt or die: how the pandemic made the shift from EBM to EBM+ more urgent” (where EBM+ includes mechanistic evidence):

Evidence-based medicine (EBM’s) traditional methods, especially randomised controlled trials (RCTs) and meta-analyses, along with risk-of-bias tools and checklists, have contributed significantly to the science of COVID-19. But these methods and tools were designed primarily to answer simple, focused questions in a stable context where yesterday’s research can be mapped more or less unproblematically onto today’s clinical and policy questions. They have significant limitations when extended to complex questions about a novel pathogen causing chaos across multiple sectors in a fast-changing global context. Non-pharmaceutical interventions which combine material artefacts, human behaviour, organisational directives, occupational health and safety, and the built environment are a case in point: EBM’s experimental, intervention-focused, checklist-driven, effect-size-oriented and deductive approach has sometimes confused rather than informed debate.

Putting it very politely. More:

While RCTs are important, exclusion of other study designs and evidence sources has been particularly problematic in a context where rapid decision making is needed in order to save lives and protect health. It is time to bring in a wider range of evidence and a more pluralist approach to defining what counts as ‘high-quality’ evidence. We introduce some conceptual tools and quality frameworks from various fields involving what is known as mechanistic research, including complexity science, engineering and the social sciences. We propose that the tools and frameworks of mechanistic evidence, sometimes known as ‘EBM+’ when combined with traditional EBM, might be used to develop and evaluate the interdisciplinary evidence base needed to take us out of this protracted pandemic.

And from the New England Journal of Medicine, “Evidence for Health Decision Making — Beyond Randomized, Controlled Trials“:

Elevating RCTs at the expense of other potentially highly valuable sources of data is counterproductive. A better approach is to clarify the health outcome being sought and determine whether existing data are available that can be rigorously and objectively evaluated, independently of or in comparison with data from RCTs, or whether new studies (RCT or otherwise) are needed.

The classic case of using an RCT where a mechanistic approach is superior is from the BMJ: “Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial.” Here is another example:

And another:

In other words, if we have mechanistic evidence that masks “work,” then masks work:

We don’t need RCTs to solve engineering problems, and whether masks “work” is an engineering problem. It follows that if masks do not work, then the failure is at the level of social relations (“populations”), and those are very difficult to evaluate using RCTs, because there are too many confounders:

But as the Bearded One did not quite say: “Cochrane have only interpreted the populations, in various ways. The point, however, is to change compliance.” And we have well-recognized ways to do this, starting with public relations. (Brownstone’s anti-masking campaign is itself enforcing a form of compliance, against mask-wearing, aided by smearing and shaming from The New Yorker, the New York Times, and sundry other liberal organs). Influencers and celebrities are available. Political figures could model masking appropriately. Mask innovation could be funded. In hospitals, we have entire departments devoted to infection control. Even the CDC, corrupt and incompetent though it is, could weigh in. It’s beyond absurd to claim that masking compliance at the population level cannot be maintained as it was at the beginning of the pandemic. Asia does, after all. And here in the grimmer United States, at least we enforce compliance on cancer stick aerosol-producers in public.

Very difficult, but not impossible:

The differences in incidence were relatively small, but see below at “small gains in protection compound.” (Note that Conly’s abuse of the Abuluk RCT was so appalling that some were led to call for a retraction.)

Conly’s Cochrane Study Fails as Epidemiology

First, Conly fails as epidemology because it adheres to droplet dogma instead of aerosol tranmission. From Osterholm et al., The Center for Infectious Disease Research and Policy:

The Cochrane review by Jefferson et al [here, Conly] states that respiratory viruses spread as follows: “People infected with a respiratory virus spread virus particles into the air when they cough or sneeze. Other people become infected if they come into contact with these virus particles in the air or on surfaces on which they land.”

This is the classic definition of droplet transmission, which focuses solely on symptomatic coughing or sneezing that produce large droplets propelled into the face of someone nearby. This review was also focused on contact transmission, which has been ruled out for SARS-CoV-2 by scientists and public health authorities like the CDC.

There is no mention of airborne or aerosol transmission, the former being defined as inhalation of “droplet nuclei” at long distances from a source, and the latter representing a more up-to-date understanding of infectious particle inhalation both near and far from a source. An RCT by MacIntyre et al showed that even for infections assumed to be droplet transmitted, N95 FFRs prevent infection, while surgical masks do not.32 This again points to the droplet paradigm being incorrect.

And:

The Cochrane review must be considered in the context of the authors’ biases. The authors focused on studies in healthcare settings that compare the “standard of care” for droplet transmission—ie, surgical masks—with an N95 or P2 FFR. However, for scientists who understand that aerosol transmission is an important and most likely the primary mode of transmission for respiratory viruses, a surgical mask would never be the standard of care.

Second, Conly fails as epidemiology because it fails to consider that small gains in protection compound over the life of a pandemic. Trisha Greenhalgh explains:

Conly’s Cochrane Study is of Dubious Provenance

The Cochrane library has rigorous standards for authorship, based on recommendations from the International Committee of Medical Journal Editors (ICMJE). If you will glance again at Conly, you will see a list as carefully ranked and ordered as any procession of Bishops and Cardinals, or any military tatoo: “Tom Jefferson, Liz Dooley, Eliana Ferroni, Lubna A Al-Ansary, Mieke L van Driel, Ghada A Bawazeer, Mark A Jones, Tammy C Hoffmann, Justin Clark, Elaine M Beller, Paul P Glasziou, ✉️ John M Conly.” In this case, the authorship roles we are about to examine are Unlisted Author (Carl Heneghan), First Author (Tom Jefferson, the dude who misstated the conclusions of his own study), and Corresponding Author (John M. Conly at last enters, stage right). Each author is dubious, each in their own way.

Let us begin with Carl Heneghan, who clearly states he is an Unlisted Author for Conly’s Cochrane Study:

The only possibly interpretation of “we” is “we,” the authors of Conly. The difficulty here is that Cochrane Library authorship standards require unlisted authors to be credited. From ICMJE:

All those designated as authors should meet all four criteria for authorship, and all who meet the four criteria should be identified as authors. Those who do not meet all four criteria should be acknowledged—see Section II.A.3 below.

And from that section:

Contributors who meet fewer than all 4 of the above criteria for authorship should not be listed as authors, but they should be acknowledged. Examples of activities that alone (without other contributions) do not qualify a contributor for authorship are acquisition of funding

Heneghan is not so acknowledged on Conly’s “Declarations of Interest” page. (Nor does Heneghan appear in a relevant Google hit.) I don’t know if this calls for a retraction, unlike Conly’s abuse of Abaluk’s Bangladesh masking RCT, but it certainly calls for a correction.

Now let us turn to Tom Jefferson, who plays the role of First Author (because he is listed first; ICMJE does not formally define the role of “first” or lead. In scientific publishing generally, the first author is considered to have made the greatest intellectual contribution to the work). Cochrane Library authors must meet certain standards for transparency, distinct from ICMJE:

Although Cochrane has adopted many of the International Committee of Medical Journal Editors (ICJME) recommendations on disclosing financial and non-financial relationships and activities, and conflicts of interest, this policy differs from that of many journals in the following ways:

• some types of financial sponsorship and support are forbidden;
• anyone involved in the creation of Cochrane Library content must disclose their conflicts of interest at the earliest opportunity in the editorial process: for Cochrane Reviews this would be at title registration stage;
• the first and last authors and at least two-thirds of the author team must be free of relevant financial conflicts of interest;
• authors must declare relevant non-financial interests.

The difficulty here is that, along with Unlisted Author Carl Heneghan, listed First Author Tom Jefferson writes for the Brownstone Institute (team effort?). I won’t say “conflicted”_{MR SUBLIMINAL But feel free to think it}. What I will say is that because the Brownstone Institute is funded by dark money (“their donor breakdown is unclear“), Cochrane cannot be certain that Heneghan and Jefferson are free from conflict. For example, here is how Jefferson (“TJ”) declared his interests:

TJ: reports declaring an opinion on the topic of the review in articles for popular media

I submit this declaration is wholly inadequate for determining Jefferson’s place in the following web of interest/Flexnet centered on the Brownstone Institute:

Again, I don’t know if Jefferson’s declaration calls for a retraction, but it certainly calls for a correction.

Finally, we come to John M. Conly, Corresponding Author (always referred to as “John M. Conly”). He’s quite the character! Pressed on the putative harms of masking — as opposed to taking precautions not to inhale or transmit an airborne lethal pathogen, mind you — he comes up with…. acne:

(I’m not denying acne is a harm to those who suffer from it, but surely there are workarounds and accommodations?) John M. Conly has also, from his perch at WHO and in Canada’s public health establishment, vociferously opposed recognition that #COVIDisAirborne:

Conly is one of Canada’s most experienced infectious diseases experts who was once head of the department of medicine at the University of Calgary and the medical director for infection prevention at Alberta Health Services.

He also holds considerable global influence in the pandemic as the chair of the WHO’s Infection Prevention and Control Research and Development Expert Group for COVID-19, which makes key decisions on the research that informs the WHO’s recommendations.

“We absolutely know there are situational airborne settings,” he added during the panel discussion. “But to be able to say that it is the only and predominant means, I think we need better scientific evidence.” [as of

Conly was quoted as saying this on Apr 21, 2021. “Ten scientific reasons in support of airborne transmission of SARS-CoV-2” was published in The Lancet on April 15, 2021, and summarized a great mass of already existing evidence. Call me crazy, but I think “one of Canada’s most experienced infectious diseases experts” should be capable of keeping up with the literature. More:

The WHO has been criticized in the past for its reluctance to acknowledge aerosol transmission — or microscopic airborne particles — as a primary driver of the pandemic, and experts say Conly is at the heart of the issue within the organization.

“Frankly, I think he just can’t admit he’s wrong,” said Linsey Marr, an expert on the airborne transmission of viruses at Virginia Tech in Blacksburg, Va.

“He does carry a lot of weight with WHO, and unfortunately I think his thinking is still stuck in what we knew 20 or 30 years ago and hasn’t updated with what we’ve learned since then — and especially what we’ve learned over the past year.”

John M. Conly plays the role of Corresponding Author, defined by Cochrane Library as follows:

The corresponding author is the one individual who takes primary responsibility for communication with the journal during the manuscript submission, peer-review, and publication process…. The corresponding author should be available throughout the submission and peer-review process to respond to editorial queries in a timely way, and should be available after publication to respond to critiques of the work and cooperate with any requests from the journal for data or additional information should questions about the paper arise after publication.

In this case, Corresponding Author = gatekeeper. I submit that a person who not only does not “follow the science” on airborne transmission but tendentiously opposes it is not fit to play this role.

Cochrane defines conflict of interest as follows:

A conflict of interest is defined as a set of conditions that pose a risk that professional judgement concerning a primary interest (such as patients’ welfare or the validity of research) can be unduly influenced (consciously or unconsciously) by a secondary interest (such as financial gain).

The aims of this policy are to identify, prevent, or manage conflicts of interest.

Dubious Unlisted Author Heneghan must be credited so that readers of the Conly study can determine whether he is conflicted or not. Both Heneghan and Jefferson should declare their dubious affiliation with dark-money funded Brownstone Institute, unless Cochrane Library wants to open a door that any other conflicted author could drive a truck through. Dubious Corresponding Author John M. Conly should resign, to be replaced by someone qualified to play the role.[1] Having a droplet goon like John M. Conly be the gatekeeper for a masking paper is like having an phlogiston-advocating alchemist handle correspondence from Lavoisier. Osterholm et al. called this “bias,” and they’re right.

Conclusion

In three sentences: Anti-mask triumphalism is the grift that keeps on grifting. RCTs are not any kind of “gold standard,” nor are meta-studies based on them. And Cochrane should fix the authorship issues with “Evidence for Health Decision Making — Beyond Randomized, Controlled Trials” so readers can tell who’s conflicted. Cochrane’s branding is “Trusted evidence. Informed decisions. Better health.” Will they need to rethink this?

[Pulls of yellow waders] Oof! Now let’s go save some lives!

NOTES

[1] I don’t view any of this as ad hominem, and in any case I’m not sure ad hominem is so very bad, if used judiciously as a heuristic:

In cases of conflict over paradigms — droplets vs. aerosols, for example — we should give consideration to the possibility that an accumulation of symbolic capital is being defended by the hegemon (see Greenhalgh on Bourdieu here). Indeed, it would be an “extraordinary claim,” albeit implicit, to blithely assume it was not.