Yves here. KLG below describes how science publications have become much less reliable over time, with the proliferation of “open access” journals part of the problem. As he said via e-mail:
As science has hypertrophied, the bad is supplanted the good as spurious quantification of scientific research has come to rule the practice of science. This has in many ways ruined the practice, the art, and the craft of scientific research. Yes, I am one of those.
This has been facilitated by the rise of open-access, mostly virtual publishing, which was not originally considered a threat. On the contrary, open-access online publishing was expected to be a positive good. And it often is. But it often is not. Predatory publishers were not long in appearing as online publishing became accepted. Neo-predatory, if not outright predatory, “journals” are now the repositories of an ever growing mountain, and a form of Gresham’s Law – the bad drives out the good – has taken over much of the scientific literature, which has become “pay-to-play.”
And the finish is something my colleagues and I have only recently determined. Pay-to-publish has extended its tentacles into medical education, probably with equally deleterious effects.
IM Doc has regularly inveighed against a narrower but more dangerous corrupting influence: the way Big Pharma games drug-related studies and touts (typically) small scale studies that promote off-label uses.
By KLG, who has held research and academic positions in three US medical schools since 1995 and is currently Professor of Biochemistry and Associate Dean. He has performed and directed research on protein structure, function, and evolution; cell adhesion and motility; the mechanism of viral fusion proteins; and assembly of the vertebrate heart. He has served on national review panels of both public and private funding agencies, and his research and that of his students has been funded by the American Heart Association, American Cancer Society, and National Institutes of Health
The continuing COVID-19 syndemic has done nothing if not shaken the foundations of biomedical science, from the basic biology of viral infection and the immune response to the redefinition of what a vaccine should do. Facts that have been well known for more than fifty years have been ignored. For example, lasting immunity to coronaviruses such as SARS-CoV-2 is elusive, whether through previous exposure or vaccination. Nevertheless, Biomedicine went all-in on vaccines that made a lot of money for Big Pharma. Yes, these are said to have prevented millions of deaths, which may be true. But that is not the common understanding or expectation of how vaccines work, and legitimate concerns about the Covid vaccines were largely ignored.
Four years into the current pandemic there are few medical interventions that work well in and of themselves, but clinical management of disease has undoubtedly improved. Given that airborne transmission of SARS-CoV-2 is an established fact now, despite considerable institutional dithering, we know how to prevent the virus from spreading. Effective ventilation/air exchange and the use of functional masks in crowded indoor spaces should be universal. The looming questions now are (1) what of long-Covid for which the only good intervention is prevention of infection in the first place, (2) how will repeated infections with emerging variants of SARS-CoV-2 affect the long-term health of afflicted patients, and (3) when, not if, will a more virulent variant emerge? COVID-19 is not the common cold. We knew in less than four years into the HIV/AIDS epidemic nearly forty years ago how to prevent the spread of HIV. This made all the difference even if there was some resistance to recommendations, from testing the blood supply to nonpharmaceutical interventions that prevented transmission of HIV. Perhaps in the coming months COVID-19 research will begin to provide the needed answers.
My thesis here is the foundations of my profession were already weakened beyond resistance to a global crisis when COVID-19 emerged in late 2019. It has been plain for years that the scientific literature is in distress, from physics to biochemistry to cell biology. The latter are weekly stops for me, and I often do not know what to believe. When I began my career as a scientific worker/scientist, this was not a problem. The literature that I read then was reliable. But that literature also had distinct levels. At the basic but rarely inconsequential end of the spectrum were the “archival” journals, as they were recognized at the time. This is where most science is published, and without this foundation very few of the “breakthroughs” published at the so-called “high end” would be possible.
My career has been contemporaneous with the development of tools to analyze the scientific literature quantitatively. This began with the scientometric [1] work of Eugene Garfield that culminated in the Science Citation Index and the Journal Impact Factor. Without going into detail, the former allows one to track the citations of any given paper in the literature (also available through PubMed). As a crude measure of the importance of a scientist’s work, the citation index has marginal utility. As a practical tool for the working scientist, the Web of Science is useful in keeping track of where her or his work is read and by whom and how it is used as a foundation for further research. The Journal Impact Factor is not nothing, but it is primarily a marketing tool that can be manipulated by publishers to inflate the importance of its products. For example, the high-end (and archival) journal of my primary field of research added “minireviews” to the front matter of each issue about thirty years ago, Citations went up markedly, because these reviews were cited frequently in the introductions to papers that required a current summary of a given field. Good business, yes, but not an original contribution to new knowledge. Still, it is widely but mistakenly believed the journal can be the judge of the science.
In my view the quantitative analysis of the scientific literature has been most effective at providing a mechanism by which the practice of science can be gamed by those willing to confuse “scientific research activities” with scientific research (see below). This has been covered in the literature read by scientists, but the rot at the core is beginning to show up in more visible places, such as an article in The Wall Street Journal last week: Flood of Fake Science Forces Multiple Journal Closures. The backstory is long and a consequence of spurious quantification of scientific “output.” This has led to corruption of the scientific literature. What is new is that legacy publishers such as Wiley are now being affected.
In the case of John Wiley & Sons (established in New York in 1807), the publisher has closed 19 journals under its banner that “were infected by large-scale research fraud” and in the past two years “has retracted more than 11,300 papers that appeared compromised.” As noted in the WSJ, these papers are the product of “paper mills” that specialize in the production of fake science that “for a price, will list a scientist as an author of a wholly or partially fabricated paper. The mill then submits the work, generally avoiding the most prestigious journals in favor of publications such as one-off special editions that might not undergo as thorough a review and where they have a better chance of getting bogus work published.” Indeed, and paper mills are adept at evolving to elude detection. But Wiley is a legacy scientific publisher that could be and should be viewed as a bulwark against sketchy authors who publish the results of sketchy “research activities.” On the other hand, maybe not. Academic publishing is a $30 billion dollar a year “enterprise.” There is a lot of money to be made by someone. [2]
Although outright fraud has been and still remains a vanishingly small fraction of the scientific literature, its persistence, including the gaming of the literature, erodes trust in science while dissolving a foundation that has been built since Philosophical Transactions of the Royal Society was first published in 1665. My question here is whether science is being harmed by a version of Gresham’s Law – Bad Money Drives out the Good: “Whenever coins containing precious metals have been used along with base metal coins of the same denomination, both legally accepted as tender, the bad coins have driven the good coins out of circulation.” A more general description of Gresham’s Law comes from Mackenzie King, longtime Prime Minister of Canada in the first half of the twentieth century:
Something analogous to Gresham’s Law will be found to obtain in the case of competing standards in Industry (or Science). Assuming there is indifference in the matter of choice between competing commodities or services, but that in the case of such commodities or services the labor standards involved vary, the inferior standard, if brought in this manner into competition with a higher standard, will drive it out, or drag the higher down to its level. This is effected by the opportunity of under-selling which comes, where in such cases human well-being is sacrificed to material ends. The superior standard, not being recognized or demanded, is unable to hold its own, and in time disappears. This Law is just as real and relentless in its operation in Industry (Science) as Gresham’s Law of the precious metals is with respect to money and the mechanism of exchange. Indeed, a more accurate exposition would describe both as manifestations of one and the same law, which I propose to call the Law of Competing Standards. I see no reason why economists (scientists) should not recognize the existence of such a law, and incorporate it immediately in (their practice) as being quite as significant as the Law of Supply and Demand, the Law of Diminishing Returns, or any other Law accorded a place in its nomenclature.
Thus, my answer to my own question is an unqualified “yes,” largely because I have seen it with my own eyes during the past 25-30 years. Bad publishing may not be driving out the good, but it certainly overwhelms the good in volume. Not coincidentally, this period has seen the rise of open-access, mostly virtual scientific publishing.
There is absolutely nothing wrong with open-access publishing. A strong case has been made that all scientific research published with public support should be “open to access by everyone.” Much is available, even from for-profit legacy publishers, immediately upon publication or after an embargo of six months to a year. But papers published 50-80-100 years ago by many journals (Science and Nature included) require a subscription or access through a library with a subscription or a one-time payment, with $32 seeming to be a popular price. Absurd, although my subscription to Nature journals available online at $30 per month is a bargain for me.
Along with virtual publication online came predatory publishers not very long after. This is not the place to discuss predatory publishers at length. Some of them may have cleaned up their act, but any scientist of a certain age who pays attention (apparently a distressingly low number) recognizes them on sight, even if the pdf downloaded from one of these journals is identical on the surface to a paper from the Journal of the American Chemistry Society (1879) or Proceedings of the National Academy of Sciences (1915). [3] An archived link to Beall’s List (a good story from an observant librarian that is not covered fairly in this Wikipedia entry) is here. Scroll to the bottom for information on a very prominent provider of open-access, online journals whose products appear often in current PubMed searches, one of which claims to be based in Basel, which is the home of Big Pharma.
A good working definition of “predatory publisher” is a business that establishes journals online (seldom on paper) that will distribute a manuscript for a fee (generally $2,000 to $5,000) and has a very low rejection rate. The latter is difficult to know, but those who submit to these online repositories know what to expect. Reviews are rapid and usually pro forma. I still get the occasional email asking me to send a paper or join the editorial board of a journal I have never heard of. These invitations also tend to advertise “special editions” about something vaguely related to what I have done in the past. Or not. A colleague recently received an invitation, stated exactly: “Inviting Articles from precious author like you – Medical & Clinical Research.” Someone somewhere undoubtedly responded in the affirmative.
So, why is this a problem? Bad science is just that and ultimately inconsequential, while good science will lead to scientific progress, however this is defined, for example in previous posts here and here. The answer comes back to the specious quantification of scientific output by individual scientists and their research groups. Although this may be changing slowly at the margin, the only thing promotion committees, institutional administrators, and sometimes peer review panels at funding agencies seem to be able to do is count: The number of papers published, and the dollars included in a grant award (especially the indirect costs – “overhead” that goes to the institution for its services). The quality of the work is seldom considered. Although this may be changing, too, quality requires judgment and imagination informed by long and deep experience. Neither of these attributes is easily quantified.
In any case, what I describe here as Gresham’s Law as Applied to Science continues to have malign effects. It is clear that much of published science is now done “quick and dirty” with the payoff having little to do with science as we should expect. For example, I repeat myself, but as of 20 May 2024 there are 422,367 results returned in a search of PubMed using “Covid” (case-insensitive) as the query in the 4.5 years since late-2019 (~8,000 per month, ~270 per day, which is absurd). “HIV AIDS” (case insensitive) returns 180,613 results in the 42 years since 1982 (~360 per month). The likely explanation, and the only plausible explanation to me, is that the scientific literature has become so large as to include much that shouldn’t exist. Open access, which is an absolute good in principle, and publication online, another near-absolute good in principle, have facilitated the dilution and subsequent diminution of value in scientific research and publishing. But the numbers are way up, and they can be counted.
The resultant imperatives felt by scientists, including yours truly, are strong but not irresistible. The bad has driven out the good, and this has not been limited to newer, online publications. Many of the early papers in legacy journals (e.g., NEJM) on COVID-19 and vaccines were rushed. In the emergency, that might have been expected but the result has been less than optimum. Publication during the early days of HIV-AIDS was also rapid, with intense rivalries obvious even to the most casual observer. But progress was made with dispatch, especially in identification of HIV as the agent of AIDS (using technology that for COVID took hours to days instead of months to years with AIDS) and the recommendation of nonpharmaceutical interventions that prevented transmission of HIV. Different virus, different mode of transmission, different pathology, although immune system dysregulation is common to both AIDS and Covid. Even so, good adherence to long established precepts of epidemiology and public health were the order of the day in the 1980s.
I realize that I could be taken as an “man yelling at clouds,” but I recently came across another excrescence of the qualitative decline of scientific publication and by extension biomedical science that is having a deleterious effect on medical education. Beginning in 2022 the first board examination taken by medical students (USMLE Step 1), usually at the end of their second year of medical school, was changed from being a scored exam to pass/fail. The reasons for this have never been explained very well to my knowledge, but it will be an interesting story eventually. Before this transition, a student’s Step 1 score was used as an objective, if imperfect, measure of how well a medical student had mastered the material required as the scientific foundation for the practice of medicine (anatomy, biochemistry, systems physiology, pharmacology, etc.). Step 1 was an achievement test, with a very strong correlation between performance in the preclinical curriculum and the Step 1 score. Naturally, a high score on Step 1 became a prerequisite for a student to match into the most competitive, largely because of a dearth of positions, residency programs. Examples include Neurosurgery, ENT, Dermatology, Urology, and Orthopedic Surgery.
This ill-considered change caused an apparent crisis for medical students as they realized that another mechanism would now be required to distinguish themselves from their peers. The answer was, naturally, “research.” A recent article, Medical Students lose in the research arms race for residency slots, by Anmol Shrestha explains:
Before (medical school), I had known publish-or-perish as something for those seeking tenure at universities. I believed that medical school was a place where I would be learning about the human body and learning bedside manner through classes like medical Spanish…But as my first year went on and I saw more and more classmates skip patient-facing classes to spend time conducting research, I realized how much the publish-or-perish environment also applied to medical students like me who are hoping to get into residency programs. [4]
Yes, there are medical students who do serious research as students. Outside of large medical schools with extensive internal research programs funded largely by NIH, this remains rare unless the student is in a program that emphasizes research, such as an MD-PhD Program that includes three years of research between the two preclinical and the two clinical years. It is coming to light that an extensive apparatus has been assembled to fill this perceived need. As several colleagues and I have learned recently, it attracts students even before they begin medical school. The article at the link notes “the average number of publications medical students have under their belts by the end of medical school has skyrocketed…and why applicants for certain specialties, like neurosurgery, average up to 29 research activities – a number higher than many PhD candidates publish by the end of their programs.”
This requires major unpacking. The first thing to notice is that “publications” are conflated with “research activities.” In this case, as my colleagues and I have discovered, these research activities are no more than lines on a curriculum vitae in the vast majority of cases. Publications, such as they exist, are generally case studies, which might be useful, or irrelevant “reviews” put together, not written, by a group of medical students from disparate medical schools in the Americas, Europe, and Asia. They do not seem to be the products of paper mills as defined above, but they are artificial assemblies facilitated by what can only be called “ringleaders.” Few of the publications have been cited or are likely to be cited more than less than a handful of times in the future. Yet they exist and can be counted in PubMed, which is their only reason for being.
As for 29 research activities being more than most PhD students will list as graduate students, yes, that is laughably true. An outstanding graduate student in an outstanding institution who is blessed with a project equal to her abilities will publish 3-5 papers and present her work at another 5-6 scientific meetings as a student (4-5 years in the US). Most will be credited with less, including that outstanding student with a difficult project. “Research activities” have little meaning in graduate school but not medical school. Gresham’s Law has reached into medical education as an unintended but perfectly predictable consequence of the brave new pass/fail world. In any case, a Residency Program Director who believes a precocious medical student has legitimately published, as a student, between 10 and 25 entries indexed in PubMed deserves what is coming. It is not clear that the resident’s future patients deserve it, though, whatever the count is.
This is all still playing out, but many medical students/graduates have some explaining to do, since some have participated in these “research activities” as free agents without the knowledge of their institution. This is unacceptable, and I confess to never imagining this could be a thing. How has this happened? The current climate of rapid, open-access, online publication with cursory peer review opened the gates for those who can pay the submission fees, which are seldom waived as far as I can tell. I anticipate this will be a big mess, to use a not-so-technical term, the resolution of which may be ugly.
Finally, my recommendation to anyone, scientist and nonscientist alike, who reads the scientific literature is caveat emptor. When a group of collaborators seems too disparate to have actually worked together, this is probably true and an alarm should ring. And if even one author uses an email address other than an institutional email, stop reading. That goes for all scientific papers, by the way, unless the author(s) explicitly state that their work is their own, alone, and unaffiliated with their home institution (a very rare event).
In the meantime, it will be necessary to go back to the future and stop counting and start reading and thinking, with care and attention. Gresham’s Law should not apply to science or medical education. Nor should it be valid in an economics true to the origin of the term as “the art of managing the household,” which is covered here at Naked Capitalism every day.
Notes
[1] There is a journal, which I have looked at on occasion. Scientometrics can be useful but also defines “meta.”
[2] As a beginning faculty member, I was apparently too obtuse for a senior colleague. He asked me if I would be willing to pay a $100 submission fee to a leading journal of which he was an Associate Editor in our common field. I answered “yes” but then asked if the page charges and surcharges for color figures/diagrams would still be charged. He answered “yes.” When I began submission to all journals was completely analog and the manuscript had to be typeset and figures converted from actual photographs for publication. This was expensive. At the time of our conversation, submission requirements were very strict, and all text and figures digital-ready at the proper resolution. The page charges that amounted to several thousand dollars were still collected even though the new technology did not require any special handling of the manuscript. Yes, someone was making a lot of money. We submitted our work to an equivalent journal published by the American Chemical Society. For free, with no charge for color if the editor agreed that it was necessary. In retrospect, perhaps wasting the $2,000 (17 years ago) unnecessarily might have been a better career move.
[3] Yes, I am aware that all journals and institutions are vulnerable to fraud, and that scientists are only people of every sort. Nevertheless, while the wheels may grind slowly, a scientist caught in a fraud eventually loses all credibility, as in the case of Andrew Wakefield.
[4] Note that to practice medicine in the United States as a licensed physician, a residency must be completed. These range from three years, including the “intern” year, in Internal Medicine and Family Medicine to five years in Urology and General Surgery and eight years in Neurosurgery.