The World Health Organization recently advised people to avoid using artificial sweeteners for weight loss or to reduce their risk of health issues like heart disease and diabetes. This was based on the agency’s review of available research on artificial sweeteners to date.
Unfortunately, people cannot be confident in those findings. That’s because existing studies on artificial sweeteners are plagued by methodological problems. Even the W.H.O. knows this, given that it ultimately described its certainty in the existing evidence as “low.” Maybe it’s true that artificial sweeteners don’t help with weight loss — but we really do not know for sure.
This is not a problem reserved for artificial sweeteners alone. The state of nutrition research is poor, and the problems afflict much of the research into dietary and lifestyle claims around things like coffee, wine, dark chocolate, fad diets, the amount you exercise — you name it. This in part explains other recent flip-flopping around whether moderate drinking is “good” for you: A recent review found the research methods used in many past studies on the benefits of drinking alcohol to be flawed.
Diet and exercise are clearly important parts of a healthy lifestyle, but it’s challenging to accurately estimate the specific effect of making any change based on how most nutrition and lifestyle research is currently conducted.
Take the case of artificial sweeteners. Randomized studies — where people are randomly assigned to one treatment or another to ensure that no other factors interfere — are considered the gold standard. But randomized trials of sweeteners are often small and brief, which makes it hard to reach reliable conclusions about their long-term effects. The way sweeteners are studied in trials is also often very different from the way people use them in the real world. For example, some trials had participants consume artificial sweeteners in addition to their typical diets, rather than replacing some real sugars in their diets with artificial sweeteners — the intervention researchers are most interested in — often over a period of just a few months.
Many studies, of both sweeteners and other diet and lifestyle behaviors, are not randomized. For example, several studies of sweeteners simply observe people over time, following both their sweetener use and their health outcomes like rates of diabetes or heart attacks. These observational studies, as they’re called, have their own problems, many of which are so serious that it is difficult to take such studies, well, seriously.
The most significant of these problems is well known: Correlation does not imply causation. If people who consume more sweeteners are more likely to have Type 2 diabetes, did the sweeteners cause the diabetes? Or are the people who use more sweeteners also more likely to have diabetes because of other aspects of their diet or health? Researchers can try to account for obvious differences between groups, but it’s impossible to account for everything.
If the typical randomized trials and observational studies of dietary and lifestyle research present so many challenges, how can we get reliable answers?
Reliability still starts with randomization. Randomization is key to establishing cause and effect; it helps make sure two groups are otherwise similar before we consider what happens to those people who consume different amounts of artificial sweeteners, red wine or dark chocolate.
In randomized trials, researchers intentionally randomize people to one group or another, but it’s difficult to conduct trials like this that are large enough and long enough to be useful. (Would you let a scientist tell you what to eat every day for the next decade?)
But there are other ways to credibly study the cause-and-effect relationships of dietary and lifestyle behaviors: by identifying situations in which people are exposed to those behaviors, not by the randomizing hands of researchers but by accident. So-called natural experiments, commonly used in economics, are extraordinarily powerful but sorely underused in medical research.
Consider, for example, that in 1953 Britain ended the rationing of sugar and sweets that had been in place since World War II. Interested in studying the effect of sugar intake in early childhood, the economists Paul Gertler and Tadeja Gracner noticed that children born in the years just before the rationing ended spent their infancy and toddler years with limited sugar in their diets because of said rationing.
Children born a few years later had early childhood diets heavier in sugar. When those children became adults, their intake of sugar continued to be higher than that of otherwise similar children who were born during sugar rationing.
By measuring the health of these two groups more than 50 years later — far longer than any clinical trial could reasonably follow people — the economists found that the additional sugar intake led to higher rates of diabetes, elevated cholesterol, arthritis and measures of chronic inflammation.
Another way people can be accidentally randomized to health behaviors is through their genes. Consider the heavily studied question of whether alcohol, in moderation, is “good” or “bad” for your health. In a study of over 500,000 Chinese adults, researchers took advantage of the genetic variations that cause some adults, randomly, to enzymatically process alcohol differently, leading to unpleasant symptoms such as flushing. Because those individuals tend to drink less alcohol, researchers can study the causal relationship between alcohol use and health outcomes by examining otherwise similar people with and without specific gene variants, an approach called Mendelian randomization.
While the jury is still out, some research using these methods suggests that even small amounts of alcohol may lead to higher risks of cardiovascular disease and cancer.
Now, here are some untested ideas in nutrition research that, using methods more often found in economics than in medicine, could take advantage of naturally occurring randomization.
Returning to the question of how early childhood sugar intake affects health, let’s say researchers could track down families with three children where the middle child was diagnosed with diabetes. The eldest child in those families may have spent several years growing up without special household attention paid to sugar intake, until the middle sibling was diagnosed with diabetes. Meanwhile, the youngest child in those families might have grown up in a particularly sugar-conscious household.
One could study these families and compare long-term health outcomes between first and third siblings who, by chance, were exposed to different sugar environments. And if they were concerned (as we’d be) that the oldest and youngest kids in families might differ in other ways besides their exposure to sugar, they could account for that by comparing first and third siblings in otherwise similar families in which the middle child did not have diabetes. This isn’t a perfect study, since siblings don’t grow up in identical environments, but it’s better than simple observational ones because it takes advantage of the random nature of siblings’ birth order.
We understand why so many simple observational studies get published; the impacts of diet are difficult to study by traditional means in medical research, and there’s great desire to better understand the health effects of the foods we eat.
But filling the research void with studies that do little to help us understand the cause-and-effect relationships of our real-life dietary decisions does little to advance understanding — in fact, it sows confusion.
Medical researchers pressured professionally to “publish or perish” are often incentivized to publish simple observational studies that lack empirical rigor. Medical journals, responding to public interest in information about diet, in turn encourage this research despite knowing its significant limitations. Media coverage may simply add to the confusion.
The now decades-old “credibility revolution” in economics advanced the use of high-quality, often creative research designs in empirical economic work — so much so that in 2021, a group of economists were awarded the Nobel Prize for their work with natural experiments.
Although medical researchers are increasingly taking advantage of natural experiments — thanks in part to large increases in digital data in recent years — these methods remain under-taught and underused, particularly when it comes to diet. This important research needs a credibility revolution of its own.