Why Most Published Research Findings Are False
John P. A. Ioannidis
John P. A. Ioannidis is in the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America. E-mail: jioannid@cc.uoi.gr
Competing Interests: The author has declared that no competing interests exist.
I try to keep up with what is happening in my chosen fields of Study: Endocrinology as well as Medical Anthropology, in addition to my craving to know what is happening in this world (I try to read BBC News, The Economist regularly and scan through the internet in search of interesting news items).
I see a medical tendency to exaggerate, and this exaggeration is related to the interest the author has in increasing his prestige or income or career opportunities.
Always look for the Competing Interests.
I am not interested in reading about the Lack of Effect of Oral Medications in Type 2 DM from a professor who has financial relationship with Insulin producing drug companies.
If one researcher has many relationships with many drug companies, the chances are his research results would be WRONG.
I will highlight this article by Dr Ioannidis who has done a favour to all or us by highlighting how often biased results are published and gain in popularity.
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. Power is also related to the effect size. Thus research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1–1.5) [7]. Modern epidemiology is increasingly obliged to target smaller effect sizes [16]. Consequently, the proportion of true research findings is expected to decrease.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
Most Research Findings Are False for Most Research Designs and for Most Fields
Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias
How Can We Improve the Situation?
Is it unavoidable that most research findings are false, or can we improve the situation? A major problem is that it is impossible to know with 100% certainty what the truth is in any research question. In this regard, the pure “gold” standard is unattainable. However, there are several approaches to improve the post-study probability.
Better powered evidence, e.g., large studies or low-bias meta-analyses, may help, as it comes closer to the unknown “gold” standard. However, large studies may still have biases and these should be acknowledged and avoided. Moreover, large-scale evidence is impossible to obtain for all of the millions and trillions of research questions posed in current research. Large-scale evidence should be targeted for research questions where the pre-study probability is already considerably high, so that a significant research finding will lead to a post-test probability that would be considered quite definitive. Large-scale evidence is also particularly indicated when it can test major concepts rather than narrow, specific questions. A negative finding can then refute not only a specific proposed claim, but also a whole field or considerable portion thereof. Selecting the performance of large-scale studies based on narrow-minded criteria, such as the marketing promotion of a specific drug, is largely wasted research. Moreover, one should be cautious that extremely large studies might be more likely to find a formally statistical significant difference for a trivial effect that is not really meaningfully different from the null [32–34].
Second, most research questions are addressed by many teams, and it is misleading to emphasize the statistically significant findings of any single team. What matters is the totality of the evidence. Diminishing bias through enhanced research standards and curtailing of prejudices may also help. However, this may require a change in scientific mentality that might be difficult to achieve. In some research designs, efforts may also be more successful with upfront registration of studies, e.g., randomized trials [35]. Registration would pose a challenge for hypothesis-generating research. Some kind of registration or networking of data collections or investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment. Regardless, even if we do not see a great deal of progress with registration of studies in other fields, the principles of developing and adhering to a protocol could be more widely borrowed from randomized controlled trials.
Finally, instead of chasing statistical significance, we should improve our understanding of the range of R values—the pre-study odds—where research efforts operate [10]. Before running an experiment, investigators should consider what they believe the chances are that they are testing a true rather than a non-true relationship. Speculated high R-values may sometimes then be ascertained. As described above, whenever ethically acceptable, large studies with minimal bias should be performed on research findings that are considered relatively established, to see how often they are indeed confirmed. I suspect several established “classics” would fail the test [36].
It is quite obvious that most of the medical practitioners especially those in private practice have absolutely no idea of what is wrong with evidence based medicine or how certain articles are being touted as the new remedy, especially if they are relying upon drug company salesman or conferences in snow resorts or Hawaii for their continuing medical education. But they can feel that they are “legally” safe, but practice of medicine was supposed to be ethically safe and not legally safe.
As an anthropologist, I look at the published medical literature, even in prestigious journals such as NEJM and see the bias, such as: talking about pregnancy outcomes in San Antonio without mentioning that majority of the people studied there are of Mexican origin and poor, studies from Atlanta talking about High Blood Pressure fail to mention that it is the city with highest proportion of Americans of African Origin, more than once in my travels they have asked me Why do Pima have such high prevalence of Diabetes and people think Pima are the only Indians with Diabetes, such labeling had made pariahs of citizens of Nauru and other Pacific Islands.
Today I was seeing a patient, 38 year old and the note was made that her random Blood sugar was over 400 mg/dl (divide it by 18 to get SI unites i.e. about 22). She was on Long Acting Insulin and also Metformin, Actos and Glipizide.
It would have been so easy to look at the paper and give her more medicine or more of the same medicine and admonish her or label her as a Non Compliant patient. Obviously she was not taking the medications, but the reasons were, all non medical:
Unemployment
Lack of self-esteem in inability to look after family
Food Stamps not allowing her a healthy nutrition.
Bad relationship with Boy friend, who is yet arrested for another DUI (Driving Under Influence).
Alcohol abuse to stave boredom
Depression and lack of interest in her health.
She herself said: if I had a job, most of these problems would disappear.
The approach to this patient is not: more Pioglitazone or more Insulin; it is not a nutrition lecture; it is not Prozac for depression, it is not admission to the in patient ward.
The correct approach is culturally sensitive counseling. How many of our providers can be counted on doing that. I am very lucky since the person I work with, an RN with CDE is extremely competent and takes care of many of the aspects and gives me ample time to do what I think should be done for a Social Disease: Culturally Sensitive Counseling.
I do 5 minutes Continuing Medical Education vignettes for my colleagues. They dismiss the ranting of professors from New Orleans who push vigorously the new generations of drugs (Byetta,ONglyza etc) but are very happy to hear when I give them the summary of article that appeared in Archives of Internal Medicine, Sept 27, 2010
Long Term Effects of a Lifestyle Intervention on Weight and Cardiovascular Risk factors in individuals with Type 2 Diabetes
I quote their conclusions
Intensive lifestyle intervention an produce sustained weight loss and improvements in fitness, glycemic control and CDG risk factors in individuals with Type 2 Diabetes.
The study lasted 4 years.
As a good hearted friend of mine from the West Coast said to me: we already know what is good for Diabetes, but so little money goes into the research and implementation of that but millions of dollars are spent by Pharma so that Bald Head professors (Bob Marley would have said that) can go around the world touting that 140 usd per month medications have dubious value over 20 usd per month medications but they certainly reduce something or other from 9.7 to 8.1 and stress the fact that the reduction is 20 per cent! Most of the erudite listeners have no idea that the way it is being presented is an exaggeration and what does 20 per cent mean to most people?
I am currently reading
Diabetes Sugar Coated Crisis
Who gets it, who profits and How to stop it
If you are among those who is not manipulated by the drug companies in your prescribing habits and one of the few who hang on to the youthful hopefulness that you had when you were a medical student, I recommend you read this book…
Within the first few pages, you would read
Social Diseases need Social Approaches….
Hope my doctor friends in the Far East read this blog!
