Real World Data – We Should Be Cautious in Accepting Reported Results

Over the past many years, health care providers and payers have been asking for better clinical studies that can be used in making treatment and coverage decisions.   It would be great to know if one treatment alternative is usually more effective or safer in a particular condition or how patients on multiple medications might respond if a new agent is added.

Unfortunately, we have few comparative effectiveness trials or studies that look at how patients respond in real world settings.  In response to the demand and to the limited publicly available data, there has been an increased push for real world data.  But how reliable is “real world data”?

Recently, some good friends, Michael  Stuart, MD,  and Sheri Strite of Delfini.org, referred me to an article written by Dr. John Ioannidis (Ioannidis JPA (2005) “Why most published research findings are false”.  PLos Med 2(8):e124).  In his essay he builds on work done by other methodologists and looks at the probability that a research finding is true (PPV).

I reproduce part of Table 4 from his essay.

Example	PPV
Adequately powered RCT with little bias and 1:1 prestudy results	0.85
Underpowered, but well-performed phase I/II RCT	0.23
Underpowered, poorly performed phase I/II RCT	0.17
Adequately powered exploratory epidemiological study	0.20
Underpowered exploratory epidemiological study	0.12
Discovery-oriented exploratory research with massive testing	0.0010

What he reports is that an adequately powered RCT (randomized controlled trial) has a PPV of 0.85.  This means that it has an 85% chance of being true and 15% of not being true.

What is very alarming is that exploratory studies have much lower PPVs.  The three types of studies listed at the bottom of the above table are closely related to what we would call real world studies.  What is of concern to me is that an adequately powered exploratory epidemiological study might only be true about 20% of the time.  Other exploratory studies or research are true less frequently.

Whether or not you agree with Dr. Ioannidis methods and results, this should tell all of us to be cautious of results from individual real world studies, unless those results are replicated across multiple studies.