Real World Data – We Should Be Cautious in Accepting Reported Results

Over the past many years, health care providers and payers have been asking for better clinical studies that can be used in making treatment and coverage decisions.   It would be great to know if one treatment alternative is usually more effective or safer in a particular condition or how patients on multiple medications might respond if a new agent is added.

Unfortunately, we have few comparative effectiveness trials or studies that look at how patients respond in real world settings.  In response to the demand and to the limited publicly available data, there has been an increased push for real world data.  But how reliable is “real world data”?

Recently, some good friends, Michael  Stuart, MD,  and Sheri Strite of Delfini.org, referred me to an article written by Dr. John Ioannidis (Ioannidis JPA (2005) “Why most published research findings are false”.  PLos Med 2(8):e124).  In his essay he builds on work done by other methodologists and looks at the probability that a research finding is true (PPV).

I reproduce part of Table 4 from his essay.

Example	PPV
Adequately powered RCT with little bias and 1:1 prestudy results	0.85
Underpowered, but well-performed phase I/II RCT	0.23
Underpowered, poorly performed phase I/II RCT	0.17
Adequately powered exploratory epidemiological study	0.20
Underpowered exploratory epidemiological study	0.12
Discovery-oriented exploratory research with massive testing	0.0010

What he reports is that an adequately powered RCT (randomized controlled trial) has a PPV of 0.85.  This means that it has an 85% chance of being true and 15% of not being true.

What is very alarming is that exploratory studies have much lower PPVs.  The three types of studies listed at the bottom of the above table are closely related to what we would call real world studies.  What is of concern to me is that an adequately powered exploratory epidemiological study might only be true about 20% of the time.  Other exploratory studies or research are true less frequently.

Whether or not you agree with Dr. Ioannidis methods and results, this should tell all of us to be cautious of results from individual real world studies, unless those results are replicated across multiple studies.

The Affordable Care Act – Overwhelming and Confusing?

This past Monday, the Department of Health and Human Services released the operating rules for state-run health insurance exchanges.  This is interesting enough in itself , since the government provides guidance on how states should establish exchanges, qualify health plans for participation and determine the eligibility of small businesses and individuals.  However, I find it extremely interesting for another reason – the volume of the rules.

Think about it.  The Affordable Care Act was about 2,400 pages long.  Very few actually knew what it contained.  Even in March of 2010, Nancy Pelosi, then Speaker of the House, was quoted as saying “We have to pass the bill so you can see what’s in it ….”  Since it was passed, many individuals and professional organizations have been asking the various agencies in the government for more details about specific sections and how they will be applied.  Often the answer is something similar to “Further guidance will be provided.”

Now we have further guidance on the state exchanges.  The final rule released this week contains 642 pages.  And I am sure that further pages will be released to explain some of the rules listed in those 642 pages.  So if just one aspect of the Affordable Care Act has already taken 642 pages to explain, how many pages will it take to explain the more than 2,400 pages of the full act.

If you look at it from this perspective, it should come as no surprise that there are still many who feel overwhelmed and possibly confused by all the implications of the Affordable Care Act.