Setting Standards for Meta-Analyses

Nancy Roach is the Founder of C3 and Chair of the Board of Directors.

On September 22, I participated in a meeting that looked at some of the profound implications of the growing amount of publicly-available data, and grappled with questions around reporting research results using this data.  The day’s agenda was titled “Interpreting and Communicating Clinical Data in the Public Domain,” and the meeting was convened by the FDA Clinical Trials Transformation Initiative (CTTI).

The meeting delved into a lot of detail:

• Legislation which requires research sponsors to make summary data publicly available in www.clinicaltrials.gov, both positive and negative results;
• Existing guidelines and best practices for both clinical trials and meta-analyses; and
• Existing efforts to clarify guidelines for meta-analyses.

At the end of the day, my personal conclusion was that meta-analyses will be used in the future to make health care recommendations, and that strong guidelines are critical.  The meeting results are still being processed, so no formal recommendations have occurred yet.

Rather than report the nuts and bolts of a very long and technical day (I typed 8 pages of single-spaced notes), I’ll talk about some of the substance that we discussed, and why it matters to us, the non-scientists in the room.   Warning – this is complicated, and I’m explaining it at a very high level.  All mistakes are mine.

What does a meta-analysis look like?

We’ve all seen headlines like:

• Aspirin Is Found to Be Fighter Against Colorectal Cancer (New York Times, August 12, 2009)
• Diabetes Drug Linked to Higher Risk of Death (New York Times, November 25, 2008)
• The Claim: Being Left-Handed Adds to the Risk of Migraines (New York Times, April 15, 2008)

The common link between the stories is that the results being reported come from a meta-analysis – a combination of multiple trials and/ or studies – rather than a clinical trial.

First, clinical trials

In a clinical trial, researchers define the question they’re asking (hypothesis), how they will collect the data they need to answer the question and how they will analyze the data (analysis plan).  All of that is called “pre-specification” – saying what you’re going to do and how you’re going to do it before actually starting the analysis.  All of that is written up in a document called a protocol.  At that point, patients are enrolled on the trial and the data is collected.

Once all of the data is collected, the results are analyzed and sometimes reported through medical journal articles and presentations at professional meetings.  If the results are meaningful to patients, they may be seen in the popular press.   Medical journals have endorsed guidelines that recommend standard ways to report randomized clinical trials, so that readers can be sure that they have a complete understanding of how the research was done, including whether appropriate pre-specification occurred.   Journals may refuse to publish research that doesn’t meet the guideline criteria.

Pre-specification is fundamental to clinical research.  Over my years of advocacy, I’ve heard Dr. Robert Temple from the Food and Drug Administration (FDA) say that without pre-specification,  data can be manipulated to show just about anything.  In other words, research that is not pre-specified can generate other research questions, but should not be used to make critical decisions in patient care.

Clinical trials are designed to answer specific questions, and are not always large enough to show infrequent or small effects. Also, data from a specific clinical trial cannot be legitimately  extrapolated to answer broader questions that were not asked in the original clinical trial.  So, is there a way to look at multiple, somewhat similar, clinical trials that may have varying results?

Now, meta-analysis

A meta-analysis combines results from several trials and/or studies, and the large combined data set can show  impacts – both bad and good – which aren’t always seen in the original trials.  And doing a meta-analysis is much faster and cheaper than doing yet another even larger clinical trial, especially a large phase 3 clinical trial which can cost over $100 million and take 5-10 years to complete.

Because they can involve tens of thousands of people, represent a great deal of research effort and financial cost, meta-analysis results can influence public policy.  For example, the April 30, 2009 New England Journal of Medicine contains an article titled, Ounces of Prevention — The Public Policy Case for Taxes on Sugared Beverages. The article cites meta-analyses to make its case.  And meta-analyses frequently ask questions that are interesting to the public, generating articles like the ones listed above.

Thus, I assumed that meta-analyses were performed and reported with the same scientific rigor as clinical trials.  As I prepared for the meeting in September, I was surprised to learn that my assumption was wrong.  Many meta-analyses do pre-specify:

• The questions they are asking
• The inclusion and exclusion criteria for potential data sources
• The list of potential data sources
• The analysis plan

For example, the Cochrane Collaboration is famous for its carefully done, completely reported meta-analyses and systematic reviews.    However, reporting standards for meta-analyses do not require that these issues be addressed in articles.  Existing guidelines for meta-analysis such as the Prisma Statement stress documentation of what was done, but not pre-specification.

Why should we care?

Between the Internet and calls for transparency in clinical research, more data is publicly available than ever before.   Medical journals are accessible, and result reporting on www.clinicaltrials.gov has opened a floodgate of data (note:  you can listen online to Deborah Zarin from the National Library of Medicine explain the requirements).

Meta-analysis is a valuable tool that can help us look at trends from multiple clinical trials and/or studies, but as with all research data, it has limitations.  As patients and advocates, it is important for us to look at all research data with a critical eye to fully understand what conclusions can be drawn from them and what might just be suggestive and require further study.

Remember the quote from Field of Dreams?  If you build it, they will come. The research translation of that is:  If the data exists, it will be analyzed. And the analyses will continue to be used to guide health policy, clinical guidelines and public opinion.

Thus, accurate and complete reporting of meta-analysis results is critical.  The Institutes of Medicine and Food and Drug Administration are working on separate guidelines, each of which will take at least two years to complete.  In the meantime, I plan to be careful about the conclusions I draw from health care articles.