Understanding Medical Research

Last Sunday’s New York Times Magazine includes a lengthy piece, “Do We Really Know What Makes Us Healthy?” which uses changing views of hormone replacement therapy as an example of why it can be difficult to make decisions based on medical research.

If you’re still confused about HRT and its risks and benefits (and aren’t we all?), this piece provides a good primer on why the advice on hormone replacement seems to be constantly changing. More than that, though, the article serves as a good starting point for understanding why that news release on breakthrough medicine may not stand the test of time.

How can you make sense of medical research and what it means for your health? Here are some basic tips:

1) Understand what a study can and can’t tell you
Some research, including some HRT research, simply looks at people over time. In an observational study, researchers might compare women who have been on HRT with those who haven’t, and compare their health outcomes. This can provide information on associations between drugs or behavior and disease, but can’t generally prove causality. Researchers may observe that women taking HRT have fewer heart attacks, but this may be due to other things about the women aside from hormone replacement.

One common example of this is that ice cream consumption and murder rates both tend to increase at the same time of the year. It would be easy to make the mistake of linking the two, while ignoring other things about it being summer that might independently increase both rates.

Other types of papers, such as case reports, may give clues to researchers, but can’t determine what those clues mean to the broader population. A randomized, controlled trial is generally considered the most reliable type of study design, but even these can have flaws.

2) Recognize that the researchers and research subjects selected may introduce biases
Women who seek out HRT, for example, may be generally healthier than those who don’t, introducing a “healthy user” bias. Those who can afford a preventive drug may be wealthier, introducing a number of confounding factors, such as access to healthy food, exercise, education, and a generally more healthy lifestyle.

Likewise, when individuals are asked about their past health behaviors, they may recall incorrectly, or give an answer that they believe is “right.” For example, if you ask subjects in a face-to-face interview about their sexual history or illegal drug use, they have certain incentives to avoid disclosing the truth.

Research studies may exclude the sickest patients, and include only women in middle life who are non-smokers and do not have existing health problems. Similarly, in practice, doctors might choose to prescribe preventive drugs to patients with longer expected life spans or who are more healthy in general, affecting the results of retrospective studies that look back at how patients who received HRT fared compared with those who didn’t.

Individuals who choose to participate in a study also may be different from those who don’t, by having an interest in that particular health factor or being more willing to comply with study protocols. The New York Times piece notes that patients who complied with HRT had better outcomes than those who didn’t, but the researchers also noted that women who faithfully took their placebo also fared better.

3) Be skeptical of breakthrough science
New findings often receive the most attention in the media, especially if they defy the conventional wisdom. However, science is a process, and over time other researchers may fail to replicate the findings, or may demonstrate that the opposite appears to be true. Unfortunately, these later refutations often fail to garner the same level of media attention.

4) Understand that the scientific process can have flaws
You may think that any paper published in a reputable journal is necessarily of good quality, but that isn’t the case. Peer reviewers sometimes lack the expertise to properly evaluate a study’s methods, the methods themselves may be flawed, or the authors may have represented the findings in a way that puts the best face on their work.

Likewise, much medical research is sponsored by entities with an interest in the outcome, such as pharmaceutical companies paying for drug trials. They are not currently required to report the results of all research, so studies that demonstrate a negative finding may never be published.

5) Understand the numbers
Not everyone can invest time in getting a good handle on statistics, but you can look for certain things in news reports. If a report says that a treatment reduces your risk by 20 percent, but your risk was only 1 percent to begin with, it’s not really a big difference that is meaningful to your life. If something is said to reduce the incidence of a complication by a certain percent, but the incidence is very low to begin with, the finding is likewise not that useful.
If a small benefit is demonstrated in a large group of research subjects, is it possible to know what that means to you as an individual?

There are a number of other statistical, bias, and methodology concerns to understand in interpreting medical research. This series, “How to Read a Paper,” is useful if you would like to explore the topic further.

Given the increasing attempts to use medical science (whether the research is well-done or not) to promote political agendas, it may be worth it to advocates to arm yourselves with the tools to understand or debunk claims. Have questions? Feel free to leave them in the comments.