Reading about addiction and recovery can be overwhelming and confusing. Media reports and experts often make strongly worded statements that are contradicted by statements from other media sources and experts. Other times, they seem to negate or minimize the lived experience of people with drug or alcohol problems and their families.

For example, it’s very common for press releases, media reports and, occasionally, researchers to make statements about a study demonstrating the effectiveness of a particular intervention. Other times, we hear people say something like, “science shows that [insert intervention] works.”

However, when we look closely at the study, we may find that the outcomes don’t fit our idea of “effectiveness” or “works.” Further, the conditions and subjects don’t resemble the real world.

This isn’t confusing just for lay people, it’s confusing for professionals and policymakers too. And, to make matters worse, most of us are pretty reluctant to question statements presented as science or evidence-based.

For this reason, I’ve been working on a guide that will hopefully allow anyone to review a study and evaluate its relevance to its goals. This way you can make an informed evaluation rather than having to rely on the reporting of others, who may see things through their own bias or interests. The guide is based on the following questions.

1. What is the treatment or intervention being studied?
2. Who were the subjects?
3. How long was the study?
4. What outcomes did the study measure? (How did they define success?)
5. What were the study methods?
6. What were the actual findings and does the authors’ discussion accurately represent the findings?
7. Were there any conflicts of interest (real or potential)?
8. What questions does the study not answer?

---

1) What is the treatment or intervention being studied?

It’s important to pay close attention to the intervention being studied. It is common for news reports about the study to describe it poorly. Further, it’s common for the study itself to obscure the details of the intervention.

Interventions might include:

• Medication with, or without, counseling
  • Opioid agonist medication
  • Opioid antagonist medication
  • Other medications
• Education
• Medical care
• Harm reduction
• Case management
• Mutual aid groups such as 12-step meetings, faith-based meetings  (Without professional treatment)
• Specific types of counseling or combinations of them, such as:
  • Motivational Interviewing
  • Twelve-step facilitation (Professionally directed treatment focused on linking and encouraging active participation in 12-step organizations.)
  • Relapse Prevention Therapy
  • Cognitive Behavioral Therapy

It’s also important to know more about how the treatment was delivered:

• Residential treatment
  • Long term
  • Short term
• Outpatient
  • How often? Once a week, or multiple times per week?
  • Over what period of time?
• Was it linked to additional services intended to support problem resolution over months and years, rather than days and weeks?
• Does the treatment in the study resemble how treatment is provided in the real world?

Question 2: Who were the subjects?

There is a wide spectrum of alcohol and other drug problems, with addiction on the most severe end and misuse on the less severe end. Further, there can even be considerable variation within a category. Additionally, there can be significant differences in where the subjects are found as well as their life experience or current conditions.

• Where were the subjects found?
  • Community population – Are the study’s findings based on a sample from the community?
  • Clinical population – Are the study’s subjects drawn from a treatment population or a treatment-seeking population?
    • If they are a treatment population, what kind of treatment were they receiving? Methadone, office-based buprenorphine, outpatient, residential, case management, etc.?
  • Something else – Some studies draw subjects from medical clinics, child protection cases, college students, mental health clinics, professional monitoring programs, corrections settings, etc.
• What were the demographics of the subjects?
  • Recovery capital
    • Are they high recovery capital clients with financial, social, family, environmental, or cultural resources to help them achieve AOD problem resolution?
    • Are they low recovery capital clients with few financial, social, family, environmental, or cultural resources to help them achieve AOD problem resolution?
  • Did the exclusionary criteria result in a group of subjects with a single problem? Meaning that they do not resemble real-world treatment populations? 
• What kind of AOD problem did the subjects have?
  • Substance use disorders (SUD) – This model is based on the DSM-5 and includes a broad spectrum of substance use problems ranging from addiction to misuse. It also may include any substance. Look for specifiers like mild, moderate, and severe to get a better understanding of the population.
  • Words like “Dependence” or “Abuse” – DSM dependence constitutes a more severe substance use problem (it may, or may not, be addiction), while abuse is considered mild in severity.
  • Substance specific – Some studies include subjects with a particular substance problem, others include polysubstance users. Many recent studies looked specifically at people with prescription opioid problems.
  • Chronic vs acute – Some studies include subjects with long-term chronic problems,  while others may focus on populations with short-term acute problems.

3) How long was the study? 

Robert DuPont once observed, “The most striking thing about substance abuse treatment is the mismatch between the duration of treatment and the duration of the illness.” ¹

Addiction is a chronic disease and recovery is a long-term process, but research is often limited to days and weeks.

The longer the study, the better. For example, Dennis, Foss, and Scott ² found relapse rates of 64% for people between 1 and 12 months abstinent. Those relapse rates drop to 34% for people with between 1 and 3 years abstinent. 

Therefore, a study that reports on any outcome at less than one year may say very little about what can be expected long term.

Look for studies that report on outcomes after one year.

4) What outcomes did the study measure? (How did they define success?)

Outcomes measured in research do not necessarily correspond well with the outcomes patients are seeking.

Common outcomes include:

• retention in treatment
• illicit opioid use
• mortality
• illicit drug use (non-opioid)
• criminal activity
• HIV risk behaviors

Less common outcomes include:

• abstinence from alcohol and other drugs
• quality of life measures like employment, housing, and family status.

The implications for this are profound. For example, a study may investigate the effects of a treatment on people with opioid use disorders. If the study is only examining the impact of the treatment on illicit opioid use, the treatment could be described as effective when subjects sustain alcohol, cocaine, or prescription opioid problems. 

Consider the outcomes you want for yourself or your loved one. Then, determine whether the study’s outcomes match your desired outcomes.

5) What were the study methods?

There are many approaches used in SUD research and each approach offers advantages and disadvantages in different situations. Methods include experimental (including randomized control trials), qualitative, case studies, meta-analysis, and observational.

It’s often said that randomized controlled trials (RCTs) are the gold standard for research. It’s important to keep a few things in mind about them. First, they lend themselves to studying easily quantifiable outcomes, which means they tend to focus on relatively narrow outcomes in relatively narrow contexts. Second, they tend to be very expensive, which means that they often only get done with financial backing from large institutions (public or private). Third, in some cases, their use may be limited by ethical problems related to using placebos or blinded treatments. Finally, they tend to eclipse experiential and local knowledge. 

It’s also important to look at other factors, for example:

• the number of subjects in the study;
• whether outcome numbers were based on all subjects or just those retained in the study;
• what measures/data were (e.g. self-report or urine drug screens).

6) What were the actual findings and does the authors’ discussion accurately represent the findings?

This sounds very straightforward, but it often requires a lot of effort to answer this question. Outcomes are sometimes reported very clearly in raw numbers and percentages, other times they are reported in the form of statistical terms that can be a challenge to decipher. 

• Do the discussion and conclusions focus on findings that support one theory/model/approach and ignore others?
• Do the discussion and conclusions overstate the real-world implications of the findings?
  • Keep in mind that statistical significance may not translate into significant improvements in quality of life.
  • Study definitions often differ from real-world definitions. For example, a recent study set a threshold for treatment engagement as taking medication for 10 or more days per month. This study was careful to characterize their findings as “increased days of medication treatment.” However, it’s easy to imagine another study or press release reporting 69% retention at 6 months.
• Finally, it’s important to ask whether the authors accurately represent the findings of its references. (For example, a 1980 letter to the editor was misrepresented and is believed to have profoundly misinformed opioid pain management policies.)

7) Were there any conflicts of interest (real or potential)?

A conflict of interest is a situation in which financial or other personal considerations have the potential to compromise or bias judgment and objectivity. It is worth noting that a conflict of interest exists whether or not decisions are affected by a personal interest.

Conflicts of interest can lead to more than unreliable information about particular treatments. For example, Stenius (2016) described alcohol and tobacco industry’s influence on the assumptions underlying policy decisions.

It is well documented how the tobacco industry for decades funded research aimed at producing uncertainty about the danger of smoking (e.g., Brandt, 2012). For alcohol, the transnational producers have invested resources in research that questions the relation between the total consumption and alcohol-related harms on a population level to prevent general regulations of the alcohol market (Adams, 2016).Stenius, Kerstin. (2016). Addiction journals and the management of conflicts of interest. The International Journal of Alcohol and Drug Research. 5. 9.

8) What questions does the study not answer?

No study can answer every question, nor should any study seek to every question. However, it can be helpful to stop and ask, what questions does the study not answer?  

There are two ways to group these questions.

First, there are questions that simply cannot be answered by the study. Considering what was asked, and not asked, provides context for the study.

Second, and maybe more important, is what questions does the study appear to have data for, but chose not to answer? For example, if a study looks at the impact of a treatment on drug use, as measured by urine drug screens, does it report on the number of subjects who were continuously abstinent?

---

1. DuPont R. (March, 2018). Interview with Brian Coon. Interview presented at the NC Recovery Alliance Summit, Durham, NC.
2. Dennis, M. L., Foss, M. A., & Scott, C. K. (2007). An Eight-Year Perspective on the Relationship Between the Duration of Abstinence and Other Aspects of Recovery. Evaluation Review, 31(6), 585–612.