Comparative Effectiveness Research
Driven by the paucity of good evidence of the effectiveness of treatments, medications, therapies and interventions in usual care, the Institute supports Comparative Effectiveness Research (CER) and its methodological developments. CER describes emerging strategies for clinicians and researchers in research study design that lead to new and critical evidence to better inform decision-makers.
Evidence-based medicine, which gained traction over the last decades and is now the preferred paradigm for effective health care, insists that clinical decision-making should be guided by evidence from clinical research. Typical clinical studies offer somewhat of a conundrum: While the integrity of most randomized controlled trials reflects scientific rigor, the evidence such trials generate rarely offer insights into usual care or real world practice.
CER uses a broad methodology and randomized studies; it deviates from the strictly controlled trial framework by adjusting design elements, such as outcomes, eligibility criteria, or treatment protocols, to drive evidence creation that answers such fundamental questions as “Is this treatment better than that one?” “Does this treatment have side effects that affect women more than men?”
CER places an especially strong emphasis on engaging with stakeholders. Unlike conventional research practice, which often takes place in artificial environments, such as laboratories, CER works with stakeholders to determine the kinds of evidence they need, and then identifies research priorities and develops research protocols.
Naturally, in the context of Integrative Medicine, CER has enormous potential to shed light on the effectiveness of many practices. CER strategies may be used to investigate the effectiveness of traditional Chinese medicine, nutritional supplements, mindfulness-based practice (yoga, meditation), acupuncture, and many other interventions and therapies, medications and treatments.
At TIIH, scholar Dr. Claudia Witt is leading numerous initiatives that provide guidance to other investigators. Dr. Witt and colleagues published a review that shows that acupuncture research already contributes to CER.
More recently, Dr. Witt convened expert and stakeholder symposia to develop Effectiveness Guidance Documents for Acupuncture and Chinese medicine to guide future research. When the documents are published, they will not only contribute to the shape of research modalities around the world, but also serve as an example of what can be produced when stakeholders and researchers collaborate.
Frequently Asked Questions
- How does CER fit into the way clinical research is conducted today?
- How is CER defined?
- Where does CER fall on the efficacy-effectiveness continuum and why? Which elements of study design may be optimized to increase the effectiveness of a research study?
- What unique features characterize a CER study?
- Why is CER important?
How does Comparative Effectiveness Research fit into the way clinical research is conducted today? hear answer
The complexity and variety of clinical decisions that confront physicians and patients in real-world settings demand better evidence than is currently available, and that is different from much of the evidence we already have. This evidence is also needed by payers, policymakers, caregivers, families and others, who also make critical decisions about health care. These decisions may be about medications, surgery, diagnostic tests, or other aspects of health care about which there is limited reliable and relevant information.
To understand why there’s a lack of good clinical evidence for decision-making relevant to typical patients, settings and circumstances, it’s helpful to briefly review the current state and practice of clinical research. Clinical research in medicine is an enterprise of extraordinary complexity, breadth and depth. All of it is aimed at the goal of improving human health.
But, although the range and diversity of medical technologies and treatments have dramatically increased over the last few decades, the strategies of clinical research have not always succeeded in yielding the full range of evidence that decision-makers need.
In practice, clinical research studies often are characterized by the following features:
- Most studies are conducted in large teaching institutions, and deliver study treatments in a very standardized way. They also assign patients to different treatment groups by chance, and also often compare treatments to placebos rather than to other commonly used treatments.
- Research studies seldom look at complex interventions, such as the combination of medication, exercise and dietary advice in cardiovascular disease, yet these complex strategies are commonly used in actual clinical practice.
- Patients included in most research studies are chosen to be very similar to one another. This means patients with more than one medical condition, or those taking multiple medications are usually not included in research studies. It is clear that studies need to include more “typical” patients. Then studies would not be so homogenous, and the effects of treatment in different sub-groups would get more careful attention. By sub-groups, we mean patients who have a common characteristic, like as race, age group and co-morbidities.
How is CER defined? hear answer
For more informed medical decision-making in real-world clinical care, there is an urgent need for more evidence from comparative effectiveness research. CER has been defined by a variety of entities. Most are similar to the definition offered by the U.S. Institute of Medicine National Priorities Committee:
Comparative Effectiveness Research (CER) is the generation and synthesis of evidence that compares the benefits and harms of best care methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.
Let’s consider an example: For patients with severe osteoarthritis of the knee, a study might compare surgery for joint replacement with the effectiveness and side effects of a new drug. . In such a study, results for different types of patients (sub-groups)—such as patients ages 18-40, patients ages 41-65, obese patients and non-obese patients—could be provided.
Understanding the distinction between efficacy and effectiveness in clinical studies is critical to our understanding of CER, and relates directly to the purpose of CER as described by the Institute of Medicine, to assist decision makers in making informed decisions.
Where does CER fall on the efficacy-effectiveness continuum and why? Which elements of study design may be optimized to increase the effectiveness of a research study? hear answer
When we measure efficacy, we are trying to answer the question: Did the treatment, intervention, or therapy perform as we expected it to perform in a perfect or ideal environment? In other words, did the intervention produce a detectable specific effect when all circumstances were optimized? This type of research excludes as many complicating influence factors as possible. A good example is the typical drug study, such as a comparison of a new drug to reduce high blood pressure with a placebo or another drug, in patients with no other medical problems and who are reminded frequently to take all of their study medications.
When we measure effectiveness, we are trying to answer the question: Did this treatment work in a real-world setting, similar to how health care would usually be delivered and used? The key question in this approach is to determine which treatment works best for which patients, and under what circumstances. A good example of an effectiveness study is a study that compares a drug with an exercise program to reduce high blood pressure in patients treated by community physicians.
To have a clear picture about the clinical evidence for an intervention, both efficacy and effectiveness are relevant. However, as its name clearly implies, Comparative Effectiveness Research focuses on developing evidence of effectiveness. And it often involves comparing at least two widely used treatment options, since those are the type of choices that patients and doctors commonly face.
It is crucial to understand that efficacy and effectiveness are at opposite poles of a continuum. Any study can be placed somewhere along this continuum—and only few studies are at one or the other extreme. Where it is placed depends on the details of the study design.
Three broad elements of study design influence the placement of each study within the efficacy effectiveness continuum. These elements are (1) the eligibility criteria for the patients that are selected to be included in the study, (2) the details of the treatments patients get during the study (the treatment protocol), and (3) the parameters that are used to measure the treatment effect (the outcome measures).
Let’s examine each of these study elements in somewhat finer detail:
- The eligibility criteria of the study participants. For example, a study which included only highly selected patients of younger age without co-morbidities and co-medication who are expected to have a better reaction to the intervention would fall on the efficacy side of the continuum. Maximum effectiveness would mean to include all types of patients who are affected by the disease without excluding subgroups of patients with risk factors e.g. high blood pressure, which might predict a less favorable response to the treatment.
- The treatment protocol. A study in which a very standardized treatment protocol as study intervention (e.g., a fixed exercise program without any flexibility) would fall on the efficacy side of the continuum. Maximum effectiveness would suggest a flexible treatment protocol as implemented in routine or usual care (e.g., exercise program individually adapted to the age and co-morbidities and preferences of the patient).
- The outcome measures. Objective outcome measures such as death and laboratory data would fall on the efficacy side of the continuum. Maximum effectiveness would rely more heavily on patient-reported outcome measures (e.g., symptom scales or quality of life measures).
When study participants are highly selected, the treatment is standardized, and all major outcomes are not directly experienced by patients, such a study would provide information on efficacy more than effectiveness. In this case the study results would be very valid for similar patients in similar settings, but less so when applied under other conditions. Thus the study is said to have a higher internal validity. This means that the results of the study are less influenced by other factors, for example, beliefs and health behaviors. We may reasonably conclude that the results actually represent the effect of the treatment we have researched.
However, when study participants represent a mixture of typical patients, the treatment protocol is more flexible, and the outcome measures are more patient-centered or subjective, the study measures effectiveness more than efficacy. When the study measures effectiveness, the results are regarded as more relevant to real-world health care decisions. In other words, this study is considered to have a higher external validity.
It is important to keep in mind that CER does not only mean doing studies that represent the extreme range of effectiveness studies. The “art” of CER requires that researchers deliberately sacrifice some degree of internal validity in order to increase the generalizability, relevance, feasibility and timeliness of research results. The right balance of the desirable features of research cannot be determined rigorously through a scientific process. The optimal balance emerges from the process of engaging stakeholders in a discussion about each element of study design, in light of current evidence, and what sorts of decisions must be made.
What unique features characterize a CER study? hear answer
The results of CER studies should directly inform a specific clinical decision from the patient perspective or a health policy decision from the population perspective. To reach this aim:
CER compares two or more health interventions (such as a therapy, medication, or other treatment) in order to determine which of these options works best for a variety of patients. The emphasis in this description of CER is on the comparison of one or more interventions to each other and their effects on patients. The use of a placebo is not generally an accepted intervention in usual care, and a comparison to a placebo or sham intervention is not regarded as a comparator with the potential to be best practice.
CER measures outcomes—both benefits and harms—that are important to patients. One organization that is engaged in promoting the enterprise of Comparative Effectiveness Research is PCORI: The Patient-Centered Outcomes Research Institute. PCORI “helps people make informed health care decisions—and improves health care delivery and outcomes—by producing and promoting high integrity, evidence-based information—that comes from research guided by patients, caregivers and the broader health care community.”
Dr. Joe Selby, PCORI’s Director, recently remarked that one of CER’s central themes is that “. . . patients value different outcomes differently.” Selby also said this about outcomes: “I think it’s much better for us to generate good comparative effectiveness on outcomes and let others [stakeholders] think about the findings.” CER empowers stakeholders to review findings from CER studies to make decisions and arrive at preferred outcomes.
CER is conducted in settings that are similar to those in which the intervention will be used in practice. This means that for interventions mainly used in outpatient primary care settings, the studies would have to be done with typical providers.
Let’s consider an example: For patients with chronic neck pain, a study compares the effectiveness of painkillers to physiotherapy. The study was designed in a group involving all relevant stakeholders. A heterogeneous sample of patients will be recruited from different cities and patients will distributed by chance (randomized) to both treatment options. Treatment will be offered at 50 outpatient clinics in usual care settings.
The primary outcome will be neck pain and disability, measured with a validated questionnaire that will be completed by the patients. Side effects and quality of life will also be measured. The treatment protocol in both groups allows some flexibility. In the medication group, patients can receive different types of painkillers and dosage can be adapted as needed. In the physiotherapy group, patients get individualized physiotherapy according to their main complaints. The results will be displayed for entire sample, but also separately for sub-groups of men and women, and for patients who have long-term versus short-term chronic neck pain.
Why is CER important? hear answer
First and foremost, the promise of CER is that it will help decision makers, particularly those faced with real-world care decisions. At the public policy perspective, CER is of increasing importance, because there will be increasing pressure on clinicians and patients to make better and more informed decisions about their health care. Better treatments, better information technology, more organized delivery systems, and increasing cost pressures will make all of this essential.
CER holds out the hope for the decision-making needs of patients whose needs are not always well served by the typical studies, for example, the elderly, children, seriously and terminally ill patients, and minorities of all types. Ultimately, the kinds of evidence that CER studies are capable of generating can contribute to an ever-evolving model of a more “personalized medicine.”
CER requires the participation of stakeholders such as patients, physicians, payers in the design and implementation of studies, as well as in the evaluation and dissemination of study results. CER studies put stakeholder priorities “into the mix” as the studies are developed and involve these stakeholders in refining which specific research questions are to be studied. Importantly, engaging stakeholders also means listening closely to the responses of patients when they are asked:
“What questions do you have about your condition?”
“What is your preference of possible outcomes related to your condition?”
“What help do you need to make which decisions?”
By seriously considering stakeholders’ input, CER has the potential to produce results that inform clinical and policy decision-making. Furthermore, it can re-shape stakeholders’ attitudes about research and regulatory authorities. The old paradigm of the clinical research enterprise can be likened to patients seeing themselves as passengers on a city bus. On the bus, the passengers are being driven from point to point, sometimes bypassing or falling short of their destinations: anonymous participants with no say about the route, the speed, or their fellow passengers.
However, a new paradigm that meaningfully engages stakeholders suggests a new metaphor: participants can be likened to individuals who share responsibilities in a car pool. Each participant brings unique skills, perspectives, resources and a personal sense of urgency about getting to a specific destination.
On the bus, it doesn’t matter how urgent things are for you—you arrive when the bus arrives at the end of the route—which might or might not be close to your destination. In the car pool, each member “drives,” offers alternatives to direction, contributes personal knowledge and experience, and talks with fellow passengers.
Comparative Effectiveness Research has the potential to put patients in the driver’s seat, or at least to share the driver’s seat with other stakeholders. When properly designed, CER offers new evidence, insight into sub-groups, and timely, relevant, and generalizable results.