You are here

Real-World Evidence: Promise and Peril For Medical Product Evaluation

Sanket S. Dhruva MD, MHS
Joseph S. Ross MD, MHS
Nihar R. Desai MD, MPH


The development and application of evidence to guide clinical decision-making underlie the promise of medicine to improve human health. For decades, the gold standard for evidence generation in medical product evaluation has been the randomized clinical trial (RCT). However, there is growing interest in the role of real-world data for evidence generation. In this manuscript, we describe what real-world evidence (RWE) means and why it has become increasingly important in the context of evolving evidentiary standards. We first review the current challenges facing those who conduct traditional RCTs. Next, we describe the benefits of RWE, but also its limitations, by discussing different data sources that may represent real-world data and, in turn, contribute to RWE medical product evaluations. Finally, we describe what RWE means for P&T committees and offer thoughts on how P&T committees can become engaged in proactive medical product evaluation.


The past several decades have witnessed impressive developments in medical products and therapeutics—small-molecule drugs, biologics, and medical devices—which have in turn improved health and outcomes for patients. Traditionally, data for these products have been generated through RCTs, the decades-long gold standard for demonstrating safety and efficacy because of their ability to mitigate bias and control for potential confounders. RCTs have well-known limitations that may limit their generalizability to real-world clinical practice. These limitations include conduct in highly selected populations and specialized environments that require intensive monitoring to ensure adherence to study protocol, neither of which may represent everyday clinical practice.1

While these limitations to traditional RCTs are well-known, the accelerating pace of medical product innovation has been accompanied by additional challenges, many of which relate to their escalating costs. The National Academy of Medicine has concluded that phase 3 trials “have become extraordinarily expensive.”2 Estimates of the cost for bringing a new drug to market range from $648 million3 to $2.6 billion,4 and the cost of developing drugs has risen approximately 8.5% beyond inflation in recent years mainly because of increasing trial costs.4 Cost drivers include: an increasing administrative burden with more complex clinical trial protocols that use multiple assessments, increased responsibility by institutional review boards because of lack of clarity about oversight mechanisms, and inefficient clinical trial monitoring.5 There has also been an increased focus on chronic disease drug development in recent years,5 which requires longer follow-up time and results in greater trial costs.6 With the success of prior therapies in some fields such as cardiovascular medicine, declining event rates have led to the requirement for even larger study populations.7 These factors related to rising costs have been exacerbated by a decrease in the rate of investment growth in medical research8 and waning site and patient participation.9 Therefore, the higher costs, complexity, and longer trial length have led to concerns that RCTs may be inadequate to keep pace with the need for evidence.9 Coupled with high failure rates, these factors may make traditional RCTs prohibitive in some areas.10


Simultaneously with the challenges in generating data through RCTs, the Food and Drug Administration (FDA) is shifting its paradigm toward a “life-cycle” regulatory approach, which means product evaluations based on integration of both pre-market and post-market evidence.11 Nearly two-thirds of the first-in-class novel therapeutic medications approved between 2005 and 2012 received FDA priority review,12 which reduces the FDA’s review time to six months from the standard 10 months while directing additional attention and resources to the review.13 The FDA is increasingly allowing therapies to come to market on the basis of fewer pre-market clinical trials, or trials that are shorter or based on surrogate measures of disease, on the premise that probable benefits outweigh risks and that uncertainty remaining at the time of approval can be addressed by more thorough post-market evaluation. Multiple pathways have existed for decades to expedite drug approval,14 and the 2012 FDA Safety and Innovation Act created an additional one, the “breakthrough therapies designation.” This FDA drug review pathway allows approval if preliminary clinical evidence indicates the drug may show substantial improvement over existing technologies.14 The “breakthrough” designation and other expedited approval pathways have been utilized increasingly in recent years.15 Through September 30, 2017, the FDA had granted 191 of 500 requests (38.2%) for “breakthrough” designation and denied 244 (48.8%), while 65 (13%) had been withdrawn.16

Expedited drug approval pathways also often reduce the quantity of evidence necessary for FDA approval,14 such as enabling enrollment of smaller study populations.17 Of the first-in-class drug approvals between 2005 and 2012, 34% were on the basis of a single pivotal trial and only 21% used an active comparator.12 Some expedited pathways also explicitly allow reliance on surrogate measures and biomarkers, instead of clinical outcomes.14,18 A surrogate measure is “… an endpoint that is used in clinical trials as a substitute for a direct measure of how a patient feels, functions, or survives. A surrogate endpoint does not measure the clinical benefit of primary interest in and of itself, but epidemiologic, therapeutic, pathophysiologic, or other scientific evidence.”19 Therefore, preliminary approval data relying on surrogate measures often require verification in post-marketing studies.20

Although post-marketing studies have become increasingly important aspects of new drug evaluation, a study examining drugs approved between 2005 and 2012 that had been based on either a single trial or were primarily focused on surrogate measures of disease identified no post-approval studies for 35% of approved indications after a median 5.5 years of follow-up.21 There were few published, randomized, controlled, double-blind studies demonstrating superior efficacy based on clinical outcomes. Similarly, 36 of 54 cancer drug approvals from 2008–2012 (67%) were made on the basis of trials primarily focused on surrogate measures; but after a mean 4.4 years of follow-up, 86% of the approved drugs had failed to show gains in overall survival in clinical trials or had not been subsequently tested.22 Recent data on 22 drugs granted accelerated approval for 24 indications between 2009 and 2013 showed that only half of confirmatory studies were completed after a minimum of three years of follow-up.23 Clinical benefit remained unconfirmed after five years for eight of the indications. Thus, to summarize, although post-approval study data are gaining significance in providing information about drug safety and effectiveness, these studies are often not completed or do not demonstrate clinical benefits when trials are done. Lack of or delays in post-approval study completion may stem from multiple factors, and increased FDA transparency24 about actions taken—or reasons for a lack of action—in response to these studies would be helpful. When post-approval studies show a lack of safety or effectiveness or are inconsistent with earlier data, those data must be integrated into a comprehensive benefit–risk assessment of the drug and, in some cases, may lead to agency action to withdraw market approval or require a more definitive study. Paradoxically, although expedited approval pathways rely on post-approval data, once a drug is approved, enrolling patients in studies can become more difficult because patients are less incentivized to participate in clinical studies since they can receive the drug without participating in a trial.14 When such studies are completed, the results must be reported at and may also be published in the peer-reviewed literature.25

FDA medical device regulation has undergone a similar transformation of enabling expedited approvals with increasing reliance on post-marketing data. The FDA introduced an expedited access pathway for medical device review in 2015.26 This pathway was created under the premise that it may be appropriate to accept greater uncertainty in order to expedite availability of devices to patients, with greater dependence on post-marketing studies.27 The 21st Century Cures Act, signed into law on December 13, 2016, created a Breakthrough Devices Program28,29 analogous to the preexisting one for drugs. However, post-approval commitments for medical device clinical studies are often not fulfilled;30 within three to five years after medical device approval, only 13% of post-marketing clinical studies are completed.31


Given the parallel trends of expediting drug and device approval with somewhat less reliance on pre-market data and the challenges of conducting traditional RCTs, there has been a move toward novel ways of generating evidence. In recent years, the volume and complexity of electronic health care real-world data, which can in turn be used for RWE medical product evaluations, has grown exponentially. Although observational data have been used to study medical products for decades, the granularity and complexity of electronic health record (EHR) data, coupled with EHRs’ linkage to longitudinal data for outcome ascertainment, and advances in data science analytics, together offer substantial promise to leverage these data for medical product evaluation, informing regulatory decision-making, P&T coverage decisions, and ultimately clinical practice.

Defining Real-World Evidence

RWE is defined in Section 3022 of the 21st Century Cures Act (which creates Section 505F of the Federal Food, Drug, and Cosmetic Act) as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than randomized clinical trials.”28,32 Although the law appears to suggest that randomized data do not make up RWE, FDA authors have clearly stated that this is not the case, writing that it is “incorrect to contrast the term ‘real-world evidence’ with the use of randomization in a manner that implies that they are disparate or even incompatible concepts.”1 More recently, FDA authors clearly said that it was “not the intent of Congress” that single-intervention clinical trials be used to generate RWE.33

The FDA has defined RWE as “information on health care that is derived from multiple sources outside typical clinical research settings, including EHRs, claims and billing data, product and disease registries, and data gathered through personal devices and health applications.”1 In August 2017, the agency finalized a guidance entitled “Use of Real-World Evidence to Support Regulatory Decision Making for Medical Devices.”34 This guidance made the key point that it does “not change FDA’s evidentiary standards for regulatory decision-making.”34

Potential of RWE

RWE can efficiently provide relevant clinical data in populations representative of those seen in clinical practice. Clinical trials are often conducted in populations who are younger, more often male, and less racially and ethnically diverse than those seen in clinical practice.3538 Everyday practice also includes other important clinical factors such as patient adherence, tolerance, comorbidities, concomitant treatments, study location, and environment, all of which are more standardized and controlled in RCTs.10 RWE may also better account for complexities—such as the fact that medications need to be tested in combination with other medications during development. Further, the patient populations receiving medical products will evolve over time; RWE can better take these factors into account, since repeating randomized trials may be implausible. Finally, RWE is rooted in clinical practice, where value considerations are becoming increasingly important.10,39 In some cases, it may be difficult (because of rapid innovation and development cycles) or unethical to randomly assign people to one treatment or another; existing real-world data may offer a reasonable substitute for prospective trial data.27

Additionally, RWE has the capacity to inform regulatory decision-making. Safety signals for new drugs and devices may not emerge until the treatments are available in clinical practice, when a larger number of patients receive them and can be followed for a longer duration than in clinical trials. Efficient detection of adverse events through RWE could inform post-market use, including drug or device label changes. Similarly, RWE may help to support new indications for therapies; a recent example is the FDA approval of additional indications for transcatheter heart valves by using data collected in a post-market valve registry that had been mandated for Medicare coverage.40 Therefore, RWE is increasingly being used to inform post-approval safety monitoring and additional indications.

Incorporating the Patient’s Voice

RWE can also better represent patients’ voices and experiences, in part through patient-generated data. Section 3011 of the 21st Century Cures Act requires the FDA to include a statement on the use of patient experience data in its regulatory decisions.28 Patient experience data was defined in the legislation as “data collected by any persons (including patients, family members and caregivers of patients, patient advocacy organizations, disease research foundations, researchers, and drug manufacturers).”28 The goal of these data is to inform the FDA about the impact of a disease or therapy on patients as well as patient preferences. While incorporating patient input regarding values and preferences will be very valuable and is already done in some RCTs through patient-reported outcomes, these data should be rigorously and systematically collected. Patient advocacy organizations may play an important role in generating and communicating these data, but at least two-thirds of these organizations receive industry funding, which may introduce conflicts of interest.41,42 Current disclosure practices of most advocacy organizations are limited,42 and these groups may receive funding from multiple companies.

Strengths and Limitations of RWE

While RWE clearly has demonstrated significant potential—which will undoubtedly continue to mature and expand—important caveats must be considered when determining if RWE can be relied upon for decision-making. RCTs have been the gold standard for decades primarily because randomization allows for the balancing of measured and unmeasured confounders, thus minimizing bias. The central limitation of using RWE for medical product evaluation is that it ultimately asks us to make inferences and evaluations from observational data, which can sometimes be a precarious endeavor because of the presence of unmeasured confounders.43 There is a need to advance statistical methods for reliable analyses of RWE.44 Understanding the various data sources that may comprise RWE is critical; they have different strengths and weaknesses. Used appropriately, RWE can both help check for consistency and also provide different perspectives on important clinical research questions.10

Ideal Data Source

The ideal data source used to inform health care–related decisions would include a representative sample of patients with the underlying condition—both when considering demographic as well as clinical factors (Table 1). This data source would be planned prospectively and would offer continuously updated longitudinal follow-up for a full array of clinical outcomes, including those related to patient-reported measures of health and well-being. This data source would also have quality control measures in place to ensure data validity. To minimize the data collection burden, such a data source would integrate with existing data systems and require minimal resources to collect data.

While no data source has these ideal characteristics, studies should be conducted with the most rigorous possible design by investigators without bias and using appropriate analytic approaches through familiarity with the research database. RWE is likely to consist of multiple real-world data sources.44 With this in mind, we will discuss different types of RWE.

Clinical Registries

Clinical registries are heterogeneous and numerous. Some of the largest registries have been established by professional societies. For example, within cardiology, the American College of Cardiology’s National Cardiovascular Data Registry includes 14 unique registries encompassing both outpatient and inpatient care, including specific procedures such as implantable cardioverter-defibrillator placement. Registries involve either systematic collection through trained abstractors or automatic capture of standard electronic data elements. Advantages of patient registries include clinically rich and generally consistent data. However, registries may not capture longitudinal data or may require significant effort and resources to do so. Additionally, registries may not necessarily be representative of the general clinical use of a given drug or device, and they may not include the full spectrum of outcomes, such as patient-reported outcome measures. Registries often exist in parallel to existing data collection systems, resulting in some duplication for greater accuracy, and they may require resources such as dedicated, trained staff. Finally, registry data may not be available in a timely manner, and therefore may lag behind clinical practice. Registries are often limited to the specific procedures, diseases, or settings that they are designed to capture.45 An example of registry data assisting in evaluation of a medical device is a registry-based analysis of the effectiveness of intra-aortic balloon pumps among patients undergoing high-risk percutaneous coronary intervention, which found significant hospital-level variation without any in-hospital mortality differences.46 The value of registries may be shown by the creation of a “Registry of Patient Registries” by the Agency for Healthcare Research and Quality.47

Claims Data

Administrative claims data are even more ubiquitous than clinical registries. Examples of claims databases include Medicare or any other insurance payer’s claims data, OptumLabs Data Warehouse (the health services platform for United Healthcare, which includes longitudinal data for more than 100 million people),48 and Premier (a health care alliance of more than 3,500 U.S. hospitals and 100,000 providers).49 Advantages of claims data include the ease of data collection and abstraction because these data are universally created for billing purposes, the ability to capture longitudinal data (provided that a patient remains within the same insurer’s system), and widely accepted coding standards that can support data consistency.45 Claims data can often be integrated with existing data platforms. However, claims data are not collected with the goal of supporting research.1 Their disadvantages relate to coarseness with a lack of detailed clinical information, including inability to identify the use of a specific drug or device, uncertainty about adequate risk adjustment, and the movement of people from one insurance plan to another. Additionally, claims data may not be able to differentiate comorbidities from complications and may not be complete within a given episode of care.45 Claims data also suffer from time lag, since coding is performed after an episode of care. Finally, claims data may be inaccurate;50 a study comparing claims to independent study physician adjudication within an observational cardiovascular study of adenosine diphosphate receptor inhibitors found that claims were only modestly accurate in identifying stroke and myocardial infarction and had even more limited accuracy for bleeding events.51 Claims data are quite accurate for identifying health care utilization, such as hospitalizations.52 As an example of claims data assisting in evaluation of a medical device, a study using New York state claims data compared patients receiving laparoscopic sterilization with hysteroscopic sterilization over a nine-year period and showed patients receiving hysteroscopic sterilization were more likely to undergo reoperation.53 Given that the Essure device (Bayer) was the only hysteroscopic sterilization therapy available during the period in question, the study served as an evaluation of that device.

Electronic Health Record Data

EHRs have among the greatest potential to act as a source of RWE. The Health Information Technology for Economic and Clinical Health (HITECH) Act, signed into law as part of the American Recovery and Reinvestment Act of 2009, legislated the adoption of electronic medical records.54 Consequently, the past decade has witnessed increasing and now near-universal adoption of EHRs, making them a ubiquitous source of data. Over the past five years, the PCORnet common data model has been developed; it is a distributed research network with data from multiple clinical data research networks (EHR-based) and patient-powered research networks mapping data in a consistent format to a single common data model.55 However, as with claims data, EHRs are not designed to support research,1 and they are infrequently optimized for this purpose. Advantages of EHRs include detailed clinical information, which consists of demographics, diagnoses, narrative text notes, electronic procedure and test reports, laboratory data, vital sign records, medication lists, order/entry, and other items.45 If patients remain in the same health care system, EHR data will also be longitudinal. Disadvantages of EHRs include their lack of interoperability and uneven data quality. If patients move between health systems with different EHRs, data will be incomplete. Additionally, while machine learning tools such as natural language processing are increasingly being applied to EHRs, including for purposes such as monitoring pharmaceuticals for adverse events,56 the unstructured nature of much of the data often makes EHR-based analyses cumbersome. Additionally, EHRs seldom systematically collect patient-reported outcome measures in an abstractable format,57 although this information can sometimes be found in clinician notes. Evolving reimbursement paradigms that are increasingly focused on quality and value of care may incentivize more accurate data entry by providers.10 An example of EHR data being used for evaluation of a medical device is a study conducted at three institutions using their own EHRs and finding that patients receiving the Thoratec Heartmate II left ventricular assist device were experiencing higher rates of device thrombosis than in the past, and that this adverse event was associated with significant morbidity and mortality.58

Pragmatic Clinical Trials

Among the most important promises of new RWE-generating mechanisms is the ability to conduct pragmatic clinical trials, in which research is embedded within routine clinical practice among patients who best represent the population of patients with a disease while maintaining the rigor of randomization.59 As traditional trials are focused on demonstrating efficacy, there is concern that risks are underestimated and benefits are overestimated.59 The hope is that pragmatic trials can address this through enrollment of larger study populations more representative of people with a given condition, including fewer restrictions on the use of concomitant therapies or on inclusion of patients with other comorbid diseases. Pragmatic clinical trials are also cheaper than traditional RCTs and may be able to obtain data on a larger number of clinical outcomes.9 Disadvantages of these trials include their requirement of an infrastructure to facilitate enrollment. There are also concerns related to the data source; with EHRs, as noted above, there may be loss to follow-up and/or incomplete or inaccurate ascertainment of outcomes because EHRs are not designed specifically for research purposes. Such studies will work only for certain endpoints that require limited monitoring.59 There may also be a higher occurrence of protocol violations.9 Whether pragmatic clinical trials will be acceptable for new pharmaceutical and device approval remains unknown.9 An example of a pragmatic clinical trial is the Post-Myocardial Infarction Free Rx Event and Economic Evaluation (MI FREEE), in which insurance-plan sponsors were randomly assigned to full prescription coverage or usual prescription coverage for patients discharged after hospitalization for myocardial infarction to understand if eliminating patient copayments would improve cardiovascular events and adherence. Claims data were used to both identify potential study participants and evaluate clinical trial outcomes, including a primary endpoint of first major vascular event or revascularization.60,61

Active Surveillance Systems

Active surveillance systems are another important source of RWE that can be used to evaluate medical product safety after approval. An active surveillance system tracks experience with devices and monitors prospectively analyzed data, with triggers set for deviation from expected outcomes.62 Advantages of active surveillance systems include their rapidity and minimal marginal costs because they rely on available real-world longitudinal data. Disadvantages include their limitation to tracking products for which relevant data are available and the need for methods and data to match patients receiving various devices. Findings from active surveillance systems also require validation, such as that offered by medical chart review or more formal epidemiologic studies, as there may be residual confounding from covariates not included in the models.62 Active surveillance systems require standardized event terms with unique methodology that must be continuously developed and reiterated for use across multiple data sources for different products. An example employing an active surveillance system is use of the DELTA (Data Extraction and Longitudinal Trend Analysis) system in a study conducted within the National Cardiovascular Data Registry’s CathPCI Registry that identified a specific femoral vascular-closure device as associated with a higher risk of vascular complications compared with alternative vascular-closure devices.63

Emerging Technologies

Emerging technologies (digital health) have the potential to contribute RWE in decision-making related to the use of medical products. Rapid technological innovation over the past several years has led to the development of thousands of health-related mobile applications, wearable technologies such as fitness trackers, and sync-able technologies such as digital weight scales and portable electrocardiographic sensors. These technologies are contributing to progressively greater data streams. They generally involve patient-generated data and also offer the prospect of recruiting patients virtually.10 For example, the Health eHeart Study is a cardiovascular-focused eCohort in which people enroll solely using the Internet and can enter their medical history and lifestyle habits, as well as connecting multiple devices and applications.64

Advantages of emerging technologies include the ability to capture detailed and longitudinal information, including patient-generated data and patient-reported outcomes. Given that nearly all of people’s activities occur outside of health care settings,10 these technologies can provide rich data for a more complete understanding of health. Disadvantages of these technologies include the potential absence of representative populations. These technologies may not be adapted into health care data platforms and may capture limited outcomes. Accuracy and validation of data from these novel technologies are also important; data suggest wearable devices may have up to 20% variation in obtaining accurate step counts,65 and blood pressure measurements from an instant blood pressure smartphone application have been shown to be inaccurate.66 Patient-reported outcome measures obtained electronically have generally been found to be equivalent to traditional paper-based reports if there are only minor changes.67 Emerging technologies also need to navigate infrastructure, legal, and privacy-related concerns. The FDA recently launched a Digital Health Innovation Action Plan, including a Pre-Certification for Software Pilot Program to help set expectations about regulation of digital health products.68 The potential of emerging technologies was demonstrated in a recent smartphone application monitoring study that recruited nearly 50,000 participants over an eight-month period.69 More than 40,000 participants uploaded data related to activity and sleep patterns through both sensors and patient questionnaires. Social media data may also contribute to this area, such as streams from Twitter and Google, which were shown in 2009 to estimate weekly influenza activity in the United States.70

Data Sharing Platforms

Data sharing platforms may also serve as an important resource in an era where regulatory decision-making increasingly relies on RWE. Through a learned intermediary or directly, data generators can share their clinical study data, allowing for investigators and regulators to maximize the value of the clinical trials that have been conducted. Data can be aggregated to allow for a more comprehensive understanding of drug or medical device safety and efficacy. Momentum for data sharing and transparency has increased in recent years. An example is the National Heart, Lung, and Blood Institute (NHLBI) Biologic Specimen and Data Repository Information Coordinating Center, which coordinates activities of the NHLBI biorepository and data repository and has been used to publish 277 articles from 47 clinical trials.71 Advantages of data-sharing platforms include the low cost of examining detailed clinical data, which can be used to conduct additional studies,72 systematic reviews, and comparative effectiveness studies, as well as to validate published results.73 Disadvantages are that data-sharing platforms are limited by data collected within clinical trials, with their attendant limitations such as external generalizability. An example of using data sharing for pharmaceutical evaluation was a re-evaluation of a double-blind, randomized trial funded by Smithkline Beecham and published in 2001 showing paroxetine was an effective treatment for depression in adolescents.74 Re-analysis of data by independent investigators, however, showed that paroxetine had no clinical benefit.75

RWE and P&T Committees

There will be growing reliance on RWE for health care decision-making in the future, along the spectrum from FDA approval to clinical decisions made between physicians and patients at the bedside. As the FDA increasingly moves toward a life-cycle approach to regulation, this will mean that more continuous evaluation of technologies and treatments will be necessary through post-market data, including assessments by P&T committees. Responding to new and emerging safety information will be particularly important given that the FDA takes a serious safety-related action for nearly one-third of drugs after their approval, including a withdrawal, imposing a boxed warning, or issuing a communication directly to clinicians.76 While real-world data sources have more often been used for safety surveillance than for effectiveness evaluations, P&T committees will similarly need to monitor and respond to new and emerging data on medical product effectiveness. Use of post-market data to inform effectiveness evaluations is becoming increasingly important as drugs and devices are approved on more limited evidence, but there may be greater bias in effectiveness evaluations, and these require the ability to critically evaluate RWE and how “close to ideal” the data source may be for the purpose (Table 1), taking into account the strengths and limitations reviewed above. Finally, RWE will increasingly supplement traditional sources of evidence to augment generalizability.

Given the central role of P&T committees in evidence-based decision-making, they will need to evaluate and apply findings from different real-world data sources to inform their formulary development, and they will have to do so more quickly than in the past. Traditional RCTs may become less frequent, supplanted by pragmatic RCTs and evaluations using nonrandomized data; P&T committees will need to consider the totality of this evidence and first evaluate the source of data, methodology, and results. Next, P&T committees should determine if, and how, the results should be considered based on their strength as evidence to inform formulary coverage decisions. When possible, such decisions should be complemented by local experience. Finally, there will have to be more rapid evolution in formulary decisions over time as new evidence becomes available because of the increasing reliance on post-approval data.

P&T committees may also have the opportunity to review internal data generated within the health system over which they have purview, particularly EHR data, which can complement published data from other sources. This will require collaboration with local experts who have both methodological and analytical expertise to consider strengths and limitations. These data have the potential to inform P&T committee decision-making that is gathered from—and more directly relevant to—the unique patient population of the P&T committee’s organization. P&T committees should proactively engage in discussions about medical product evaluation—data sources, methods, and analytic plans—for specific products where additional data would help inform evidence-based determinations; this will ensure that the outcomes observed within their facilities will support their decision-making, addressing the most pertinent and actionable knowledge gaps for their patients.

Once these evaluations have been completed, it is equally important that P&T committees take action. For example, formularies for health plans providing prescription drug coverage to Medicare beneficiaries made statins widely available before and after the 2013 American College of Cardiology/American Heart Association Cholesterol Guideline77 recommended their use to reduce cardiovascular risk; however, in response to these same guidelines, which relegated nonstatin therapies to second-line use given their limited clinical benefits, formularies on average increased in restrictiveness, but they could have become more restrictive to promote the use of the most effective drugs for Medicare beneficiaries.78 Similarly, there are also opportunities for formularies to change in response to new evidence on drug safety in order to promote safer prescribing, such as by increasing restrictions on opioid medications given the ongoing opioid crisis79 and by curtailing access to drugs with safer available alternatives that receive FDA boxed warnings.80


Now more than ever, we are witnessing the emergence of exciting possibilities for the generation of RWE that can support medical product evaluation to inform patients, clinicians, payers, and policymakers—including P&T committees. RCTs remain the gold standard of evidence, but with the challenges facing the traditional research infrastructure, we are likely to see fewer traditional trials in the future and their replacement with pragmatic trials and other forms of RWE. The evolution of research paradigms will continue to accelerate with the availability of larger volumes of electronic health care data. But depending on the data source, there is a significant need to ensure that analyses are reliable and can serve as the foundation for generalizable knowledge about safety and effectiveness. P&T committees can proactively engage in medical product evaluations to focus on policy questions and take action to ensure the highest-quality care for their patients.


Characteristics of an Ideal Data Source to Generate Real-World Evidence

  • Patient population representative of those with the underlying condition (e.g., demographics, clinical comorbidities)
  • Prospectively planned
  • Continuously updated with minimal resources
  • Longitudinal follow-up
  • Rich clinical data: clinician-entered, patient-reported, and patient-generated
  • Quality control measures in place
  • Integrated within existing data systems
Author bio: 
Dr. Dhruva is a fellow and Dr. Ross is a Co-Director in the National Clinician Scholars Program, Department of Internal Medicine, at the Yale School of Medicine in New Haven, Connecticut. Dr. Dhruva is a cardiologist at the Veterans Affairs Connecticut Healthcare System in West Haven, Connecticut. Dr. Ross is a Physician Investigator at the Center for Outcomes Research and Evaluation of Yale-New Haven Hospital in New Haven, Connecticut; Associate Professor in the Section of General Internal Medicine, Department of Internal Medicine, at the Yale School of Medicine; and Associate Professor in the Department of Health Policy and Management, Yale School of Public Health. Dr. Desai is a Physician Investigator at the Center for Outcomes Research and Evaluation, Yale-New Haven Hospital; and Assistant Professor in the Section of Cardiovascular Medicine, Department of Internal Medicine, in the Yale School of Medicine.


  1. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us?. N Engl J Med 2016;375;(23):2293–2297.
  2. Institute of Medicine. The National Academies Collection: Reports funded by National Institutes of Health. Transforming Clinical Research in the United States: Challenges and Opportunities: Workshop Summary Washington, DC: National Academies Press (US), National Academy of Sciences. 2010;
  3. Prasad V, Mailankody S. Research and development spending to bring a single cancer drug to market and revenues after approval. JAMA Intern Med 2017;177;(11):1569–1575.
  4. DiMasi JA, Grabowski HG, Hansen RW. Innovation in the pharmaceutical industry: New estimates of R&D costs. J Health Econ 2016;47:20–33.
  5. Sertkaya A, Wong HH, Jessup A, Beleche T. Key cost drivers of pharmaceutical clinical trials in the United States. Clin Trials 2016;13;(2):117–126.
  6. U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Examination of clinical trial costs and barriers for drug development July 25, 2014; Available at: Accessed February 27, 2018
  7. Antman EM, Harrington RA. Transforming clinical trials in cardiovascular disease: mission critical for health and economic well-being. JAMA 2012;308;(17):1743–1744.
  8. Moses H3rdMathesonDH, Cairns-Smith S, et al. The anatomy of medical research: US and international comparisons. JAMA 2015;313;(2):174–189.
  9. Jones WS, Roe MT, Antman EM, et al. The changing landscape of randomized clinical trials in cardiovascular disease. J Am Coll Cardiol 2016;68;(17):1898–1907.
  10. National Academies of Sciences, Engineering, and Medicine. Real-World Evidence Generation and Evaluation of Therapeutics: Proceedings of a Workshop Washington, DC: The National Academies Press. 2017;
  11. Food and Drug Administration. Total product life cycle (TPLC) data sources and disclaimers  Available at: Accessed February 27, 2018
  12. Downing NS, Krumholz HM, Ross JS, Shah ND. Regulatory watch: characterizing the US FDA’s approach to promoting transformative innovation. Nat Rev Drug Discov 2015;14;(11):740–741.
  13. Food and Drug Administration. Priority review  Available at: Accessed February 27, 2018
  14. Darrow JJ, Avorn J, Kesselheim AS. New FDA breakthrough-drug category—implications for patients. N Engl J Med 2014;370;(13):1252–1258.
  15. Kesselheim AS, Wang B, Franklin JM, Darrow JJ. Trends in utilization of FDA expedited drug development and approval programs, 1987–2014: cohort study. BMJ 2015;351:h4633
  16. Food and Drug Administration. CDER breakthrough therapy designation requests received by fiscal year December 31, 2017; Available at: Accessed February 27, 2018
  17. Moore TJ, Furberg CD. Development times, clinical testing, post-market follow-up, and safety risks for the new drugs approved by the US Food and Drug Administration: the class of 2008. JAMA Intern Med 2014;174;(1):90–95.
  18. Food and Drug Administration. Accelerated approval program  Available at: Accessed February 27, 2018
  19. Robb MA, McInnes PM, Califf RM. Biomarkers and surrogate endpoints: developing common terminology and definitions. JAMA 2016;315;(11):1107–1108.
  20. Food and Drug Administration. Post-marketing requirements and commitments: introduction  Available at: Accessed February 27, 2018
  21. Pease AM, Krumholz HM, Downing NS, et al. Postapproval studies of drugs initially approved by the FDA on the basis of limited evidence: systematic review. BMJ 2017;357:j1680
  22. Kim C, Prasad V. Cancer drugs approved on the basis of a surrogate end point and subsequent overall survival: an analysis of 5 years of US Food and Drug Administration approvals. JAMA Intern Med 2015;175;(12):1992–1994.
  23. Naci H, Smalley KR, Kesselheim AS. characteristics of preapproval and postapproval studies for drugs granted accelerated approval by the US Food and Drug Administration. JAMA 2017;318;(7):626–636.
  24. Sharfstein JM, Stebbins M. Enhancing transparency at the US Food and Drug Administration: moving beyond the 21st Century Cures Act. JAMA 2017;317;(16):1621–1622.
  25. Cruz ML, Xu J, Kashoki M, Lurie P. Publication and reporting of the results of post-market studies for drugs required by the US Food and Drug Administration, 2009 to 2013. JAMA Intern Med 2017;177;(8):1207–1210.
  26. Food and Drug Administration. Guidance for industry and Food and Drug Administration staff: expedited access for pre-market approval and de novo medical devices intended for unmet medical need for life threatening or irreversibly debilitating diseases or conditions April 13, 2015; Available at: Accessed February 27, 2018
  27. Faris O, Shuren J. An FDA viewpoint on unique considerations for medical-device clinical trials. N Engl J Med 2017;376;(14):1350–1357.
  28. Upton F.H.R.6—21st Century Cures Act. 114th Congress (2015–2016)July 13, 2015 Available at: Accessed February 27, 2018
  29. Food and Drug Administration. Breakthrough devices program: draft guidance for industry and Food and Drug Administration staff October 25, 2017; Available at: Accessed February 27, 2018
  30. Reynolds IS, Rising JP, Coukell AJ, et al. Assessing the safety and effectiveness of devices after US Food and Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med 2014;174;(11):1773–1779.
  31. Rathi VK, Krumholz HM, Masoudi FA, Ross JS. Characteristics of clinical studies conducted over the total product life cycle of high-risk therapeutic medical devices receiving FDA pre-market approval in 2010 and 2011. JAMA 2015;314;(6):604–612.
  32. Food and Drug Administration. Real world evidence  Available at: Accessed February 27, 2018
  33. Jarow JP, LaVange L, Woodcock J. Multidimensional evidence generation and FDA regulatory decision making: defining and using “real-world” data. JAMA 2017;318;(8):703–704.
  34. Food and Drug Administration. Use of real-world evidence to support regulatory decision-making for medical devices August 31, 2017; Available at: Accessed February 27, 2018
  35. Murthy VH, Krumholz HM, Gross CP. Participation in cancer clinical trials: race-, sex-, and age-based disparities. JAMA 2004;291;(22):2720–2726.
  36. Gurwitz JH, Col NF, Avorn J. The exclusion of the elderly and women from clinical trials in acute myocardial infarction. JAMA 1992;268;(11):1417–1422.
  37. Dhruva SS, Redberg RF. Variations between clinical trial participants and Medicare beneficiaries in evidence used for Medicare national coverage decisions. Arch Intern Med 2008;168;(2):136–140.
  38. Dhruva SS, Bero LA, Redberg RF. Gender bias in studies for Food and Drug Administration pre-market approval of cardiovascular devices. Circ Cardiovasc Qual Outcomes 2011;4;(2):165–171.
  39. Sanghavi DM, Conway PH. Paying for prevention: a novel test of Medicare value-based payment for cardiovascular risk reduction. JAMA 2015;314;(2):123–124.
  40. Shuren J, Zuckerman B. How creative FDA regulation led to first-in-the-world approval of a cutting-edge heart valve June 14, 2017; Available at: Accessed February 26, 2018
  41. Rose SL, Highland J, Karafa MT, Joffe S. Patient advocacy organizations, industry funding, and conflicts of interest. JAMA Intern Med 2017;177;(3):344–350.
  42. McCoy MS, Carniol M, Chockley K, et al. Conflicts of interest for patient-advocacy organizations. N Engl J Med 2017;376;(9):880–885.
  43. Hannan EL. Randomized clinical trials and observational studies: guidelines for assessing respective strengths and limitations. JACC Cardiovasc Interv 2008;1;(3):211–217.
  44. The Network for Excellence in Health Innovation. Real world evidence: a new era for health care innovation September 2015; Available at: Accessed February 27, 2018
  45. Rumsfeld JS, Joynt KE, Maddox TM. Big data analytics to improve cardiovascular care: promise and challenges. Nat Rev Cardiol 2016;13;(6):350–359.
  46. Curtis JP, Rathore SS, Wang Y, et al. Use and effectiveness of intra-aortic balloon pumps among patients undergoing high risk percutaneous coronary intervention: insights from the National Cardiovascular Data Registry. Circ Cardiovasc Qual Outcomes 2012;5;(1):21–30.
  47. Agency for Healthcare Research and Quality. Registry of patient registries  Available at: Accessed February 27, 2018
  48. Mayo Clinic. OptumLabs  Available at: Accessed February 27, 2018
  49. Premier. Available at: Accessed February 27, 2018
  50. Fisher ES, Whaley FS, Krushat WM, et al. The accuracy of Medicare’s hospital claims data: progress has been made, but problems remain. Am J Public Health 1992;82;(2):243–248.
  51. Guimaraes PO, Krishnamoorthy A, Kaltenbach LA, et al. Accuracy of medical claims for identifying cardiovascular and bleeding events after myocardial infarction: a secondary analysis of the TRANSLATE-ACS study. JAMA Cardiol 2017;2;(7):750–757.
  52. Krumholz HM, Dhruva SS. Real-world data on heart failure readmission reduction: real or real uncertain?. J Am Coll Cardiol 2017;69;(19):2366–2368.
  53. Mao J, Pfeifer S, Schlegel P, Sedrakyan A. Safety and efficacy of hysteroscopic sterilization compared with laparoscopic sterilization: an observational cohort study. BMJ 2015;351:h5162
  54. Obey D. H.R. 1—American Recovery and Reinvestment Act of 2009. 111th Congress (2009–2010)February 17, 2009 Available at: Accessed February 27, 2018
  55. Fleurence RL, Curtis LH, Califf RM, et al. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014;21;(4):578–582.
  56. Luo Y, Thompson WK, Herr TM, et al. Natural language processing for EHR-based pharmacovigilance: a structured review. Drug Saf 2017;40;(11):1075–1089.
  57. Basch E. Patient-reported outcomes—harnessing patients’ voices to improve clinical care. N Engl J Med 2017;376;(2):105–108.
  58. Starling RC, Moazami N, Silvestry SC, et al. Unexpected abrupt increase in left ventricular assist device thrombosis. N Engl J Med 2014;370;(1):33–40.
  59. Ford I, Norrie J. Pragmatic trials. N Engl J Med 2016;375;(5):454–463.
  60. Choudhry NK. Randomized, controlled trials in health insurance systems. N Engl J Med 2017;377;(10):957–964.
  61. Choudhry NK, Avorn J, Glynn RJ, et al. Full coverage for preventive medications after myocardial infarction. N Engl J Med 2011;365;(22):2088–2097.
  62. Resnic FS, Gross TP, Marinac-Dabic D, et al. Automated surveillance to detect postprocedure safety signals of approved cardiovascular devices. JAMA 2010;304;(18):2019–2027.
  63. Resnic FS, Majithia A, Marinac-Dabic D, et al. Registry-based prospective, active surveillance of medical-device safety. N Engl J Med 2017;376;(6):526–535.
  64. Guo X, Vittinghoff E, Olgin JE, Marcus GM, Pletcher MJ. Volunteer participation in the Health eHeart study: a comparison with the US population. Sci Rep 2017;7;(1):1956
  65. Case MA, Burwick HA, Volpp KG, Patel MS. Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA 2015;313;(6):625–626.
  66. Plante TB, Urrea B, MacFarlane ZT, et al. Validation of the instant blood pressure smartphone app. JAMA Intern Med 2016;176;(5):700–702.
  67. Muehlhausen W, Doll H, Quadri N, et al. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes 2015;13:167
  68. Food and Drug Administration. FDA announces new steps to empower consumers and advance digital healthcare July 27, 2017; Available at: Accessed February 27, 2018
  69. McConnell MV, Shcherbina A, Pavlovic A, et al. Feasibility of obtaining measures of lifestyle from a smartphone app: the MyHeart Counts cardiovascular health study. JAMA Cardiol 2017;2;(1):67–76.
  70. Ginsberg J, Mohebbi MH, Patel RS, et al. Detecting influenza epidemics using search engine query data. Nature 2009;457;(7232):1012–1014.
  71. Coady SA, Mensah GA, Wagner EL, et al. Use of the National Heart, Lung, and Blood Institute data repository. N Engl J Med 2017;376;(19):1849–1858.
  72. Angraal S, Ross JS, Dhruva SS, et al. Merits of data sharing: the Digitalis Investigation Group Trial. J Am Coll Cardiol 2017;70;(14):1825–1827.
  73. Gay HC, Baldridge AS, Huffman MD. Feasibility, process, and outcomes of cardiovascular clinical trial data sharing: a reproduction analysis of the SMART-AF trial. JAMA Cardiol 2017;2;(12):1375–1379.
  74. Keller MB, Ryan ND, Strober M, et al. Efficacy of paroxetine in the treatment of adolescent major depression: a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry 2001;40;(7):762–772.
  75. Le Noury J, Nardo JM, Healy D, et al. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ 2015;351:h4320
  76. Downing NS, Shah ND, Aminawung JA, et al. Post-market safety events among novel therapeutics approved by the US Food and Drug Administration between 2001 and 2010. JAMA 2017;317;(18):1854–1863.
  77. Stone NJ, Robinson JG, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults. J Am Coll Cardiol 2014;63;(25):2889–2934.
  78. Dhruva SS, Desai NR, Karaca-Mandic P, et al. Medicare formulary changes after the 2013 American College of Cardiology/American Heart Association cholesterol guideline. J Am Coll Cardiol 2017;69;(2):244–246.
  79. Samuels EA, Ross JS, Dhruva SS. Medicare formulary coverage restrictions for prescription opioids, 2006 to 2015. Ann Intern Med 2017;19, 167;(12):895–896.
  80. Dhruva SS, Karaca-Mandic P, Shah ND, et al. Association between FDA black box warnings and Medicare formulary coverage changes. Am J Manag Care 2017;23;(9):e310–e315.