Patients Rising Now is welcomes the opportunity to provide recommendations and comments about the pros and cons of ICER’s August 21st document outlining proposed changes to its Value Assessment Framework’s processes and methodology. As advocates for patients with life-threatening and life-altering conditions, we are making suggestions about opportunities for implementing and improving ICER’s processes and activities to better consider the needs of those patients. 

Our comments are organized into the following sections:

  • Patients’ Perspectives and Concerns
  • Methodologies and Processes
  • Health System Improvement and Policymakers’ Perspectives
  • QALYs and Other Analytical Issues
  • Additional Questions

Patients’ Perspectives and Concerns:

Summarizing the Patient Perspective

As we have consistently stated, patients’ perspectives, concerns and viewpoints are not adequately included in ICER’s methodology and overall activities. ICER clearly needs to do more work on receiving and integrating input from “patients, families, and patient organizations, as well as relevant sources of patient-generated evidence.”[i]  We believe that including a “Patient Perspectives” chapter in ICER reports is a needed and helpful improvement; however, we are concerned that ICER will only summarize relevant sources rather than extensively present, discuss, and incorporate patients’ perspectives into its report, analytics, conclusions and metrics – leading to ICER continuing to undervalue patients’ real concerns. As has been seen, such incomplete evaluations can empower payers and others to reflexively deny coverage to patients without organizations doing their own analyses or directly obtaining patient input. To provide a more complete evaluation, we believe ICER should be more explicit about how it will decide what are “relevant sources of patient-generated evidence,” and how it will “summarize” that information in the chapter.

Along those same lines, we are also concerned about how ICER will go about obtaining those perspectives. Specifically, ICER states in the document that it already seeks “patient input through individual patient interviews, focus groups, partnering with patient organizations to conduct surveys, and by requesting existing resources from patient groups….. and will continue to do so,”[ii] but in the past we have noted ICER’s inadequate attempt at a focus group that included only three individuals.[iii] A focus group must be robust enough to capture useful information, and a group of three individuals simply isn’t sufficient. Therefore, we urge ICER to be thoughtful in how it constructs and conducts focus groups, and would suggest it contract with organizations experienced in conducting such research rather than doing it themselves.  We ask that you please provide additional details on how ICER will develop focus groups and related activities, how it will incorporate information obtained from such activities into its analyses and reports, the steps it will take to partner with patient and clinician organizations for such activities, and its plans for retaining experts in conducting such research to gather and analyze that information.

Real World Evidence

ICER’s efforts to use real world evidence (RWE) have historically been insufficient, so we are encouraged that ICER will be seeking to use more RWE, and that ICER will be using RWE as a companion to traditional research information since “randomized controlled clinical trials have their own limitations and are often inadequate to address all questions relevant to assessments of comparative clinical effectiveness.”[iv]

Further, we are encouraged by ICER’s statement about its “commitment to explore forming collaborations with organizations to leverage these kinds of [real world] data for new analyses.”[v] We believe that ICER should work collaboratively with the most insightful experts about each of the disease areas in which it decides to work: clinicians who work with people who have those conditions, and the people who are actually living with those conditions.

ICER Webinars

ICER is proposing “to create a series of webinars that will describe the principles of health technology assessment and economic modeling for a lay audience.”[vi] Since this is part of ICER’s “commitment to facilitate effective stakeholder engagement,” Patients Rising Now is uniquely qualified to collaborate on such an undertaking, given our focus on providing patients with such information in ways that are useful to them.  Patients Rising Now would be very happy to work with ICER to develop, produce and distribute such seminars, and we look forward to ICER’s feedback on this partnership offer.

Ultra-Rare Conditions

It appears that ICER is proposing to no longer examine treatments for ultra-rare conditions using $500,000/QALY as the upper limit for its cost-effectiveness threshold.[vii] We are very concerned that this could significantly reduce the implied incentives for developing new treatments for ultra-rare and orphan conditions. ICER should recognize the need for individualized approaches for analysis for rare and ultra-rare populations – including cost-effectiveness and cost thresholds. Development costs for rare and ultra-rare diseases are not less because they treat fewer people. We believe the framework update should reconsider these equity concerns. The small populations of rare and ultra-rare diseases is precisely why they need to be assessed with more consideration than people living with cardiovascular disease or diabetes.

Methodologies and Processes

Report Updates

We are encouraged that ICER has recognized the need to update reports and note ICER’s proposal to potentially update reports a year after they are finalized. However, new discoveries and additional information about how to diagnose and treat serious health conditions do not follow a yearly cycle.  The FDA and other clinically focused organizations (including the NIH and the CDC) release and publicize new information to guide clinical care, self-care, and policy decisions as soon as they can by updating their documents, websites and other vehicles for communicating critical new health care knowledge to clinicians, patients and policymakers. This contrasts with Medicare coverage update procedures and timelines, and how most private sector companies update premiums and benefits annually for each new plan year. ICER’s proposed arbitrary one-year anniversary update serves neither purpose and is extremely problematic.

If ICER’s goal is to provide information for U.S. insurance companies to make coverage and payment decisions, then report updates should be made in the middle of each year so plans can integrate that information into their internal health technology assessment process for their next plan year’s offerings. However, if ICER’s goal is to provide information for policymakers, clinicians and patients, then report updates should be made whenever there is new information related to the benefits, risks, and use of specific treatment and diagnostic options.  While NICE in the UK serves a very different function from that of ICER because it is directly connected to a health care payment and delivery system, we note that NICE has a very specific standard for updating its reports based upon new evidence: “Once published, all NICE guidance is regularly checked, and updated in the light of new evidence or intelligence if necessary.”[viii] We can see no reasonable rationale to explain how the 1-year anniversary updating policy would benefit specific stakeholders, and urge ongoing real-time updates. 

Per-patient Budget Impact

ICER’s proposal to “present a cumulative per-patient potential budget impact for each year over the five-year time horizon”[ix] is an interesting step toward more granular data analysis.  However, we strongly suggest that ICER also present information and analyses of patients in different types of insurance coverage situations, e.g., Medicare versus private high-deductible insurance versus ACA Marketplace Silver benchmark plans.

Health System Improvement and Policymakers’ Perspectives

“Real World” Health Care Delivery and Finance

We believe it is extremely important for ICER as an organization to understand, communicate, and incorporate into its work the multifaceted nature of health care delivery and financing in the U.S.  Health care delivery and finance is complicated both by the patchwork nature of the public-private system and the independent authorities of the 50 states, five territories and the District of Columbia in regulating various aspects of health care within their borders and as governed by their own constitutions, laws and regulations. Anyone seeking to improve health care in the U.S. and create a “more effective, efficient, and just health care system”[x] must work within a reality-based framework; otherwise, those efforts will not translate to the real world.

We are concerned that ICER’s thinking ignores the U.S. health financing structures and public and private expenditure decision processes.  For example, with this statement: “Societal resources may be drawn into health spending from other domains of social spending that are much more important to people with lower incomes (such as public education). Some people with lower incomes are likely to be forced out of insurance markets all together,”[xi] ICER ignores the ACA’s low-income subsidies and Medicaid expansion.  We are concerned, therefore, that ICER’s value framework will not be reality-based, and therefore will not be a reliable measure of the real world.

Inaccurate or Incomplete Understanding of Current Health Care Landscape

We found it refreshing that ICER noted one of the unusual aspects of health care in the U.S.:  “estimates of demand based on current funding may be distorted because health insurance is a tax-credited employment benefit, meaning that health insurance coverage decisions do not necessarily match population preferences.”[xii] However, we are troubled that ICER appears to misunderstand the details of the situation. The U.S. method for financially incentivizing health insurance for individuals is primarily a tax deduction rather than a tax credit; and the tax deduction is only available within the realm of employer-provided health benefits and the self-employed who show a profit. Further, employed individuals who purchase their own health insurance (i.e., people who work for small employers) and Medicare beneficiaries (who pay Medicare Part B and D premiums) only get a tax deduction for total health expenses (including premiums) that exceed 7.5% of their Adjusted Gross Income.[xiii]

On a related front, we are encouraged that ICER recognizes that the U.S. health care system is composed of multiple payers, but again, we are concerned ICER misunderstands the roles and responsibilities of policymakers as expressed in this statement: “While ICER acknowledges that different payer types may have different perspectives, we believe it is important for policymakers not to view health investment as less worthwhile if the return on investment is realized by a different (type of) payer. ICER’s economic evaluations will therefore continue to be conducted using a broad health care sector perspective, with a societal perspective as a scenario analysis. ICER may consider payer-specific analyses where considered particularly relevant and when data are available.”[xiv] This myopic and inaccurate perspective on how policymakers operate and the kind of information they need to enact good policy is alarming. 

We are similarly perplexed by ICER’s misrepresentation of health care decision makers with this statement: “Population-level decisions and policies have always been made by life science companies, insurers, and clinical organizations looking at evidence in the same general way.”[xv] We are aware that companies of all sorts look at evidence; that is not in dispute. However, there are significant reasons that health care connected companies (life sciences, clinical, and insurers) do not look at overall population data and evidence “in the same general way,” but rather look at data and evidence that is relevant to their specific groups and geographic locations of interest. For example, the Veterans Administration does not look at children, while Medicaid programs clearly are concerned about populations of children. The Maine Medicaid program is not concerned about Dengue fever or Chagas, but the Texas Medicaid program might be concerned about such infectious diseases.

Similarly, a life science company developing a new treatment is concerned about the groups in the U.S. covered by different insurance programs, such as Medicaid, Medicare, private insurance, and so forth. That data is very important as they model how a new treatment might be used and how much revenue they might receive from the new treatment.

During the September 4th webinar, ICER’s President stated that “Our value framework is for the population. It’s really meant to reflect…. Yes, I mean it has the patient perspective built in, but it’s built in in a way that will in a sense support a population framework for policy making because we have one price across the population. You know, we can try to find subpopulations in which there might be a better value-based price, but there’s probably going to be one price. There’s going to be one coverage policy unless we can find good subpopulation data” (emphasis added).[xvi] We note that if this was the case, then the current controversy over surprise billing would not be happening, since those uniform prices would be known and all payers would already be paying the same for health care products and services.

Despite what ICER seems to think, there is not one price for medicines in the U.S., and each insurance program will be providing different revenue amounts per patient, and the total amount from each type of payer will depend on the number of people covered by each insurance program – some treatments are used primarily by children with Medicaid, others by the elderly with Medicaid, and so forth.  We are extremely alarmed at this fundamental mischaracterization of drug pricing in the U.S., and urge ICER to adopt a more realistic approach to the issue going forward.

QALYs and Other Analytical Issues

We are encouraged that ICER recognizes that “Ethicists, health economists, and health technology assessment groups have long recognized that pure QALY maximization does not incorporate all the values that societies wish to consider when making prioritization decisions for health care spending.”[xvii]  However, that statement belies the fact that ICER still uses QALYs as a central factor in its analyses, and is proposing to keep QALYs in a prominent position with its overall value framework process into 2020 and beyond.

Mischaracterization of QALY

ICER describes the QALY as “the gold standard for measuring how well a medical treatment improves and lengthens patients’ lives,”[xviii] but that assertion is questionable at best.  While the QALY may have “served as a fundamental component of cost-effectiveness analyses in the US and around the world for more than 30 years,”[xix] its use as an analytical tool does not automatically make QALY appropriate or beneficial as a decision guide for coverage or reimbursement.  We urge ICER use rhetoric that more appropriately describes the QALY, and also improve its description of what ICER’s overall work and products actually are.

Additional Assessment Tools

ICER has stated that it will “focus on the need for further research into methods for quantitative and/or qualitative incorporation [of additional dimensions of value] into technology assessments [because] methods for empirically integrating them into a value-based price are not well established,”[xx] but ICER has already started using evLYG, a new and untested assessment metric, without such evaluation or review. Similarly, ICER changed how it calculates the Budget Impact Threshold earlier this year by going from a 2-year average of FDA approvals to a 5-year average.[xxi] Previously, ICER has stated that it will make updates outside of its every 3-year updates as it deems appropriate, but we are concerned that this means ICER will continue to make adjustments to its processes and methodologies without adequate – or any – outside consultation, particularly with patient or clinician expert groups, and without reasonable announcements or explanation of its rationale for the changes.  By making changes to its processes between announced updates, ICER is creating additional confusion that erodes confidence in any products ICER may produce.  

Budget Impact Calculations

In calculating budget impact, ICER will “assume that an equal proportion of patients (20%) would be treated with the new treatment each year over five years, arriving at a cumulative 100% uptake at five years.”[xxii] With this arbitrary assumption, ICER is ignoring existing knowledge and expert insights: a great deal of research has been done on adoption of innovations – both conceptual (e.g., Rogers[xxiii]), and practical (e.g., actual market adoption information for all manner of innovations).  We recognize that ICER’s intent is “NOT meant to represent our assumptions of the budget impact of new interventions that are most likely in the real world,”[xxiv] but then this explicitly raises the question as to ICER’s thinking and rationale for how an analysis based on entirely unrealistic assumptions would be an appropriate starting place for competent and responsible policymakers. We are totally unaware of any policymakers who would ever consider basing their decisions on analyses that were NOT based on some factual information grounded in what was at least an attempt at real world assumptions and modeling – and particularly for their population or program of oversight or concern.  We would welcome a specific explanation of the utility of ICER’s analytical vision as it relates to real world policymakers. 

Also concerning is the fact that in setting its budget threshold amounts, ICER uses mean rather than median income as the basis for its QALY thresholds, which may not be appropriate. As Bang and Zhao have concluded, “mean and median-based QALYs


be considered together as complementary tools in CEA for informed decision making, acknowledging the pros and cons of each.”[xxv] And further, that the “median might be an important parameter that may be more relevant to consumers than providers, more to patients and practicing physicians than policymakers, and more on the individual level than the societal level.”[xxvi]ICER should explain why it is using mean income ($54,420 in 2018) rather than median ($63,179 in 2018),[xxvii] particularly because most lower income people receive financial assistance to cover health care costs, meaning that the major economic impact of health care costs in the U.S. is on middle income individuals and families.

Additional Questions

  • In choosing to implement the Likert Scale for voting by appraisal committees, why was the 3-point scale chosen, and not a 5-point or 7-point scale, either of which would permit more nuanced information and prioritization of perspectives from advisors?  What analysis did ICER undertake to determine that the 3-point scale was the appropriate range?
  • It is our understanding that the German rating system is based upon evaluation of actual clinical trials, whereas ICER’s analysis is based more often on meta-analyses, making the two incompatible.  Why is ICER proposing to also include ratings based on the German system?[xxviii]

  • For the new Comparative Effectiveness Rating Matrix graphic, given our understanding of the new meanings for C++ and C-, we believe that the ordering in the graphic should be changed so that C++ is above C+, which would create a more orderly progression of potential benefit and certainty in the ratings scale ICER has chosen to deploy.  If that is not correct, we ask that ICER please explain the new ratings matrix in more detail because it is unclear.

In conclusion, we continue to be very concerned that ICER undervalues actual patient perspectives, concerns, experiences, and views. If ICER’s goal is to “help the United States evolve toward a health care system that provides fair pricing, fair access, and a sustainable platform for future innovation,”[xxix] then its process should support that goal.  The only way to do that is by creating models and using assumptions that reflect the reality of the U.S. health care ecosystem.  Because the health care “system” in the U.S. is multi-payer and pluralistic, continuing to conduct unsophisticated assessments that suppose a uniform integrated system means ICER’s work is not only unhelpful, it could be destructive if it leads to uninformed decisions and bad policy. 

[i] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 39

[ii] Ibid., P. 43

[iii] Patients Rising Now Letter to ICER “Draft Evidence Report “Esketamine for the Treatment of Treatment-Resistant Depression,” April 15, 2019

[iv] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 5

[v] Ibid., P. 6

[vi] Ibid., P. 46

[vii] Ibid., P. 15


[ix] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 37


[xi] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 17

[xii] Ibid., PP. 16-17


[xiv] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 26

[xv] Ibid., P. 3

[xvi] “ICER’s Value Framework and Patient Feedback: Q&A with Steve Pearson, September 4, 2019, [Time code 57:22]

[xvii] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 31

[xviii] Ibid., P. 11

[xix] Ibid.,  P. 11

[xx] Ibid., P. 13


[xxii] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 36

[xxiii] “Diffusion of Innovations,” by Everett M. Rogers

[xxiv] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 36

[xxv] “Median-Based Incremental Cost-Effectiveness Ratio (ICER),” J Stat Theory Pract. 2012 ; 6(3): 428–442

[xxvi] “Cost-Effectiveness Analysis: a proposal of new reporting standards in statistical analysis,” J Biopharm Stat. 2014; 24(2): 443–460


[xxviii] “2020 Value Assessment Framework – Proposed Changes,” August 21, 2019, P. 10

[xxix] Ibid., P. 2