Skip to main content

Enhance Transplant Program Performance Monitoring

Toolkit sectionsToolkit
Click to go to the background sectionBackground
Click to go to the policy changes sectionPolicy changes
Click to go to the implementation timeline sectionImplementation timeline
Click to go to the Risk-adjustment and performance evaluation sectionRisk-adjustment and performance evaluation
Click to go to the post implementation monitoring sectionPost implementation monitoring
Click to go to the education and resources sectionEducation and resources

Background

In December 2021, the OPTN Board of Directors approved changes to the OPTN Management and Membership Policies (previously contained in the OPTN Bylaws) that allow for a more holistic evaluation of transplant programs. OPTN Management and Membership Policies (PDF) outline membership requirements for organizations and individuals to participate in the U.S. transplant network.

The OPTN Membership and Professional Standards Committee (MPSC) had historically used 1-year post-transplant graft and patient survival to identify underperforming transplant programs. This metric only evaluates one phase of patient care.

The new transplant program performance monitoring system involves four risk-adjusted measures related to the patient journey:

  • 90-day graft survival hazard ratio
  • 1-year conditional on 90-day graft survival hazard ratio
  • Offer acceptance rate ratio
  • Pre-transplant mortality rate ratio (previously waitlist mortality rate)
Waitlist management and post transplant outcomes

These metrics are defined as follows:

  • 90-day graft survival hazard ratio: A program’s rate of graft failure from date of transplant to 90 days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.
  • 1-year conditional on 90-day graft survival hazard ratio: A program’s rate of graft failure from day 90 post-transplant to day 365 post-transplant, conditional on the graft surviving for the first 90-days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.
  • The offer acceptance model reports a program’s actual rate of accepting organ offers relative to the expected rate, based on similar offers across the nation, during a 1-year evaluation interval.
  • The pre-transplant mortality model reports a program’s pre-transplant mortality rate relative to the mortality rate experienced by similar patients across the nation over a 2-year evaluation interval.

The Scientific Registry of Transplant Recipients (SRTR) provides statistical and other analytic support to the OPTN. The SRTR produces the models to calculate each of the four performance metrics and provides the MPSC with reports using these models. These reports allow the MPSC to identify transplant programs that meet the OPTN performance requirements for review. The SRTR has prepared FAQs with more information.

In developing this system, the MPSC focused on areas within the OPTN’s authority under the OPTN Final Rule. The MPSC strives to support and collaborate with transplant programs to provide performance improvement opportunities, while also meeting its responsibility to monitor member performance to identify potential patient safety issues and ensure public trust in the transplantation system. The MPSC aimed to develop a performance monitoring system that provides maximum support for the OPTN strategic plan.

Policy changes

  • As provided in OPTN Management and Membership Policies, Appendix D, D.12.A Transplant Program Performance, the MPSC will review a transplant program’s performance if the transplant program meets the criteria (for either adult transplants or pediatric transplants) for:
    • 90-day graft survival hazard ratio
    • 1-year conditional on 90-day graft survival hazard ratio
    • Pre-transplant mortality rate ratio
    • Offer acceptance rate ratio
  • OPTN Management and Membership Policies, Appendix D, D.12.A. Transplant Program Performance also clarifies, but does not substantively change, transplant program obligations to participate in the performance review process.
  • In addition, Appendix L: Review and Actions now includes a section that codifies the current peer review process.
  • Appendix M: Definitions includes new definitions for the performance metrics and an updated definition of the Scientific Registry of Transplant Recipients (SRTR). Appendix M: Definitions no longer includes definitions of two defunct MPSC subcommittees.

Enhance Transplant Program Performance Monitoring System

Implementation timeline

The OPTN implemented the new performance monitoring system in phases:

The MPSC delayed implementation of the pre-transplant metrics to allow programs time to better understand the metrics, review their applicable data and practices, and make practice changes as desired. Specifically, the MPSC postponed the implementation of the pre-transplant metrics until at least one complete evaluation interval had passed following the December 2021 Board approval. This ensured that the MPSC would only evaluate programs based on transplant program performance on these two metrics using data collected after Board approval.

  • The offer acceptance model reports a program’s offer acceptance practices during a 1-year evaluation interval.
  • The pre-transplant mortality model reports the mortality rate for candidates on the waiting list at any time over a 2-year evaluation interval.

The data collection period for the pre-transplant metrics began Jan. 1, 2022. The offer acceptance rate ratio criteria took effect on July 27, 2023, and the pre-transplant mortality rate ratio was implemented on July 25, 2024.

Risk-adjustment and performance evaluation

The MPSC strongly believes that it is important to use risk-adjusted metrics to have a fair and accurate performance monitoring system. Risk adjustment allows the MPSC to focus on a program’s performance based on elements of patient care that are within the control of the program (i.e., elements of patient care that the program can affect through performance improvement initiatives). In addition, the MPSC believes that the use of risk-adjusted metrics is the best way to avoid risk-averse behavior that could result in unintended negative consequences, such as greater organ non-use (discard) or a decrease in organ transplants.

All four metrics in the new performance monitoring system are risk-adjusted. Risk-adjusted models compare a program’s observed performance to its expected performance. The SRTR attempts to adjust for each program’s respective risk level by taking into account many factors. These include program-specific characteristics of transplant candidates, recipients, and donors. A program’s risk level is reflected in its expected performance number. A program’s performance will be compared to the national performance for patients or offers with similar characteristics.

Some specific questions regarding the importance of risk adjustment are addressed below. For more information on risk adjustment and the SRTR models developed for the MPSC performance evaluation metrics, a presentation titled “MPSC: Enhance Transplant Program Performance Monitoring System: Proposed Metrics” may be viewed here. This video addresses frequently asked questions.

Without risk adjustment, all programs would be expected to achieve the same observed survival rate, regardless of the acuity of their patients or the quality of the donor organs they transplant. The goal of using risk adjustment is to be able to differentiate between program performance, given the known patient and donor factors.

The MPSC wants to understand if a program’s outcomes differ significantly from what would be expected from that program based on the outcomes of similar patients or organ offers nationally; or, in other words, what is the individual program’s effect on the outcome of interest (graft survival, pre-transplant mortality, offer acceptance).

Some transplants involve greater risk than others. The risk adjustment models do not expect higher-risk transplants to succeed at the same rate as lower-risk transplants. If the higher-risk transplants a program performs are associated with a bad outcome, and the model expects those transplants to have a poorer outcome, then the program may still be considered to perform as expected.

In this example, assume two liver transplant programs perform the same number of transplants and that most of the donor and recipient characteristics are the same between the programs. One program utilizes a high volume of donor after cardiac death (DCD) livers, while the other program does not use any.

  • The model expects that liver transplants using a DCD liver have an approximately 60 percent higher risk of graft failure based on national data.
  • Because of risk adjustment, the model would expect the program that uses DCD livers to have more graft failures (a lower “expected” rate) than the program that uses none.
  • However, a program that rarely performs DCD transplants would be expected to have fewer poor outcomes and would be identified as performing worse than expected if the program had poor outcomes in average-risk transplants.

If the SRTR models did not use risk adjustment, the MPSC would not be well-equipped to understand if a program’s outcomes differ from what would be expected based on patient and donor risk factors that are beyond the program’s control, such as the age of the recipient or whether a donor is a donor after cardiac death or donor after brain death. Instead, the MPSC would only have the raw, absolute number of events at a program. The MPSC would not be able to determine easily if those events were similar to events experienced by other programs or were associated with more complex transplants.

The SRTR risk adjustment models are limited to the data currently collected by the OPTN. The MPSC acknowledges that current risk adjustment models cannot adjust for all risks; however, the MPSC does not believe that the lack of perfect risk adjustment supports the complete rejection of risk adjustment. If the MPSC and the transplant community acknowledge that transplant programs have different risk profiles – different patient acuity, different case mix, and different donor availability depending on where they are located and the donor supply - the way to reflect that difference is through risk adjustment.

No. Risk-adjusted models cannot be built for some types of transplants, because there are not enough events or available factors that predict the outcome. For example, there were no predictors identified for pediatric kidney deceased donor 1-year conditional on 90-day graft survival, so a risk-adjusted model was not built.

In cases such as these, the national failure rate is used to determine an expected rate for a program.

The SRTR is continuously improving its risk adjustment models and encourages any feedback from the community and the MPSC. Every six months, the SRTR analyzes national data to determine if the risk associated with a variable has changed and refits the models to capture the current risk profile.

On its website Exit Disclaimer, the SRTR provides information on each of the performance monitoring metric models under the “Tools” tab. This information includes a list of the model elements, the model coefficients, and the element plots that provide a visualization of the relationship between the element and the predicted risk. These tools also include a tab for “Other Elements” that provides a list of elements that the SRTR investigated and found did not add predictive value to the model, or in other words, the element did not improve their ability to predict the outcomes.

In order to include a data element in a risk adjustment model, the variable must be included in OPTN data collection. For any variable with data collected and available, the SRTR can evaluate whether the variable improves the risk adjustment. The SRTR Review Committee determines if a variable should be included in a model.

  1. Any potential adjuster must affect a different proportion of patients at different programs

    For example, if every program in the country used 10 percent donor DCD organs for transplants, DCD would not be a good candidate for adjustment because adjusting for it would not make a difference, since it is the same at all programs. In reality, DCD is a good adjuster because the willingness to use DCD organs varies between programs.

  2. Any potential adjuster must predict, or be associated with, the outcome

    We know from experience that DCD donor organs tend to have poorer outcomes on average than brain dead donor organs. For example, in liver transplants, DCD donors have approximately a 60 percent higher risk of failure nationally than brain dead donors. Yet, the increased risk of using DCD donor organs is acceptable as compared against the alternative outcome of not being transplanted. We want to adjust for this risk.

  3. Any potential adjuster cannot be a treatment by the program that would affect the outcome

    For example, the program’s choice of immunosuppression regimen, which is a treatment decision by the program, may affect the outcome. This is precisely what the MPSC is trying to determine – if there is a treatment practice that is causing or contributing to poorer outcomes. We do not want to adjust that out of the metric, as it may be something the program needs to evaluate and adjust to achieve better outcomes.

MPSC members will evaluate whether factors that are not included in the risk adjustment model appear to be a factor at a program under review. Any factors identified will be referred either to the OPTN organ-specific committee or the OPTN Data Advisory Committee if the data point is not currently collected, or to the SRTR for investigation for inclusion in the model if the data point is currently collected.

Post implementation monitoring

The MPSC is evaluating the effectiveness of the changes to the transplant program performance monitoring system and any unintended consequences at least annually.

The MPSC reviews the number of programs identified by each metric, the number of unique programs identified, and the organ distribution of programs identified semiannually following the receipt of reports from the SRTR.

Fall 2024 MPSC Metrics Overview (PDF)

Due to the staged implementation timeline and variations in the cohort periods, the evaluation metrics included in each annual evaluation report may vary but could include the following:

  • National mean utilization through organ yield rates for each organ (Heart, Kidney, Liver, Lung, and Pancreas) before and after implementation.
  • Number of waiting list additions before and after implementation including:
    • National mean listing rate for each organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability across programs by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability based on risk stratification (e.g., MELD, dialysis time), and demographics (e.g., age, ethnicity)
  • Offer Acceptance rates before and after implementation including:
    • National mean offer acceptance rate by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability across programs by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability based on risk stratification and demographics
  • Waitlist mortality rates before and after implementation including:
    • National mean waitlist mortality rate by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability across programs by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • ariability based on risk stratification and demographics
  • Number of graft failures (or patient deaths for pancreas) during the first year post-transplant before and after implementation
    • National one-year survival rates by organ transplanted (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability across programs by organ (Heart, Kidney, Liver, Lung, and Pancreas)
    • Variability based on risk stratification and demographics

Data Report MPSC Performance (PDF)

Education and resources

Instructional PDFs

In the OPTN learning management system

Use this link to access these professional education resources and learning modules:

  • QLT150: MPSC transplant program performance monitoring
  • QLT151: MPSC post-transplant outcomes and risk adjustment (confers CEPTC credits)
  • QLT152: MPSC Offer Acceptance Rate Ratio and Risk Adjustment (confers CEPTC credits)
  • QLT153: MPSC pre-transplant mortality rate ratio and risk adjustment (confers CEPTC credits)
  • OPTN Offer Acceptance Collaborative Playlist (confers CEPTC credits)
    • The modules have information about:
      • MPSC transplant program performance monitoring, the offer acceptance rate ratio, and OPTN tools to drive improvement.
      • Effective practices related to offer acceptance and hear lessons learned from previous collaborative participants.

Community webinar

The OPTN hosted a webinar for transplant professionals June 28, 2024, with information on the pre-transplant mortality rate ratio metric. Scientific Registry of Transplant Recipients (SRTR) Director Jon Snyder, Ph.D. and SRTR Biostatistician Grace Lyden, Ph.D., joined Zoe Stewart Lewis MD, chair of the OPTN Membership and Professional Standards Committee (MPSC) to discuss the new metric and answered participants' questions. Watch the webinar recording.

Additional resources

Access the Offer Acceptance Collaborative Improvement guide here.

Learn more about monitoring and performance reviews: What to Expect: Performance Reviews (PDF)

SRTR reporting

The SRTR has also added information to transplant programs’ SRTR secure sites to help you understand whether your program meets the new MPSC performance monitoring criteria. This information is intended to support you as you plan and evaluate your program’s performance.

Public comment resources

The proposal's public comment page has information and educational resources about the metrics, including:

  • A video introduction of the changes outlining what's involved
  • A recording of an Aug. 4, 2021, webinar with Rich Formica, M.D., providing a more detailed analysis
  • A September 2021 recording of MPSC chair, Ian Jamieson, and SRTR director, Jon Snyder, discussing the four evaluation metrics involved, including risk-adjustment and frequently asked questions

Contact us

Email MPSCReports@unos.org with questions about the transplant program performance monitoring system.