Skip to main content

Enhance Transplant Program Performance Monitoring

Toolkit sectionsToolkit

Background

In December 2021, the OPTN Board of Directors approved bylaw changes that will allow for a more holistic evaluation of transplant programs. OPTN Bylaws outline membership requirements for organizations and individuals to participate in the U.S. transplant network.

The OPTN Membership and Professional Standards Committee (MPSC) has historically used 1-year post-transplant graft and patient survival to identify underperforming transplant programs. This metric only evaluates one phase of patient care.

The new monitoring system involves four risk-adjusted measures related to the patient journey:

  • 90-day graft survival hazard ratio
  • 1-year conditional graft survival hazard ratio
  • Pre-transplant mortality rate ratio (previously waitlist mortality rate)
  • Offer acceptance rate ratio
Waitlist management and post transplant outcomes

In developing this new system, the MPSC focused on areas within the OPTN’s authority under the OPTN Final Rule. The MPSC strives to support and collaborate with transplant programs to provide performance improvement opportunities, while also meeting its responsibility to monitor member performance to identify potential patient safety issues and ensure public trust in the transplantation system. The MPSC aimed to develop a performance monitoring system that provides maximum support for the OPTN strategic plan by increasing the number of transplants, promoting equitable access to transplantation, and fostering innovation. The committee’s proposal received public comments from Aug. 3-Sept. 30, 2021.

Bylaws changes

  • As outlined in the policy notice, these changes revise the OPTN Bylaws, Appendix D, D.13.A. Transplant Program Performance, and establish new criteria to identify transplant programs that will enter into MPSC performance review. Note: the changes will apply to Appendix D, D.12.A, until all approved changes to the bylaws have been implemented.
  • The changes also affect Appendix M, M.3 Medical Peer Review; Appendix M, M.6 Peer Visits; Appendix M, M.9 Informal Discussions; Appendix M, M.15 Costs; and Expenses Appendix N: Definitions.
  • The MPSC will review a transplant program’s performance if the transplant program meets the criteria (for either adult transplants or pediatric transplants) for:
    • 90-day graft survival hazard ratio
    • 1-year conditional graft survival hazard ratio
    • Pre-transplant mortality rate ratio
    • Offer acceptance rate ratio
  • OPTN Bylaw, Appendix D, D.12.A. Transplant Program Performance also clarifies, but does not substantively change, transplant program obligations to participate in the performance review process.
  • In addition, Appendix L: Review and Actions now includes a section that codifies the current peer review process.
  • Appendix M: Definitions includes new definitions for the performance metrics and an updated definition of the Scientific Registry of Transplant Recipients (SRTR) Exit Disclaimer. Appendix M: Definitions no longer includes definitions of two defunct MPSC subcommittees.

Enhance Transplant Program Performance Monitoring System

Implementation timeline

The OPTN will implement the new performance monitoring system in phases, beginning with implementation of the post-transplant graft survival metrics that are similar to the metrics currently used to monitor performance. 

The MPSC is waiting to implement the pre-transplant metrics to allow programs time to better understand the metrics, review their applicable data and practices, and make practice changes as desired.

Specifically, the pre-transplant metrics will not be implemented until at least one complete evaluation interval has passed following the December 2021 Board approval. This will ensure that the MPSC will only evaluate programs based on transplant program performance on these two metrics using data collected after Board approval.

  • The offer acceptance model reports a program’s offer acceptance practices during a 1-year evaluation interval.
  • The pre-transplant mortality model reports the mortality rate for candidates on the waiting list at any time over a 2-year evaluation interval.

The data collection period for the pre-transplant metrics began Jan. 1, 2022. Transplant programs should be aware that the MPSC may evaluate data collected as of Jan. 1, 2022, in performance reviews that occur in 2023 and 2024.

The SRTR provides statistical and other analytic support to the OPTN. The SRTR produces the models to calculate each of the four performance metrics and will provide the MPSC with reports using these models. These reports allow the MPSC to identify transplant programs that meet the OPTN Bylaw requirements for review. The SRTR has prepared FAQs with more information Exit Disclaimer.

The MPSC will begin evaluating transplant programs based on the new metrics and criteria on the following timeline:

90-day graft survival hazard ratio: A program’s rate of graft failure from date of transplant to 90 days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.

1-year conditional on 90-day graft survival hazard ratio: A program’s rate of graft failure from day 90 post-transplant to day 365 post-transplant, conditional on the graft surviving for the first 90-days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.

  • Beginning July 14, 2022, the MPSC will consider program performance on the above metrics for transplants that occurred from January 2019 through June 2021 (excluding the March 13– June 12, 2020, SRTR COVID carve-out).

Offer acceptance: A program’s actual rate of accepting organ offers relative to the expected rate, based on similar offers across the nation.

  • If implemented in July 2023, the MPSC will consider program performance on offers received from January 2022 through December 2022. 
  • The MPSC will send notice to transplant programs six months prior to implementation. 

Pre-transplant mortality: A program’s pre-transplant mortality rate relative to the mortality rate experienced by similar patients across the nation.

  • If implemented in July 2024, the MPSC will consider program performance on mortality for candidates appearing on the waiting list at any time during the period of January 2022 through December 2023.  
  • The MPSC will send notice to transplant programs six months prior to implementation.

Prior to implementation, the OPTN will provide educational offerings to help programs understand how various scenarios may impact each metric, and how to find and interpret data available on the SRTR secure site. As materials are developed, members will receive more information in emails and on this website about how to register for events such as webinars and access pre-recorded materials.

The OPTN will support programs in this phased approach. The OPTN will notify programs at least one cycle prior to implementation if, based on current pre-implementation data, they would have been identified if the respective criteria had been in effect. This will allow programs to evaluate their performance and make adjustments ahead of implementation. The OPTN will also collaborate with high-performing programs to identify effective practices and develop resources for members.

Risk-adjustment and performance evaluation

The MPSC strongly believes that it is important to use risk-adjusted metrics to have a fair and accurate performance monitoring system. Risk adjustment allows the MPSC to focus on a program’s performance on those elements of patient care that are within the control of the program that the program can affect through performance improvement initiatives.  In addition, the MPSC believes the use of risk-adjusted metrics is the best way to avoid risk-averse behavior that could result in the unintended negative consequence of greater organ non-use (discard) and a decrease in organ transplants.

Some specific questions regarding the importance of risk adjustment are addressed below. For more information on risk adjustment and the SRTR models developed for the MPSC performance evaluation metrics, a presentation titled “MPSC: Enhance Transplant Program Performance Monitoring System: Proposed Metrics” may be viewed here. This video addresses frequently asked questions. Additional information is being prepared for members.

All four metrics in the new performance monitoring system are risk-adjusted. Risk-adjusted models compare a program’s observed performance to its expected performance. The SRTR attempts to adjust for each program’s respective risk level by taking into account many factors. These include program-specific characteristics of the transplant candidates, recipients and donors. A program’s risk level is reflected in its expected performance number. A program’s performance will be compared to the national performance for patients or offers with similar characteristics. A downloadable PDF provides a quick review about how risk adjustment works for the four metrics used to evaluate performance.

Without risk adjustment, all programs would be expected to achieve the same observed survival rate, regardless of the acuity of their patients or the quality of the donor organs they transplant. The goal of using risk adjustment is to be able to differentiate between program performance, given the known patient and donor factors.

The MPSC wants to understand if a program’s outcomes differ significantly from what would be expected from that program based on the outcomes of similar patients or organ offers nationally; or, in other words, what is the individual program’s effect on the outcome of interest (graft survival, pre-transplant mortality, offer acceptance).

Some transplants involve greater risk than others. The risk adjustment models do not expect higher-risk transplants to succeed at the same rate as lower-risk transplants. If the higher-risk transplants a program performs are associated with a bad outcome, and the model expects those transplants to have a poorer outcome, then the program can still be considered to be performing as expected.

In this example, assume two liver transplant programs perform the same number of transplants and that most of the donor and recipient characteristics are the same between the programs. One program utilizes a high volume of donor after cardiac death (DCD) livers, while the other program does not use any.

  • The model expects that liver transplants using a DCD liver have an approximately 60 percent higher risk of graft failure based on national data.
  • Because of risk-adjustment, the model would expect the program that utilizes DCD livers to have more graft failures (a lower “expected” rate) than the program that uses none.
  • However, a program that rarely performs higher-risk transplants would be expected to have fewer poor outcomes, and would be identified as performing worse than expected if the program had poor outcomes in average-risk transplants.

If the SRTR models did not use risk adjustment, the MPSC would not be well-equipped to understand if a program’s outcomes differ from what would be expected based on patient and donor risk factors that are beyond the program’s control, such as the age of the recipient or whether a donor is a donor after cardiac death or donor after brain death. Instead, the MPSC would only have the raw, absolute number of events at a program. The MPSC would not be able to determine easily if those events were similar to events experienced by other programs, or were associated with more complex transplants.

The SRTR risk adjustment models are limited to the data currently collected by the OPTN. The MPSC acknowledges that current risk adjustment models cannot adjust for all risks; however, the MPSC does not believe that the lack of perfect risk adjustment supports the complete rejection of risk adjustment. If the MPSC and the transplant community acknowledge that transplant programs have different risk profiles – different patient acuity, different case mix, and different donor availability depending on where they are located and the donor supply - the way to reflect that difference is through risk adjustment.

No. Risk-adjusted models cannot be built for some types of transplants, because there are not enough events or available factors that predict the outcome. For example, there were no predictors identified for pediatric kidney deceased donor 1-year graft survival, so a risk-adjusted model was not built.

In cases such as these, the national failure rate is used to determine an expected rate for a program.

The SRTR is continuously improving its risk adjustment models and encourages any feedback from the community and the MPSC. Every six months, the SRTR analyzes national data to determine if the risk associated with a variable has changed and refits the models to capture the current risk profile.

On its website Exit Disclaimer, the SRTR provides information on each of the performance monitoring metric models under the “Tools” tab. This information includes a list of the model elements, the model coefficients, and the element plots that provide a visualization of the relationship between the element and the predicted risk. These tools also include a tab for “Other Elements” that provides a list of elements that the SRTR investigated and found did not add predictive value to the model, or in other words, the element did not improve their ability to predict the outcomes.

In order to include a data element in a risk adjustment model, the variable must be included in OPTN data collection. For any variable with data collected and available, the SRTR can evaluate whether the variable improves the risk adjustment. The SRTR Review Committee determines if a variable should be included in a model.

  1. Any potential adjuster must affect a different proportion of patients at different programs

    For example, if every program in the country used 10 percent donor DCD organs for transplants, DCD would not be a good candidate for adjustment because adjusting for it would not make a difference, since it is the same at all programs. In reality, DCD is a good adjuster because the willingness to use DCD organs varies between programs.

  2. Any potential adjuster must predict, or be associated with, the outcome

    We know from experience that DCD donor organs tend to have poorer outcomes on average than brain dead donor organs. For example, in liver transplants, DCD donors have approximately a 60 percent higher risk of failure nationally than brain dead donors. Yet, the increased risk of using DCD donor organs is acceptable as compared against the alternative outcome of not being transplanted. We want to adjust for this risk.

  3. Any potential adjuster cannot be a treatment by the program that would affect the outcome

    For example, the program’s choice of immunosuppression regimen, which is a treatment decision by the program, may affect the outcome. This is precisely what the MPSC is trying to determine – if there is a treatment practice that is causing or contributing to poorer outcomes. We do not want to adjust that out of the metric, as it may be something the program needs to evaluate and adjust to achieve better outcomes.

MPSC members will evaluate whether factors that are not included in the risk adjustment model appear to be a factor at a program under review. Any factors identified will be referred either to the OPTN organ-specific committee or the OPTN Data Advisory Committee if the data point is not currently collected, or to the SRTR for investigation for inclusion in the model if the data point is currently collected.

Education and resources

The proposal's public comment page has information and educational resources about the metrics, including:

  • A video introduction of the changes outlining what's involved
  • A recording of an Aug. 4, 2021, webinar with Rich Formica, M.D., providing a more detailed analysis
  • A September 2021 recording of MPSC chair, Ian Jamieson, and SRTR director, Jon Snyder, discussing the four evaluation metrics involved, including risk-adjustment and frequently asked questions

Instruction

Use this link to access these professional education resources and learning modules:

  • QLT150: MPSC transplant program performance monitoring
  • QLT151: MPSC post-transplant outcomes and risk adjustment (confers CEPTC credits)
  • QLT153: MPSC pre-transplant mortality rate ratio and risk adjustment (confers CEPTC credits)

The SRTR will also add information to transplant programs’ SRTR secure sites prior to implementation to help you understand whether your program meets the new MPSC performance monitoring criteria. This information is intended to support you as you plan and evaluate your program’s performance.