Reader Disagreement Index: a better measure of overall review quality monitoring in an oncology trial compared to adjudication rate

摘要

Purpose: Blinded Independent Central Review (BICR) is highly recommended by regulatory authorities for oncologyregistration trials. “Adjudication rate” provided by “Two Reviewers and Adjudicator Paradigm” of BICR has been part ofreviewer performance metrics and trial efficacy. However, adjudication rate does not consider the adjudicator agreementor disagreement rate of a reviewer. We analyzed that Reader Disagreement Index (RDI) is a better measure thanadjudication rate to monitor reviewer performance in BICR.Methods: BICR adjudication data from 3 different clinical trials including 10 board-certified radiologist reviewers usingResponse Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria was analyzed. RDI for each reviewer was calculatedusing the below mentioned formula. Reviewer adjudication rate and adjudicator agreement rate was calculated for eachreviewer along with RDI. RDI was used to identify the discordant reviewer with highest disagreement rate.RDI (%) = Number of cases where adjudicator disagreed with given reader/Total number of cases read ×100 Results: RDI identified the discordant reviewer in all 3 studies. Discordant reviewers identified using RDI were not thereviewers with highest adjudication or lowest agreement rates. Adjudication rate identified the discordant reviewer in 1of the 3 studies. Reviewer with lowest adjudicator agreement could not have been identified as discordant reviewer usingonly adjudication rate in monitoring reviewer performance. RDI is more robust in identifying a discordant reviewer whoneither has highest adjudication nor lowest agreement rate.Conclusions: RDI is more reliable measure of reviewer performance as compared to adjudication rate and could beeffectively used to monitor reviewer performance as it combines both reviewer adjudication percentage and adjudicationagreement percentage.