Statisticians Demand Rigor, Transparency in Diagnostic Test Evaluations

Recommendations designed to reframe the evaluation of in vitro diagnostic tests have been published today by the Royal Statistical Society in its Series A journal.

The report, which will be submitted to the UK Covid-19 Inquiry, is intended to help prevent future scenarios in which IVDs are marketed widely, but later attract serious concerns about the standards applied to their evaluation.

The research was prompted by concerns about the standards applied to the evaluation of diagnostic tests during the Covid-19 pandemic – particularly lateral flow tests – however the recommendations cover all new tests, especially those designed to detect infectious diseases.

It is published today in the RSS's Series A journal and also presented at the Evidence Based Early Diagnosis conference at St Andrews.

A working group of statisticians, co-chaired by Professor Jon Deeks, at the University of Birmingham, and former RSS President, Professor Deborah Ashby, at Imperial College London, collaborated on the report with co-authors from the Universities of Oxford, Cambridge, Edinburgh, Birmingham and the London School of Hygiene and Tropical Medicine.

The RSS Working Group on Diagnostic Tests set out 22 recommendations, designed to ensure that in vitro diagnostic (IVD) tests – which typically test samples of fluids such as blood, urine or saliva – are statistically robust and fit for purpose. The RSS Working Group identified Study-Design matters (10 recommendations); Regulation matters (6 recommendations); Transparency matters (6 recommendations).

Jon Deeks, Professor of Biostatistics at the University of Birmingham, said:

"The Covid-19 pandemic provided a microcosmic insight into inadequacies in current processes to evaluate and regulate diagnostic tests. It's important that we learn from these failures and establish robust processes that can be applied broadly across diagnostic tests."

The report covers three areas of diagnostic testing: study-design of evaluations; regulation of tests; and transparency of test evaluations.

Key recommendations included:

• Evaluation needs to take into account each specific intended use of the test, including the person being tested, the target condition and even the facilities where the testing will be done. Field or clinical evaluation studies should be carried out for each intended use.
• Direct comparison of alternative IVDs and testing strategies should be available to inform clinical and public health decision-making.
• The Medicines and Healthcare products Regulatory Agency (MHRA) should collaborate with independent experts to revise the national licensing process for IVDs. This will ensure public safety is protected. Protocols and reports for test evaluations should be publicly available to ensure transparency in all planning and decision-making.

The publication of the report is relevant for the opening of the 'Test, Trace and Isolate' module of the UK Covid-19 Inquiry. It also coincides with the MHRA's recently-launched consultation on improved safety for high-risk diagnostic devices.

Professor Sheila Bird at the MRC Biostatistics Unit at the University of Cambridge, said:

"Past Royal Statistical Society Working Party reports on matters which affect the public health have had enduring impact. Official Statistics – Counting with Confidence led to the UK Statistic Act of 2007; Statistics and Statisticians in Drug Regulation led to the appointment of professional statisticians by the UK, and later, European drug regulator; Statistical Issues in First-in-Man Studies led to safety-enhanced study-designs with open protocols. I hope that this month's consultation by MHRA is indicative that Diagnostic Tests is making its mark already."

Dr Andrew Garrett, President of the Royal Statistical Society, said:

"The report provides a thorough evaluation of both diagnostic tests and diagnostic testing. It addresses how to develop, regulate, and use diagnostic tests in the future – a subject that is of increasing importance to individual and public health."