Speaker: Daniel Ramos Castro.
Abstract: It is increasingly common in forensic science to report evidential findings in terms of a likelihood ratio (LR). Such analyses are often supported by (semi-)automated LR systems based on statistical methods, which allows for validation and performance measurements. The log-likelihood ratio cost (Cllr) is a popular evaluation metric for such systems, penalizing misleading LRs more strongly when they are further from 1. This systematic review aims to give some intuition for the numbers that can be expected. We studied 136 publications since 2006 that report on a (semi)-automated LR system. We looked at whether the $C_{llr}$ was reported, and what values were found. This may provide some framework of reference for scientists to judge if their system performs well or not. We find that the use of this metric heavily depends on the field, with the metric highly prevalent in fields such as biometrics or microtraces and conspicuously absent in the forensic DNA analysis. Although the number of publications on (semi-)automated likelihood ratio systems has increased over the years, the proportion of these reporting Cllr has remained more constant over time. The results do not show a clear pattern for the Cllr values. These depend heavily on the forensic area, type of analysis and dataset used.
With the increase in LR systems, comparisons between systems become more important. This is hampered by different studies using different datasets. We advocate the increased use of freely available benchmark datasets, common in many disciplines, to bring the field forward.
This presentation is based on a recent collaboration with Stijn van Lierop, Rolf Ypma and Marian Sjerps from the Netherlands Forensic Insititute.