GAO: FBI hasn’t tested accuracy of facial recognition
20 June, 2016
category: Biometrics, Government
It’s no surprise that the FBI is using facial recognition with its Next Generation Identification (NGI) system but a GAO report states that the agency hasn’t tested the accuracy of the biometric modality.
Prior to deploying the system, the FBI conducted limited testing to evaluate whether face recognition searches returned matches to persons in the database within a candidate list of 50, but has not assessed how often errors occur. FBI officials stated that they do not know, and have not tested, the detection rate for candidate list sizes smaller than 50, which users sometimes request from the FBI.
By conducting tests to verify that the facial recognition aspect of NGI is accurate for all allowable candidate list sizes, the FBI would have more reasonable assurance that the system provides leads that help enhance, rather than hinder, criminal investigations. Additionally, the FBI has not taken steps to determine whether the face recognition systems used by external partners, such as states and federal agencies, are sufficiently accurate to support FBI investigations. By taking such steps, the FBI could better ensure the data received from partners is accurate and do not unnecessarily include photos of innocent people as investigative leads.
FBI officials also have not assessed the false positive rate of the face recognition searches. The FBI initially established a 20% false positive rate, however, according to GAO’s review of testing and acquisition documents, the FBI decided that the false positive requirement was not relevant when NGI-IPS was returning a candidate list of 50 potential matches, and therefore did not test the requirement.
FBI officials stated that the system returns a ranked list of candidates that law enforcement officials must analyze to determine if any of candidates have value as an investigative lead. Because the results are not intended to serve as positive identifications, the false positive rate requirement is not relevant, FBI officials stated.
This is contrary to what the National Science and Technology Council and the National Institute of Standards and Technology. Both groups say the detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition system.
Given that the accuracy of a system can have a significant impact on individual privacy and civil liberties as well as law enforcement workload, it is essential that both the detection rate and the false positive rate for all allowable candidate list sizes are assessed prior to the deployment of the system. Conducting such accuracy tests prior to accepting and deploying subsequent changes to the system would help ensure that the system is capable of producing sufficiently accurate search results.
The full GAO report can be downloaded here.
