Rekognition, a facial surveillance product owned by Amazon and sold to web developers, falsely identified 28 members of Congress as police suspects in a test carried out by the American Civil Liberties Union (ACLU).
Legislators from both parties and across the country were falsely matched, with Congressional members of colour disproportionately identified incorrectly, including six members of the Congressional Black Caucus.
Several months ago, the Congressional Black Caucus sent a letter to Jeff Bezos that expressed concern about the "profound negative unintended consequences" face surveillance technology could have for African Americans, undocumented immigrants and protests.
In response, Amazon said it took issue with the settings used for the facial recognition tool during the tests. The ACLU and other civil rights groups have been pressuring Amazon to stop selling governments access to Rekognition since May, citing the software's use by law enforcement agencies in Oregon and Florida.
According to the ACLU, they used Rekognition's default settings to compare every member of Congress with 25,000 public arrest photos, and verified the results with an independent expert. The test reportedly mirrored how a sheriff's department in Oregon is deploying Rekognition to compare people's faces againsts a mugshot database.
"Our test reinforces that face surveillance is not safe for government use," said Jacob Snow, technology and civil liberties attorney at the ACLU Foundation of Northern California. "Face surveillance will be used to power discriminatory surveillance and policing that targets communities of colour, immigrants and activists. Once unleashed, that damage can't be undone."
Facial recognition is already widely used in China for police purposes, and a number of firms there are innovating in the technology, with several valued in the billion-dollar range. Japan's SoftBank Vision Fund recently invested almost $1bn (£762m) in SenseTime, an AI-powered facial recognition solution. Amazon has been promoting a wide range of uses for Rekognition, from detecting offensive content online to identifying celebrities.
"Congress should press for a federal moratorium on the use of face surveillance until its harms, particularly to vulnerable communities are fully considered," said Neema Singh Guliani, ACLU legislative counsel. "The public deserves a full debate about how and if face surveillance should be used."