Amazon scraps AI recruiting tool after it showed bias against women

  • Wednesday, October 10th, 2018
  • Author: Tim Maytom
  • Share this article:

Amazon has reportedly scrapped an in-house recruitment tool powered by AI after it was found to show bias against women. The tool, which used machine learning to recommend candidates, apparently learned its biases from studying resumes submitted to Amazon from the past 10 years.

According to Reuters, which cites five people familiar with Amazons machine learning efforts, the company has been working on computer programs designed to filter job applications since 2014, but within less than two years found that the programs were demonstrating significant bias when it came to software developer jobs and other technical roles.

“Everyone wanted this holy grail,” said one of the insiders quoted in the Reuters story. “They literally wanted it to be an engine where Im going to give you 100 resumes, it will spit out the top five, and well hire those.”

The experimental hiring tool used artificial intelligence trained by observing patterns in resumes submitted to the company over a 10-year period. It then gave candidates a score ranging from one to five stars. However, because most of the resumes it studied were from men, reflecting the male-dominated tech industry, the system taught itself to actively discriminate against women, penalising resumes that included the word “womens” (as in “womens football club captain”) and downgrading graduates from two all-womens colleges.

While the machine learning team at Amazon was able to edit the program to make it neutral to these particular terms, they were not able to guarantee that similar biases against women or other groups would not still creep in. The team behind the project was ultimately disbanded last year when executives lost hope in the idea. While the program was operational, recruiters looked at the recommendations generated by the tool, but never relied soley on those rankings.

According to a 2017 survey by talent software firm CareerBuilder, 55 per cent of human resources managers in the US expect AI to be a regular part of their work within the next five years. However, Amazons inability to stop existing prejudices and preconceptions from colouring its AIs judgement should serve as a warning to those looking to factor machine learning and other automated tools into the process of recruitment.