Companies that are planning to use artificial intelligence for recruitment should think twice before doing that. A new report revealed that Amazon’s AI machine learned gender bias and weeded out women as potential job candidates. The machine even downgraded applicants based on the school they attended.

A growing number of employers are using AI to boost the efficiency of their hiring process. The machine can be utilized to evaluate resumes, narrow down a list of applicants, and recommend candidates for the right post within a company. It can then pass on its findings to its live counterpart for human assessment. While AI is an effective tool for screening resumes, it has been shown to develop bias, as proven by Amazon’s experiment.

Reuters reported that the retail giant spent several years developing an AI that would vet job applicants. The machine was trained to look at the resumes that the company received for the past ten years. But as most of these applications were from male applicants, the patterns the AI identified were strongly oriented to that sex. In short, Amazon’s AI learned gender bias.

For instance, the AI developed a preference for terms like “captured” or “executed,” which were words commonly used by male engineers. The machine also began to penalize applications that included the word “women” or “women’s.” So describing yourself as the head of the “women’s physics club” was a strike against you.

Amazon had to ditch its AI recruitment system because it was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. https://t.co/JWhbBNOl0V pic.twitter.com/sy512kwcNt

— Reuters Tech News (@ReutersTech) October 10, 2018

A source familiar with Amazon’s AI program also admitted that the machine even downgraded applicants who graduated from two all-women’s universities. The names of the universities were not specified in the report.

The bias shown by the AI’s algorithm became noticeable a year after the project started, and Amazon admittedly tried to correct its AI. The company’s engineers initially edited the system to make it neutral to these specific words. However, there was no way of proving that the machine would not learn another way to sort candidates in a discriminatory manner.

The project was eventually shelved in 2017 because company executives lost confidence in it. The AI also reportedly failed at providing choices for strong and effective job candidates.

Fortunately for Amazon, the AI hiring experiment was just a trial run. The machine was never utilized by a larger group and was never used as the main recruiting agent. Nevertheless, the possibility is high that a qualified applicant was weeded out simply because she was a woman and did not think to use a masculine term like “capture.”

[Featured image via Pexels]