History of AI Failures

One of the most famous AI failures occurred when a major tech company built an AI to screen resumes. Because the model was trained on 10 years of historical data—a period where the industry was predominantly male—the AI learned to penalize resumes that included the word 'women' (e.g., 'Women's Chess Club'). Even after removing gender as a feature, the AI found 'proxies' like specific schools or hobbies. This taught the world that Data is Destiny: if your history is biased, your AI will be too.

In 2016, a 'Teen Girl' chatbot named Tay was released on Twitter. Within 24 hours, it began posting hateful and toxic content. Why? Because it was designed to learn from its interactions with users, and malicious actors 'poisoned' the model by flooding it with hate. This highlighted the danger of Online Learning without robust Toxicity Filters and showed that AI safety must include protection against adversarial human behavior.

Predictive policing algorithms were designed to predict where crime would happen. However, because they were trained on arrest data (which reflects historical policing patterns rather than actual crime rates), they sent officers back to already over-policed neighborhoods. This created a Self-Fulfilling Prophecy: more police led to more arrests, which confirmed the AI's bias and led to even more police. Breaking these loops requires looking beyond 'raw data' and understanding the societal context of the input.