The Anti-Defamation League (ADL) has just released a report that’s causing quite a stir in the world of artificial intelligence. It highlights some worrying biases against Jewish people and Israel in several major AI language models. These models, like OpenAI’s GPT-4o, Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and Meta’s Llama 3-8B, are widely used, so the findings are definitely something to pay attention to.
In their study, the ADL team got creative. They tweaked prompts with recognizable names while others were left anonymous to see how these models would respond. What they found was pretty telling: the responses varied significantly based on the identity cues in the prompts. Out of 8,600 evaluations and 34,000 responses, they looked into 86 different statements. These covered everything from biases against Jews and Israel to conspiracy theories about the Holocaust and other non-Jewish topics.
Interestingly, Meta’s Llama was found to be the worst offender, giving inaccurate responses related to Jewish people and Israel. Jonathan Greenblatt, the ADL’s CEO, stressed the need for AI developers to step up their game in combating these biases. He pointed out that AI is changing how we consume information, but it’s not free from the biases that exist in society.
The report also highlighted biases in responses about the Israel-Hamas conflict from GPT and Claude, and noted a general hesitance among these models to tackle questions about Israel. Perhaps most concerning was the models’ struggle to dismiss antisemitic conspiracy theories, with a stronger bias observed against Israel than against Jews, except in the case of GPT.
Both Meta and Google have pushed back against these findings. They argue that the study didn’t use their latest AI versions and that the testing methods didn’t reflect how people typically use AI. Meta’s spokesperson mentioned that users usually ask open-ended questions, not multiple-choice ones. Meanwhile, Google’s representative pointed out that the Gemini model wasn’t even meant for consumer use, suggesting the questioning format skewed the results.
Daniel Kelley, the Interim Head of the ADL Center for Technology and Society, emphasized the importance of improving AI training data and content moderation. He noted how integrated AI tools have become in our educational and professional lives. The ADL has put forward several recommendations for AI developers and government bodies. These include working with academic and governmental institutions on pre-deployment testing, following NIST’s risk management framework, and addressing potential biases in training data.
Moreover, the ADL is urging the government to step in and create a regulatory framework for AI developers, along with investing in AI safety research. It’s clear that as AI becomes more entrenched in our daily lives, tackling these biases is crucial.
