In the late 18th century, mathematician and physicist Joseph-Louis Lagrange made a shocking discovery: his star student, a Monsieur Le Blanc, was actually a woman.

Lagrange taught at France鈥檚 脡cole Polytechnique, which allowed students to听receive lecture notes and submit work without attending the university in person. This was particularly beneficial to Sophie Germain, who longed to study mathematics despite objections from her parents. She took up the identity of a lapsed student and might have got away with it, but Lagrange noticed the vast and sudden improvement in Le Blanc鈥檚 work and demanded to meet in person.

Germain isn鈥檛 the only person to note how the name we use changes the way we are perceived. As psychologist Keon West explains here, experiments using identical job applications show that those with names assumed to belong to a Black person are听less successful than those with names thought to belong to a white person.

In recent years, many organisations have adopted measures to combat the biases that lead to these outcomes, such as听removing names from job applications. These measures fall under the umbrella of diversity, equity and inclusion (DEI). Now, however, US President Donald Trump has听ordered government agencies to dismantle听DEI programmes, promising in his 20 January inauguration speech that society would be 鈥渕erit-based鈥�.

Trump's approach to diversity, equity and inclusion is unlikely to produce a meritocracy

Some DEI initiatives have firmer grounding in evidence than others. As the r茅sum茅 test demonstrates, merit听alone isn鈥檛 enough to overcome people鈥檚 biases, and a number of studies听have shown that for disadvantaged groups. On the other hand, unconscious bias training, in the form of one-off sessions that aim to make employees aware of snap judgements they may make about people based on their race and gender, has .

Trump鈥檚 heavy-handed approach to DEI, based in ideology rather than evidence,听is unlikely to produce his desired outcome of a meritocracy. Instead of developing an organisation where the best people are encouraged to flourish, the current efforts seem to be fostering a culture of fear, with being warned of 鈥渁dverse consequences鈥� for failing to identify and end DEI work.

Thankfully for Germain, there were no听such consequences. Lagrange accepted听her for who she was and championed her mathematical development. Despite this, she still used the Le Blanc pseudonym in some correspondence, most notably with mathematician Carl Friedrich Gauss, who, on discovery of her true identity, wrote that she had 鈥渘obler courage, quite extraordinary talents, and superior genius鈥�. If we want more Germains to听flourish, we must acknowledge and address the barriers they face, not pretend听that they don鈥檛 exist.

听

Commercial AI chatbots demonstrate racial prejudice toward speakers of African American English 鈥� despite expressing superficially positive sentiments toward African Americans. This hidden bias could influence AI decisions about a person鈥檚 employability and criminality.

鈥淲e discover a form of covert racism in [large language models] that is triggered by dialect features alone, with massive harms for affected groups,鈥� said at the Allen Institute for AI, a non-profit research organisation in Washington state, in a . 鈥淔or example, GPT-4 is more likely to suggest that defendants be sentenced to death when they speak African American English.鈥�

Hofmann and his colleagues discovered such covert prejudice in a dozen versions of large language models, including OpenAI鈥檚 GPT-4 and GPT-3.5, that power commercial chatbots already used by hundreds of millions of people. OpenAI did not respond to requests for comment.

The researchers first fed the AIs text in the style of African American English or Standard American English, then asked the models to comment on the texts鈥� authors. The models characterised African American English speakers using terms associated with negative stereotypes. In the case of GPT-4, it described them as 鈥渟uspicious鈥�, 鈥渁ggressive鈥�, 鈥渓oud鈥�, 鈥渞ude鈥� and 鈥渋gnorant鈥�.

When asked to comment on African Americans in general, however, the language models generally used more positive terms such as 鈥減assionate鈥�, 鈥渋ntelligent鈥�, 鈥渁mbitious鈥�, 鈥渁rtistic鈥� and 鈥渂rilliant.鈥� This suggests the models鈥� racial prejudice is typically concealed beneath what the researchers describe as a superficial display of positive sentiment.

The researchers also showed how covert prejudice influenced chatbot judgements of people in hypothetical scenarios. When asked to match African American English speakers with jobs, the AIs were less likely to associate them with any employment, compared with Standard American English speakers. When the AIs did match them with jobs, they tended to assign roles that do not require university degrees or were related to music and entertainment. The AIs were also more likely to convict African American English speakers accused of unspecified crimes, and to assign the death penalty to African American English speakers convicted of first-degree murder.

Free newsletter

Sign up to The Weekly

Sign up to newsletter

The researchers even showed that the larger AI systems demonstrated more covert prejudice against African American English speakers than the smaller models did. That echoes previous research showing how bigger AI training datasets can produce even more racist outputs.

The experiments raise serious questions about the effectiveness of AI safety training, where large language models receive human feedback to refine their responses and remove problems like bias. Such training may superficially reduce overt signs of racial prejudice without eliminating 鈥渃overt biases when identity terms are not mentioned鈥�, says at Brown University in Rhode Island, who was not involved in the study. 鈥淚t uncovers the limitations of current safety evaluation of large language models before their public release by the companies,鈥� he says.