Ů��С��Ƶ

Black people in the US may be missing out on healthcare because a widely used algorithm is racially biased. The proportion of black people referred for extra care would more than double if the bias were removed, according to new research.

Algorithms are fast becoming a key part of healthcare. Such technologies are used to screen somewhere between 100 and 200 million people in the US, says at the University of California, Berkeley.

One example is an algorithm that is used to predict the future health of individuals based on their past health records. Once the algorithm is fed data about a person’s diagnoses, prescriptions and procedures, it spits out a number that predicts the cost of the person’s future healthcare.

Hospitals, healthcare systems and some health insurance providers use this score to identify people who are likely to need more care in the future. Those who have the highest predicted costs can be referred for extra medical care to help prevent them getting sicker, says Obermeyer.

Obermeyer and his colleagues wanted to find out more about how the algorithm worked. “Along the way, we noticed that there was this fairly stark difference in the risk scores that black and white patients had at the same level of health,” he says.

Ahead in line

The team found that black people assigned the same score as white people went on to have worse health outcomes. “You can think of it as healthier white patients being put ahead in line of sicker black patients when it comes to allocating enrolment into this programme,” says Obermeyer.

When the team ran a simulation that eliminated the bias, the proportion of people referred for extra treatment who were black increased from 17.7 per cent to 46.5 per cent. It is difficult to estimate how many black people are missing out on healthcare as a result, says Obermeyer, because he doesn’t know how many health organisations use the algorithm in this way.

This doesn’t mean that the algorithm isn’t working, says Obermeyer. “The data are just a reflection of the society that produced the data,” he says. “Black patients will generate lower costs than white patients, and that’s because of a variety of socioeconomic factors related to access to healthcare, as well as direct effects of race on the doctor-patient relationship.”

“The design of different systems, whether we’re talking about legal systems or computer systems, can create and reinforce hierarchies precisely because the people who create them are not thinking about how social norms and structures shape their work,” says at Princeton University, and author of Race After Technology. “Indifference to social reality is, perhaps, more dangerous than outright bigotry.”

Significant and prevalent

Bias in algorithms is likely to be “significant and prevalent”, says at Stanford University in California. “It’s hard to put a number on it, but this could affect a large fraction of the [US] population,” he says.

Obermeyer and his colleagues have tweaked the algorithm to predict other variables that are less racially biased, and have been working with the manufacturer of the algorithm to improve it.

The researchers have chosen not to name the manufacturer because they don’t want to single out one company. Many other companies and academics have developed similar algorithms, and none of them have realised they may be biased, says Obermeyer.

The hospitals, healthsystems and government agencies using the algorithms haven’t either. “It’s tempting to think they should have known better, but nobody knew better,” says Obermeyer. Regulations that require manufacturers to audit their algorithms might help in future, but users must also take responsibility for making sure the algorithms they use aren’t biased, he says.

In the meantime, there are some signs of progress, says , ethics fellow at the Alan Turing Institute in London. The organisation has helped create guidance on the responsible and ethical design and implementation of AI systems, .