Ů��С��Ƶ

Oi, AI – what do you think you’re looking at? Understanding why machine learning algorithms can be tricked into seeing things that aren’t there is becoming more important with the advent of things like driverless cars. Now we can glimpse inside the mind of a machine thanks to a test that reveals which parts of an image an AI is looking at.

Artificial intelligences don’t make decisions in the same way that humans do. Even the best image recognition algorithms can be  in images that are just white noise, for example.

It’s a big problem, says Chris Grimm at Brown University in Providence, Rhode Island. If we don’t understand why these systems make silly mistakes, we should think twice about trusting them with our lives in things like driverless cars, he says.

So Grimm and his colleagues created a system  when it decides what the image is depicting. Similarly, for a document-sorting algorithm, the system highlights which words the algorithm used to decide which category a particular document should belong to.

Peek inside

It’s really useful to be able to look at an AI and find out how it’s learning, says , a researcher at Google. Grimm’s tool provides a handy way for a human to double-check that an algorithm is coming up with the right answer for the right reasons, he says.

To create his attention-mapping tool, Grimm wrapped a second AI around the one he wanted to test. This “wrapper AI” replaced part of an image with white noise to see if that made a difference to the original software’s decision.

If replacing part of an image changed the decision, then that area of the image was likely to be an important area for decision-making. The same applied to words. If changing a word in a document makes an AI classify a document differently, it suggests that word was key to the AI’s decision.

Grimm tested his technique on an AI trained to sort images into one of 10 categories, including planes, birds, deer and horses. His system mapped where the AI was looking when it made its categorisation. The results suggested that the AI had taught itself to break down objects into different elements and then search for each of those elements in an image to confirm its decision.

Horse’s head

For example, when looking at images of horses, Grimm’s analysis showed that the AI first paid close attention to the legs and then searched the image for where it thought a head might be – anticipating that the horse may be facing in different directions. The AI took a similar approach with images containing deer, but in those cases it specifically searched for antlers. The AI almost completely ignored parts of an image that it decided didn’t contain information that would help with categorisation.

Grimm and his colleagues also analysed an AI trained to play the video game Pong. They found that it ignored almost all of the screen and instead paid close attention to the two narrow columns along which the paddles moved. The AI paid so little attention to some areas that moving the paddle away from its expected location fooled it into thinking it was looking at the ball and not the paddle.

Grimm thinks that his tool could help people work out how AIs make their decisions. For example, it could be used to look at , making sure that they don’t accidentally come up with the right answers by looking at the wrong bit of the image. “You could see if it’s not paying attention to the right things,” he says.

But first Grimm wants to use his tool to help AIs learn. By telling when an AI is not paying attention, it would let AI trainers direct their software towards relevant bits of information.

Reference: