Ames looked at the paper. “A rifle? I don’t think so. It’s just a bunch of lines.”
“It is a rifle,” said Bucky.
“No, it’s not,” said Ames.
“I see a rifle.”
“What about the picture before that, the dog? Did you actually see that?”
“The picture before this was a house.”
“You’re right. Of course, you’re right.” Ames put the folder of images back in the satchel. Then he stood up and said, “I’m switching you off now. Thank you for your help.”
“You are welcome,” said Bucky.
Ames walked up to it, twisted the key, and pulled it out. He looked up at the bot a moment longer, then walked up to the camera and turned it off. The video cut to him sitting on the couch in his living room. His eyes were wide open, and he looked a little crazed. He held up the image with the pattern of curved lines.
“This is what we call a fooling image. The D-17s have standard AI image-detection capabilities, and—like other AI systems—they aresusceptible to misidentifying images such as this based upon their pattern-recognition algorithms. To a machine, this nonsense looks like a rifle. Bucky should have identified it as such. Instead, it saw what the image really is. The thing is, Bucky has seen this image before. He’s said it’s a rifle. He’s been corrected. They all have. It never matters. It’s always a rifle, because verbal inputs from humans don’t change the algorithm.Except, tonight was different. It didn’t see a rifle. Not only that, but the bot corrected itself once it realized what was expected of it. It adjusted. It’slying to me. Further, I purposely misidentified the prior image as being of a dog, and Bucky corrected me to tell me it was a house. It remembered. Their processors are not supposed to work this way. That is remarkable and, frankly, it is concerning. I do not know what it means. But it requires further investigation. I have downloaded Number 20’s source code via the console in the Vault and I will be scouring it.”
Ames reached forward and switched the camera off.
The video cut to Ames again, but he appeared to be outside in the dark somewhere. He was lit by a dim light source somewhere to his front and left, casting his face half in shadow and reflecting pinpoints of light in his eyes. He spoke quickly. “The algorithm that governs the behavior of the D-17s is the product of a decade of work across multiple agencies. And yet, someone highly skilled, at some point, somehow, inserted additional code that was encrypted, and almost undetectable. After considerable effort, I was able to decrypt it. What I found was an elegantly simple neural network, called Praetorian, and it…” He trailed off. “Well, I’d rather not make a definitive statement. Better to ask the thing itself.” He looked around, as if worried someone might be listening. Then he said, “If I’m seeing what I think I’m seeing… everyone at this facility is in danger.” He turned away from the camera, growing emotional. Then he turned back in, leaned closer, and said in a harsh whisper, “If I’m right, I’m going to burn the whole fucking thing down.”
He turned off the camera.
CHAPTER 42
BRODIE AND TAYLOR SAT SILENTLY, engrossed in Roger Ames’s video. He was back down in the Vault announcing the date—April 24—and the time—2:36A.M.—then activating Bucky and greeting it. He repeated the initial series of questions and received the same answers.
Then he said, “Number 20, in order to test your capacity to store and recall information, you have been loaded with a large database of facts about American history, is that correct?”
“Yes.”
“Who was James Madison?”
“James Madison was one of America’s Founding Fathers, and the fourth president of the United States.”
“Who won the Battle of Antietam?”
“The Union won the Battle of Antietam.”
“What is the Triangle Shirtwaist Factory?”
“The Triangle Shirtwaist Factory was a sweatshop in the New York City neighborhood of Greenwich Village. It is notable for the fire of 1911, in which one hundred and forty-six garment workers died, making it the deadliest industrial disaster in the history of the city.”
“Who were the Black Panthers?”
Bucky paused. “I can describe the organization or the superhero.”
Ames gave him an odd look. “The organization.”
“The Black Panther Party was a leftist Black power organization active in the 1960s and 1970s.”
“Since when do you know about superheroes, Number 20?”
“It is in my database.”
“No, it’s not. We did not give you data about popular culture.”
