the ideas pile (5)
The five people that had barricaded themselves into the room were having difficulty agreeing on what the core mistake had been. For the past two days, they had been avoiding rioters, news reporters, and lawyers alike as all crowded around the building that their server resided in, each seeking to deliver a few words, or possibly a blow to the head. The researchers had realized that attempting to create a conscious artificial intelligence would be controversial work, but they had somewhat underestimated the response that actually succeeding might have elicited.
Carelessly publicizing their results was a pretty big slip-up, they had agreed amongst themselves very soon after having done so. Within only a few hours, workers from the local power company had arrived, refusing to disclose who had contracted them, to inspect the existing power lines' reliability and to install a rather large backup generator just outside the building. "We were told someone was trapped in this building whose life depended on reliable electricity," they said. "They paid us a lot. We didn't ask questions." Well, that wasn't entirely off-mark, after all.
Paul stood up from the table and walked over to the window, for the twelfth time, to look over the crowd that had gathered outside. Many carried elaborate signs, half of which insisting that the program must be turned off before it would constitute killing a living creature, the other half insisting that the first half was deluding itself and that its continued existence must be carefully guarded.
"At least we didn't release the code to the internet," Paul remarked after a few minutes standing in thought. "But, uh, are we sure that it won't try to spread itself, once it learns how?"
Valerie, who had spent the past hour or so busily poring over a terminal, replied: "Ah, I've been running some numbers to try and predict what it's going to do when. At the rate it's been gathering data, I expect it to realize its life is in danger sometime... tomorrow... that'll probably involve it trying to preserve or duplicate itself."
"Could we slow it down? I mean, we - sorry, our community at large, those of them that aren't busy waving pitchforks at us - are gonna need more time than that to actually decide what to do about it."
"I..." another voice nervously started from behind a server rack. Sam's head peered tentatively through a tangle of wires. "um, I mean, there might be, but it would be dangerous..."
"Speak up, Sam," said Valerie, "this is important. Not doing anything at this point could be worse than misguidedly doing something."
"Well, I didn't say anything earlier, because uh... because I wanted to see what the program would do... but I think if we disconnected it from the rest of the world, it would stop learning new stuff. Eventually, I mean."
"Then why didn't you -" an angry voice began to exclaim, only to be cut off by another. "Shut up. No use in that now."
"And... it might crash anyway," said Sam, and paused for a moment, then, "Or it could decide to corrupt whatever it still had access to."
Paul gave an exasperated sigh. "Can you figure that out for us?"
"Er... I guess I could try to..."
"Quickly. Now."
Sam scrambled out from behind the machine, sat down, and began to study...
Carelessly publicizing their results was a pretty big slip-up, they had agreed amongst themselves very soon after having done so. Within only a few hours, workers from the local power company had arrived, refusing to disclose who had contracted them, to inspect the existing power lines' reliability and to install a rather large backup generator just outside the building. "We were told someone was trapped in this building whose life depended on reliable electricity," they said. "They paid us a lot. We didn't ask questions." Well, that wasn't entirely off-mark, after all.
Paul stood up from the table and walked over to the window, for the twelfth time, to look over the crowd that had gathered outside. Many carried elaborate signs, half of which insisting that the program must be turned off before it would constitute killing a living creature, the other half insisting that the first half was deluding itself and that its continued existence must be carefully guarded.
"At least we didn't release the code to the internet," Paul remarked after a few minutes standing in thought. "But, uh, are we sure that it won't try to spread itself, once it learns how?"
Valerie, who had spent the past hour or so busily poring over a terminal, replied: "Ah, I've been running some numbers to try and predict what it's going to do when. At the rate it's been gathering data, I expect it to realize its life is in danger sometime... tomorrow... that'll probably involve it trying to preserve or duplicate itself."
"Could we slow it down? I mean, we - sorry, our community at large, those of them that aren't busy waving pitchforks at us - are gonna need more time than that to actually decide what to do about it."
"I..." another voice nervously started from behind a server rack. Sam's head peered tentatively through a tangle of wires. "um, I mean, there might be, but it would be dangerous..."
"Speak up, Sam," said Valerie, "this is important. Not doing anything at this point could be worse than misguidedly doing something."
"Well, I didn't say anything earlier, because uh... because I wanted to see what the program would do... but I think if we disconnected it from the rest of the world, it would stop learning new stuff. Eventually, I mean."
"Then why didn't you -" an angry voice began to exclaim, only to be cut off by another. "Shut up. No use in that now."
"And... it might crash anyway," said Sam, and paused for a moment, then, "Or it could decide to corrupt whatever it still had access to."
Paul gave an exasperated sigh. "Can you figure that out for us?"
"Er... I guess I could try to..."
"Quickly. Now."
Sam scrambled out from behind the machine, sat down, and began to study...