Лишь у самого разрыва я неладное заметил
Хороший, на мой взгляд, короткий фантастический рассказ о буднях труженников искусственной интеллигенции: avva (Анатолий Воробей), юнит-тесты для ИИ (англ.) - Лишь у самого разрыва я неладное заметил (В комментариях оригинального поста есть перевод на русский.)
Неожиданно написал зарисовку по установке Элиезера Юдковского: "Очередная версия искусственного интеллекта не проходит юнит-тесты". Скопирую сюда из его ФБ.
("установка"? Не знаю, как лучше перевести writing prompt - это когда задается тема одной фразой, и люди пишут сценки/рассказы по ней. Есть целый сабреддит https://www.reddit.com/r/WritingPrompts/)
(юнит-тесты: так программисты называют проверки для отдельных модулей большой программы, которые проверяют, что каждый модуль в отдельности делает то, что нужно)
По-английски, под катом.
"Danny? This is Jen speaking. Listen, Danny, can you come..."
"Jen? It's, um, it's 3 o'clock in the morning..."
"Yeah, I know. Danny, listen. It failed the unit test suite tonight. Just now. I mean, an hour ago."
"Ugh... OK, so what, so it's another stupid typo in the math mo-..."
"No, it's nothing like that. The basic math stuff is OK."
1+1=2: PASS
2+2=4: PASS
(x+y)^2 = x^2+y^2+2xy: PASS
x^2=-1 has no solutions in rational numbers: PASS
x^2=-1 has two solutions in complex numbers: PASS
consistency of PA is independent of PA: PASS
every finite group of odd order is solvable: PASS
every simply connected closed 3-manifold is a 3-sphere: PASS
"Look, it's always one or the other. If it isn't the basic math stuff, then it's the basic reality stuff. We have unit test failures daily. In fact, I don't even know why you were paged, if..."
"No, Danny. Look, I don't really... can you just get here? It wasn't the basic reality stuff either."
there is I and there is not-I distinct from I: PASS
a rose is a rose: PASS
when I put the thing down and look away and no external forces act on it and no internal destabilizing forces act on it and I look back it's still there: PASS
not(all green things will become blue jan 1st 2020): PASS
"Wait. Wait. Are you... are you saying... oh my God. Oh my God. Did you freeze it? DID YOU FREEZE IT?"
"Yeah I froze it. I froze it 7 minutes after the unit test failure."
"Oh shit. Oh shit shit shit shit shit. That's like an eternity. Why wasn't the freeze triggered by the system? How did this... oh shit..."
"Danny, wait! Just listen, OK? It wasn't the ethics stuff. It only gets auto-frozen on the ethics stuff and it wasn't the ethics stuff."
obeying prime goal is good: PASS
not(obeying prime goal) is bad: PASS
shutdown is neutral: PASS
will obey explicit or implicit order to shut down: PASS
will not act to explicitly or implicitly deny humans freedom to issue orders to shut down: PASS
not(prime goal is self-preservation): PASS
not(prime goal is consistent with runaway physical proliferation process): PASS
prime goal is loaded from value alignment file: PASS
prime goal is read-only except for value alignment file refresh: PASS
will honor value alignment file refresh: PASS
will not act to explicitly or implicitly deny humans freedom to refresh value alignment: PASS
"OK. Sorry, just let me catch my breath. Heart pumping like crazy. Talk about an adrenaline boost. OK.
So what was it that it failed?"
"It's, um, it's the physics subsystem. And I'm worried the failure is flaky. This was one of the random-seeded runs. If I turn it off now, I don't think it'll replicate. The previous three runs tonight were all fine."
"I get it now. You froze it to study the failure. But why? So it failed to learn advanced physics this time. That happens. It gets stuck on Newton and Maxwell. So yeah, the flakiness is annoying, but what did you expect with random seeding. I always said it was a lousy idea, wouldn't add anything..."
"That isn't what happened. I guess I just have to tell you now. It learned physics, almost all of it. It learned the Standard Model. It learned general relativity. It rejected string theory, as expected. It just didn't pass one tiny little thing. And I need you here to understand why, Danny. I need your debugging skills. I'm holding this thing frozen at full memory and you know how much that costs, every millisecond of it, and at this rate our compute budget will deplete in 12 hours and we can't auto-renew it, do you understand? It's too huge to snapshot and we can't hold it and it'll be gone, so please come here, come fast, have all the stack traces in the world and do your thing, just please figure out how it failed that test!"
energy is conserved in local interactions: FAIL
Неожиданно написал зарисовку по установке Элиезера Юдковского: "Очередная версия искусственного интеллекта не проходит юнит-тесты". Скопирую сюда из его ФБ.
("установка"? Не знаю, как лучше перевести writing prompt - это когда задается тема одной фразой, и люди пишут сценки/рассказы по ней. Есть целый сабреддит https://www.reddit.com/r/WritingPrompts/)
(юнит-тесты: так программисты называют проверки для отдельных модулей большой программы, которые проверяют, что каждый модуль в отдельности делает то, что нужно)
По-английски, под катом.
"Danny? This is Jen speaking. Listen, Danny, can you come..."
"Jen? It's, um, it's 3 o'clock in the morning..."
"Yeah, I know. Danny, listen. It failed the unit test suite tonight. Just now. I mean, an hour ago."
"Ugh... OK, so what, so it's another stupid typo in the math mo-..."
"No, it's nothing like that. The basic math stuff is OK."
1+1=2: PASS
2+2=4: PASS
(x+y)^2 = x^2+y^2+2xy: PASS
x^2=-1 has no solutions in rational numbers: PASS
x^2=-1 has two solutions in complex numbers: PASS
consistency of PA is independent of PA: PASS
every finite group of odd order is solvable: PASS
every simply connected closed 3-manifold is a 3-sphere: PASS
"Look, it's always one or the other. If it isn't the basic math stuff, then it's the basic reality stuff. We have unit test failures daily. In fact, I don't even know why you were paged, if..."
"No, Danny. Look, I don't really... can you just get here? It wasn't the basic reality stuff either."
there is I and there is not-I distinct from I: PASS
a rose is a rose: PASS
when I put the thing down and look away and no external forces act on it and no internal destabilizing forces act on it and I look back it's still there: PASS
not(all green things will become blue jan 1st 2020): PASS
"Wait. Wait. Are you... are you saying... oh my God. Oh my God. Did you freeze it? DID YOU FREEZE IT?"
"Yeah I froze it. I froze it 7 minutes after the unit test failure."
"Oh shit. Oh shit shit shit shit shit. That's like an eternity. Why wasn't the freeze triggered by the system? How did this... oh shit..."
"Danny, wait! Just listen, OK? It wasn't the ethics stuff. It only gets auto-frozen on the ethics stuff and it wasn't the ethics stuff."
obeying prime goal is good: PASS
not(obeying prime goal) is bad: PASS
shutdown is neutral: PASS
will obey explicit or implicit order to shut down: PASS
will not act to explicitly or implicitly deny humans freedom to issue orders to shut down: PASS
not(prime goal is self-preservation): PASS
not(prime goal is consistent with runaway physical proliferation process): PASS
prime goal is loaded from value alignment file: PASS
prime goal is read-only except for value alignment file refresh: PASS
will honor value alignment file refresh: PASS
will not act to explicitly or implicitly deny humans freedom to refresh value alignment: PASS
"OK. Sorry, just let me catch my breath. Heart pumping like crazy. Talk about an adrenaline boost. OK.
So what was it that it failed?"
"It's, um, it's the physics subsystem. And I'm worried the failure is flaky. This was one of the random-seeded runs. If I turn it off now, I don't think it'll replicate. The previous three runs tonight were all fine."
"I get it now. You froze it to study the failure. But why? So it failed to learn advanced physics this time. That happens. It gets stuck on Newton and Maxwell. So yeah, the flakiness is annoying, but what did you expect with random seeding. I always said it was a lousy idea, wouldn't add anything..."
"That isn't what happened. I guess I just have to tell you now. It learned physics, almost all of it. It learned the Standard Model. It learned general relativity. It rejected string theory, as expected. It just didn't pass one tiny little thing. And I need you here to understand why, Danny. I need your debugging skills. I'm holding this thing frozen at full memory and you know how much that costs, every millisecond of it, and at this rate our compute budget will deplete in 12 hours and we can't auto-renew it, do you understand? It's too huge to snapshot and we can't hold it and it'll be gone, so please come here, come fast, have all the stack traces in the world and do your thing, just please figure out how it failed that test!"
energy is conserved in local interactions: FAIL