Global warming and the Singularity

[pozorvlak]

Last night I made a serious strategic error: I dared to suggest to some Less Wrongers that unFriendly transcendent AI was not the most pressing danger facing Humanity.

In particular, I made the following claims:

1. That runaway anthropogenic climate change, while unlikely to cause Humanity's extinction, was very likely (with a probability of the order of 70%) to cause tens of millions of deaths through war, famine, pestilence, etc. in my expected lifetime (so before about 2060).
2. That with a lower but still worryingly high probability (of the order of 10%) ACC could bring about the end of our current civilisation in the same time frame.
3. That should our current civilisation end, it would be hard-to-impossible to bootstrap a new one from its ashes.
4. That unFriendly AI, by contrast, has a much lower (<1%) chance of occurring before 2060, but that its consequences include Humanity's total extinction.

I'm a pessimist. I make no apology for this fact. But note that I'm actually less pessimistic in this regard than the Singularitarian Nick Bostrom, whose paper on existential risks lists runaway ACC among the "bangs" (total extinction risks) rather than the "crunches" (permanent end of industrial civilisation). Defending my numbers is complicated by the fact that they're all pulled out of thin air extremely ballpark estimates¹, but I'll give it a go.

I was challenged in many regards:

• Steven Kaas pointed me at this paper by Annan and Hargreaves, and claimed that it showed that ACC has "basically no chance" of bringing about the end of civilisation.
• Roko Mijic disputed point 3, saying "what we have actually done is mine a lot of ores out of the ground, refined them into metals and other products, and put them on the surface where they're easy to get. We even created these handy repositories of human-useful material called landfill sites. The easy-to-get fossil fuels - oil and gas - have been depleted, but remember that the industrial revolution ran on coal, and would easily have been run on wood or charcoal. Oil has only actually been critical since the beginning of the century, and even then it is perfectly possible to run cars on sugar-cane ethanol."
• Shane Legg said "I don't worry about AGW because of two reasons. Firstly, the number of people worried about it and who know about it and who do research on this topic is at least 1,000x greater than AI safety, maybe 100,000x. Thus the marginal value of your time working on AGW is likely to be much smaller than in AGI. Secondly, by all accounts that I've heard, AGW becomes a real problem 50+ years from now, whereas by my estimates we will have AGI well before then."

I'll address these points in turn.

Steven: I don't think that paper supports your conclusions. First off, it's an attempt to calculate the "climate sensitivity" S, defined as the temperature increase at equilibrium due to a doubling in greenhouse gas (GHG) concentrations over pre-industrial levels. They conclude that, with an appropriate choice of prior pdf, S can be concluded to be less than 4C with 95% probability² (rather more controversially, they claim that the distribution of S does not have a "fat tail"). This is not an estimate of the total warming that will occur, which depends on several other factors:

• The actual increase in GHG concentrations might be greater than a doubling. Note that it's still accelerating.
• A 4C rise may well (indeed, probably will) take us out of the region in which temperature response is approximately linear: as I mentioned in the Facebook discussion, there are many positive feedback systems involved, all of which are proceeding much faster than predicted even two years ago.
• Their discussion of the expected costs of ACC makes use of a model that assumes gradual increase of costs with increased warming. More modern models are significantly more pessimistic. Indeed, most polynomial models assume a higher exponent than 2.

Roko: I've been predicting for some time that landfill mining will be a major economic activity of the late 21st/early 22nd century. People can certainly scratch out a living like that, as Smokey Mountain and a thousand other Third World waste dumps make clear. But there's reason to doubt that it would be sufficient to fuel a post-Collapse industrial bootstrap. With ores, you see, you can pretty much apply Process X to the stuff you dig up and get Useful Product Y: with landfill, you have a much more complex sorting task. We can assume that a post-Collapse world will be considerably more despotic, amoral and resource-starved than anything we're used to, and that slavery will be widespread, but will that be enough to extract useful raw materials on an industrial scale? I could be wrong, but I don't think so. Note that it's not cost-effective to recycle a lot of waste even with current technology³, and having to start from nothing and build up from scratch is astonishingly limiting. I don't know if you've ever received any wilderness survival training: it's not the same thing, but you might find it eye-opening.

As for fuel: the amount of energy we use per year passed the amount produced by the biosphere some time ago. You can't run an industrial civilisation on wood and charcoal. Biofuels are, basically, a scam. There's no shortage of coal, unfortunately (which among its other sins releases more radioactive isotopes per watt generated than fission power), but modern mines tend to work by mountaintop removal - an extremely brutal, but also rather high-tech, approach. Are there substantial coal reserves that would be accessible without modern tools? Pass.

This is probably the weakest plank of my argument. But to quote Bostrom again, do you really want to do the experiment?

Shane: You have a good point regarding marginal contributions. But you're on much weaker ground regarding time-scales. While things aren't predicted to get really bad until mid-century, they're already disastrous for millions of the world's poor. And, assuming a transcendent AI doesn't appear to save or damn us all, the time when we can make a difference by our actions is now. Well, it would have been better to have acted decisively ten or fifteen or twenty years ago, but there's no sense crying over spilt glaciers. Given that, and I can't emphasise this enough, my disaster scenario is already happening, I think the onus is on you to explain why it's so overwhelmingly probable that we'll be saved at the last minute by a deus ex machina.

¹ Apart from anything, accurate determination of these numbers requires one to accurately predict the behaviour of politicians decades in the future. If you can do that, then you are Hari Seldon and I claim my five pounds.
² I'll just note in passing, that 5% is of the same order as my 10% ballpark estimate.
³ Yes, I've heard of rag-and-bone men, and I know fine well that we now throw away a lot of things that our grandparents would have repaired or sold for scrap. Things were generally a lot easier to mend or reduce to their constituent parts then.