“Thou shalt not make a machine in the likeness of a human mind.”
—Orange Catholic Bible (from Dune, by Frank Herbert)
“Never trust anything that can think for itself if you can't see where it keeps its brain.”
—Arthur Weasley (from Harry Potter and the Chamber of Secrets, by J K Rowling)
“When you meet anything that's going to be human and isn't yet, or used to be human once and isn't now, or ought to be human and isn't, you keep your eyes on it and feel for your hatchet."
—Mr Beaver (from The Lion, the Witch and the Wardrobe, by C S Lewis)
It’s fascinating the number of times books warn us about things that think, but aren’t human. And with the very rapid progress in AI over the last twelve months, it can feel as if we’re on the brink of discovering why.
As Shadow One-Boxing wrote earlier this week:
“AI is making breakthroughs in maths, including solving a Millennium Prize problem (one of a set of seven especially famous unsolved problems, each with a £1m prize attached, so hardly overlooked by humans), something which less than two years ago the average of AI experts’ predictions thought would be decades away. It can write highly constrained yet coherent formal poetry (and I think “ash settles where the cadence dwells” is pretty evocative), and prose that can win prestigious literary competitions. It can identify sounds from spectrogram images and describe them in qualitative terms.
It cracked an unsolved Enigma message from a handwritten image containing some ambiguous symbols/letters, autonomously researching the historical context and coding up an Enigma simulator. Someone asked it to translate the audio dialogue from the classic game Putt-Putt into French for their kid; it extracted the audio files, transcribed them, translated them, generated audio with similar voice styles to the original actors, and put the new audio back into the game. It can complete Portal and successfully play Minecraft and Factorio…
…Seriously, it’s changing extremely quickly; if you’re thinking “lol, they can’t count the Rs in strawberry” you’re like someone reasoning about the capabilities of modern supercomputers based on the ZX Spectrum you had in the 80s, and you should update.”
And as someone who’s been lurking on the fringes of the rationalist/effective altruist/AI safety crowd for one, arguably two, decades - I first came across Eliezer Yudkowsky in 2007, on a friend’s atheist blog, and have been reading Scott Alexander since 20161 - it has also been fascinating to watch the mainstream commentariat discover these subcultures over the last couple of weeks, to varying degrees of bemusement, confusion and outrage.2
And yes, some of it is a bit bizarre. The Harry Potter fan-fiction; the focus on shrimp welfare; the polyamory; the fact that they sometimes get together to sing secular hymns to global logistics; the obsession with prediction markets; the discussions of simulations and interdimensional acausal trade. And then people like that are plugged into the bosses of some of the richest and most powerful companies in the world?3
But you know something? The people who’ve got a whole bunch of things right about AI and its progress over the last ten years have earned more right to be heard than the people who were calling it a ‘stochastic parrot’ right up until the day it solved Navier-Stokes.
That doesn’t mean they’ll keep getting things right, or even that they necessarily agree with each other. But they’ve done a whole lot more thinking, in more depth, than most others, and some of it is worth reading - not least because this is what many of the people running our frontier AI companies also believe the technology might be capable of.
So how should we think about AI?