And now for something completely different. [Tech blog post, by me.]
[Recycled (part of) a mailing list post: another crack at trying to explain what was significant about LispMs.]
One of the much-ignored differences between different computer architectures is the machine language, the Instruction Set Architecture (ISA). It's a key difference. And the reason it doesn't get much attention is that these days, there's really only one type left: the C machine.
There used to be quite a diversity -- there were various widely-divergent CISC architectures, multiple RISC ones, Harvard versus von Neumann designs, stack machines versus register machines, and so on.
Most of that has gone now -- either completely disappeared, or shrunk into very specific niches.
I'm interested in weirdness and oddities, and the Lisp Machines had one of the weirder architectures.
The Lisp language's fundamental bricks are lists, and there are a small core set of tools for manipulating lists: one for extracting the first item, one for separating off the rest of a list, tools for manipulating "atoms" -- list members -- and their contents, and for building new lists out of atoms. Lisp Machine processors didn't execute Lisp code directly -- it was compiled down to a machine language -- but that machine language understood lists, atoms, heads and tails, and it stored lists as lists in special kinds of memory, which had hardware tags containing metadata about the contents.
The C model is that memory is a long flat series of bytes or words. You allocate blocks of that resource for this and that, and software keeps track of what is where and what it is. If you need to manage more memory than you have, you implement this in hardware, which lies to the software about what blocks are where.
The C model is much simpler, which means it's much cheaper to implement. Processors started out as a whole bunch of hand-wired components. Later, some of them were integrated into small chips, but a processor was still a whole cage full of boards. Then those were shrunk down to a bunch of chips, and finally, onto one chip. In parallel and overlapping with this, there were tiny simple processors designed and built as just one chip, while the contemporaneous minicomputers still had whole assemblies of boards. It wasn't a linear development -- there was a lot of overlap, and by the time that single-chip CPUs were getting reasonably capable (e.g. the Motorola 68000 in the late 1970s), the single-chip minicomputer processors looked huge and overcomplicated and expensive.
So rather than minis evolving into micros, the microprocessors started over, from the first very simple designs, which were gradually upgraded. But lessons learned from minis with the basic C-style design predominated in the new microchips, because it scales down a long way: you can implement it on tiny, simple chips, because really, it doesn't call for much complexity. So the first-gen microprocessors were just gradually upgraded with wider and wider word sizes -- 4, 8, 9, 16, 18, 32, 36 -- until they grew up and standardised into really big and complicated 32-bit single-chip processors, as capable as the 1970s minicomputer processors that went before them -- but faster and cheaper, so they outcompeted the mini-derived kit into oblivion.
(And to be fair, they didn't start with C - it's only when the 16-bit micros started to appear that C became a viable development tool. It was too complex for 8-bits, and 8-bit C compilers were very rudimentary -- e.g. the Hisoft C compiler I used on my ZX Spectrum only supported integer arithmetic.)
Meantime, a different branch of the computer evolutionary tree developed fancy single-user workstations, based on the most powerful, productive language: Lisp Machines. Not doing the minicomputer thing -- no rings, no privilege levels, no fancy instructions for maths, but instead, their complexity was in an entirely orthogonal direction -- hardware that was actually fitted to the language, so the processor directly handled the core constructs of the language: managing lists. The chips had facilities for taking the first element of a list, the rest of it, for splitting and concatenating lists, for traversing lists, etc. You didn't deal in bits and bytes, in blocks of RAM, you deal in higher-level structures and the processor implemented them in hardware.
Meantime, the crazy mixed-up proprietary world of proprietary OSes on complicated, incompatible architectures seized on one element of IBM's Stretch mainframe. This became the next big idea. It said, look, you have all these fancy facilities microcoded into your processors, designed to support 1970s languages and OSes that you don't use much any more. (For instance, I have heard it argued that the Motorola 680x0 design can be considered as an implementation of the DEC PDP-11.) Now everything is Unix and C, and it doesn't need all that legacy stuff. So throw it away, get rid of all the microcode, and just implement the bare minimum.
So we got the 2nd generation of microprocessors: dead simple, small, cheap, fast. RISC.
And RISC boxes, with a few exceptions, only ran Unix.
For a while, this was such a compelling solution that it killed off Lisp Machines in academia, their stronghold, because LispMs were expensive and ran an arcane language which is capital-H Hard to learn and to use, and universities are always short of money. (Which is why NeXT failed too.)
Only one proprietary architecture survived this: 80x86, running DOS, mainly due to a critical mass of DOS apps. x86 boxes ran Unix fine, and more cheaply than proprietary Unix boxes, but the business world was wedded to DOS and DOS apps, partly due to Microsoft's very successful building of a practical monopoly on business computing.
IBM tried to address the failings of DOS and the PC design with a new and better OS, OS/2, running on new and better x86 hardware -- but it didn't work, because it didn't run DOS apps well.
Microsoft shrugged, bolted the OS/2 1.2 GUI onto its failed Windows product for DOS, enhanced its ability to multitask DOS apps, and to its surprise created a huge hit.
RISC, oddly, ended up converging with x86. Modern x86 chips are RISC chips, with a translation layer bolted on top. Complicated but it works really well and they still have that critical mass of software. Meantime, playing catch-up, all the workstation RISC chips got bigger and more complex too, until they're just as big and just as complex as x86, but they don't have the volume, so they ended up more expensive, running proprietary Unix.
And here we are today. There are really just 2 families of processor and OS left.
X86, running Windows. (Windows has never been a hit on RISC chips and I am willing to bet that it never will.)
And RISC, running Unix -- which today means ARM running Linux, plus some small proprietary contenders on the sides: AIX on POWER, Solaris on SPARC, both looking moribund.
ARM has survived because it's stayed simple. It doesn't compete directly against x86: it focuses on small, battery-powered, passively-cooled devices, or embedded stuff. ARM chips are considerably slower than x86, and all the proprietary ARM OSes are dead now. Symbian, RISC OS, QNX, vxWorks, iTron -- all dead or nearly, or specialised into niches where they don't compete with the big boys.
On the high end, there's one real player: x86 running Windows... and the parasitic FOSS Unix that burst out of Wintel's chest, Linux. All the other Unixes are basically dead, except tiny FOSS players, which in turn parasitise on Linux. (Oh, all right, they're commensal or symbiotic, but it doesn't sound as good.)
What have they got in common? Really only one thing: the C model.
LispMs weren't out-competed -- they just got caught in the crossfire.
I am not, note, arguing that we should try to reanimate their corpse. The only other interesting development I've seen, though, is the Mill CPU from OOB Computing.
Google it, watch the (long) videos. It's hard to understand but it seems to be trying to find a way to cross the DSP model of processor design with general-purpose CPUs.
They're not that interested in the software side.
I am. I've spent 20y looking at a 30,000' view of comparative OS design.
Where we are now is a position of business pragmatism: taking what works, improving it, and sod the theoretical stuff. We've been doing nothing else since the 1980s. Finally, in the 1990s, the hardware-memory-managed world of Unix workstations converged with the x86 world. The workstations died out, x86 was re-invigorated.
We have Unix and we have Windows, which successfully transitioned from its DOS-based roots (which grew out of the 16-bit PDP-11 via CP/M) and made the leap to NT, which is something more like the hardware-memory-managed VAX and VMS.
Now, like the end of Orwell's Animal Farm, it's hard to see the difference.
“The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.”
Both have complex monolithic kernels, kernel-space subsystems, and userlands on top. Both support multiple architectures. Both have a VM-like model of execution, with processes in their own memory spaces. Both are primarily written in C-family languages. Windows has borrowed design elements from Unix and is still doing so -- for instance the way that the console-mode kernel and its services are being split off from a separate GUI layer, held together via scripting laguages. Meanwhile Unix has thrived by borrowing its UI wholesale from Windows.
(And a bit from the Mac, but then, Windows borrowed direct from the Mac too, so it's interwoven.)
They are built out of compiled, static binaries, running on a very simplistic memory model shaped by the assumptions of C. Any interpreted dynamic stuff is a thin layer on top.
What we have are two different implementations of the same basic concepts, which have fed off one another.
The temporary 1980s explosion in multitasking GUI OSes -- AmigaOS, Atari MINT, RISC OS, BeOS, EPOC, NewtonOS, GEOS, QDOS, Taos/Intent, QNX -- they're basically all gone. Most are dead and forgotten, all that survive at all are obscure and niche.
Lots of stuff has been tried, but really, taking a long view, you can see that they all share the same model and the same assumptions. Stuff that experiments with changing this in profound ways -- microkernels (Chorus, Sprite, Amoeba, Hurd, CoyotoOS), network OSes (Plan 9/Inferno, possibly arguably HeliOS), OSes written in managed code (e.g. Microsoft Singularity) -- has all failed: the established model is too entrenched.
Incremental improvement doesn't change the fundamentals. Stuff that does change the fundamentals breaks compatibility so never catches on.
But it works. It's not elegant, it's not great, but it does the job. It's "good enough". Worse-is-better won.
But now, I really think it's holding us back. Everything that has been tried to move it forward has failed. We're still polishing the same turd.
So maybe, we should re-evaluate the other option, the way that we didn't go.
Trying to fix what we have has failed. It's good enough. It's messy, insanely complicated, more duct tape than original design, but it works. We have tried to re-design it but no change to it gives enough benefit to be worth the cost.
So maybe we need something totally different, from that other, long-abandoned school of design. The MIT/Stanford style, the Lisp style. Metaprogramming, programmable programming languages, code that is a list which manipulates lists, and which is lists all the way down to the metal.
Only because it's the only other model that really competed, that made it right into the GUI and Internet era. It's the other side of the coin from the line that went Fortran -> Basic -> C, compilers and interpreters as 2 separate things.
There were OSes in high-level languages in the minicomputer era (e.g. Burroughs); they died out. Not enough advantages over the C model.
There were OSes and languages that got away from native-code compilation - Taos, Intent, Java, Limbo on Dis on Inferno. They died out, or were absorbed (Java). Not enough advantages.
There are niche OSes that are strong in embedded systems, that eschew the whole Unix/NT multitasking model - vxWorks made it to Mars, even! But they're only strong in their special niches and even there, they're losing ground.
There is only one alternative that has lasted, which has stood up to the assault of the C monoculture. It's in the FOSS world as Emacs and SBCL. It's inside lots of commercial software.
It's the only other option. It nearly made it, but RISC killed it, and x86 killed RISC on the desktop, leaving us back at an enhanced Square 1.
I can't see any other contender.
One of the much-ignored differences between different computer architectures is the machine language, the Instruction Set Architecture (ISA). It's a key difference. And the reason it doesn't get much attention is that these days, there's really only one type left: the C machine.
There used to be quite a diversity -- there were various widely-divergent CISC architectures, multiple RISC ones, Harvard versus von Neumann designs, stack machines versus register machines, and so on.
Most of that has gone now -- either completely disappeared, or shrunk into very specific niches.
I'm interested in weirdness and oddities, and the Lisp Machines had one of the weirder architectures.
The Lisp language's fundamental bricks are lists, and there are a small core set of tools for manipulating lists: one for extracting the first item, one for separating off the rest of a list, tools for manipulating "atoms" -- list members -- and their contents, and for building new lists out of atoms. Lisp Machine processors didn't execute Lisp code directly -- it was compiled down to a machine language -- but that machine language understood lists, atoms, heads and tails, and it stored lists as lists in special kinds of memory, which had hardware tags containing metadata about the contents.
The C model is that memory is a long flat series of bytes or words. You allocate blocks of that resource for this and that, and software keeps track of what is where and what it is. If you need to manage more memory than you have, you implement this in hardware, which lies to the software about what blocks are where.
The C model is much simpler, which means it's much cheaper to implement. Processors started out as a whole bunch of hand-wired components. Later, some of them were integrated into small chips, but a processor was still a whole cage full of boards. Then those were shrunk down to a bunch of chips, and finally, onto one chip. In parallel and overlapping with this, there were tiny simple processors designed and built as just one chip, while the contemporaneous minicomputers still had whole assemblies of boards. It wasn't a linear development -- there was a lot of overlap, and by the time that single-chip CPUs were getting reasonably capable (e.g. the Motorola 68000 in the late 1970s), the single-chip minicomputer processors looked huge and overcomplicated and expensive.
So rather than minis evolving into micros, the microprocessors started over, from the first very simple designs, which were gradually upgraded. But lessons learned from minis with the basic C-style design predominated in the new microchips, because it scales down a long way: you can implement it on tiny, simple chips, because really, it doesn't call for much complexity. So the first-gen microprocessors were just gradually upgraded with wider and wider word sizes -- 4, 8, 9, 16, 18, 32, 36 -- until they grew up and standardised into really big and complicated 32-bit single-chip processors, as capable as the 1970s minicomputer processors that went before them -- but faster and cheaper, so they outcompeted the mini-derived kit into oblivion.
(And to be fair, they didn't start with C - it's only when the 16-bit micros started to appear that C became a viable development tool. It was too complex for 8-bits, and 8-bit C compilers were very rudimentary -- e.g. the Hisoft C compiler I used on my ZX Spectrum only supported integer arithmetic.)
Meantime, a different branch of the computer evolutionary tree developed fancy single-user workstations, based on the most powerful, productive language: Lisp Machines. Not doing the minicomputer thing -- no rings, no privilege levels, no fancy instructions for maths, but instead, their complexity was in an entirely orthogonal direction -- hardware that was actually fitted to the language, so the processor directly handled the core constructs of the language: managing lists. The chips had facilities for taking the first element of a list, the rest of it, for splitting and concatenating lists, for traversing lists, etc. You didn't deal in bits and bytes, in blocks of RAM, you deal in higher-level structures and the processor implemented them in hardware.
Meantime, the crazy mixed-up proprietary world of proprietary OSes on complicated, incompatible architectures seized on one element of IBM's Stretch mainframe. This became the next big idea. It said, look, you have all these fancy facilities microcoded into your processors, designed to support 1970s languages and OSes that you don't use much any more. (For instance, I have heard it argued that the Motorola 680x0 design can be considered as an implementation of the DEC PDP-11.) Now everything is Unix and C, and it doesn't need all that legacy stuff. So throw it away, get rid of all the microcode, and just implement the bare minimum.
So we got the 2nd generation of microprocessors: dead simple, small, cheap, fast. RISC.
And RISC boxes, with a few exceptions, only ran Unix.
For a while, this was such a compelling solution that it killed off Lisp Machines in academia, their stronghold, because LispMs were expensive and ran an arcane language which is capital-H Hard to learn and to use, and universities are always short of money. (Which is why NeXT failed too.)
Only one proprietary architecture survived this: 80x86, running DOS, mainly due to a critical mass of DOS apps. x86 boxes ran Unix fine, and more cheaply than proprietary Unix boxes, but the business world was wedded to DOS and DOS apps, partly due to Microsoft's very successful building of a practical monopoly on business computing.
IBM tried to address the failings of DOS and the PC design with a new and better OS, OS/2, running on new and better x86 hardware -- but it didn't work, because it didn't run DOS apps well.
Microsoft shrugged, bolted the OS/2 1.2 GUI onto its failed Windows product for DOS, enhanced its ability to multitask DOS apps, and to its surprise created a huge hit.
RISC, oddly, ended up converging with x86. Modern x86 chips are RISC chips, with a translation layer bolted on top. Complicated but it works really well and they still have that critical mass of software. Meantime, playing catch-up, all the workstation RISC chips got bigger and more complex too, until they're just as big and just as complex as x86, but they don't have the volume, so they ended up more expensive, running proprietary Unix.
And here we are today. There are really just 2 families of processor and OS left.
X86, running Windows. (Windows has never been a hit on RISC chips and I am willing to bet that it never will.)
And RISC, running Unix -- which today means ARM running Linux, plus some small proprietary contenders on the sides: AIX on POWER, Solaris on SPARC, both looking moribund.
ARM has survived because it's stayed simple. It doesn't compete directly against x86: it focuses on small, battery-powered, passively-cooled devices, or embedded stuff. ARM chips are considerably slower than x86, and all the proprietary ARM OSes are dead now. Symbian, RISC OS, QNX, vxWorks, iTron -- all dead or nearly, or specialised into niches where they don't compete with the big boys.
On the high end, there's one real player: x86 running Windows... and the parasitic FOSS Unix that burst out of Wintel's chest, Linux. All the other Unixes are basically dead, except tiny FOSS players, which in turn parasitise on Linux. (Oh, all right, they're commensal or symbiotic, but it doesn't sound as good.)
What have they got in common? Really only one thing: the C model.
LispMs weren't out-competed -- they just got caught in the crossfire.
I am not, note, arguing that we should try to reanimate their corpse. The only other interesting development I've seen, though, is the Mill CPU from OOB Computing.
Google it, watch the (long) videos. It's hard to understand but it seems to be trying to find a way to cross the DSP model of processor design with general-purpose CPUs.
They're not that interested in the software side.
I am. I've spent 20y looking at a 30,000' view of comparative OS design.
Where we are now is a position of business pragmatism: taking what works, improving it, and sod the theoretical stuff. We've been doing nothing else since the 1980s. Finally, in the 1990s, the hardware-memory-managed world of Unix workstations converged with the x86 world. The workstations died out, x86 was re-invigorated.
We have Unix and we have Windows, which successfully transitioned from its DOS-based roots (which grew out of the 16-bit PDP-11 via CP/M) and made the leap to NT, which is something more like the hardware-memory-managed VAX and VMS.
Now, like the end of Orwell's Animal Farm, it's hard to see the difference.
“The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.”
Both have complex monolithic kernels, kernel-space subsystems, and userlands on top. Both support multiple architectures. Both have a VM-like model of execution, with processes in their own memory spaces. Both are primarily written in C-family languages. Windows has borrowed design elements from Unix and is still doing so -- for instance the way that the console-mode kernel and its services are being split off from a separate GUI layer, held together via scripting laguages. Meanwhile Unix has thrived by borrowing its UI wholesale from Windows.
(And a bit from the Mac, but then, Windows borrowed direct from the Mac too, so it's interwoven.)
They are built out of compiled, static binaries, running on a very simplistic memory model shaped by the assumptions of C. Any interpreted dynamic stuff is a thin layer on top.
What we have are two different implementations of the same basic concepts, which have fed off one another.
The temporary 1980s explosion in multitasking GUI OSes -- AmigaOS, Atari MINT, RISC OS, BeOS, EPOC, NewtonOS, GEOS, QDOS, Taos/Intent, QNX -- they're basically all gone. Most are dead and forgotten, all that survive at all are obscure and niche.
Lots of stuff has been tried, but really, taking a long view, you can see that they all share the same model and the same assumptions. Stuff that experiments with changing this in profound ways -- microkernels (Chorus, Sprite, Amoeba, Hurd, CoyotoOS), network OSes (Plan 9/Inferno, possibly arguably HeliOS), OSes written in managed code (e.g. Microsoft Singularity) -- has all failed: the established model is too entrenched.
Incremental improvement doesn't change the fundamentals. Stuff that does change the fundamentals breaks compatibility so never catches on.
But it works. It's not elegant, it's not great, but it does the job. It's "good enough". Worse-is-better won.
But now, I really think it's holding us back. Everything that has been tried to move it forward has failed. We're still polishing the same turd.
So maybe, we should re-evaluate the other option, the way that we didn't go.
Trying to fix what we have has failed. It's good enough. It's messy, insanely complicated, more duct tape than original design, but it works. We have tried to re-design it but no change to it gives enough benefit to be worth the cost.
So maybe we need something totally different, from that other, long-abandoned school of design. The MIT/Stanford style, the Lisp style. Metaprogramming, programmable programming languages, code that is a list which manipulates lists, and which is lists all the way down to the metal.
Only because it's the only other model that really competed, that made it right into the GUI and Internet era. It's the other side of the coin from the line that went Fortran -> Basic -> C, compilers and interpreters as 2 separate things.
There were OSes in high-level languages in the minicomputer era (e.g. Burroughs); they died out. Not enough advantages over the C model.
There were OSes and languages that got away from native-code compilation - Taos, Intent, Java, Limbo on Dis on Inferno. They died out, or were absorbed (Java). Not enough advantages.
There are niche OSes that are strong in embedded systems, that eschew the whole Unix/NT multitasking model - vxWorks made it to Mars, even! But they're only strong in their special niches and even there, they're losing ground.
There is only one alternative that has lasted, which has stood up to the assault of the C monoculture. It's in the FOSS world as Emacs and SBCL. It's inside lots of commercial software.
It's the only other option. It nearly made it, but RISC killed it, and x86 killed RISC on the desktop, leaving us back at an enhanced Square 1.
I can't see any other contender.