The death of the filesystem [tech blog post, by me - please comment on LJ, not on Twitter/FB]
I've written a few times about a coming transition in computing -- the disappearance of filesystems, and what effects this will have. I have not had any constructive dialogue with anyone.
So I am trying yet again, by attempting to to rephrase this in a historical context:
There have been a number of fundamental transitions in computing over the years.
1st generation
The very early machines didn't have fixed nonvolatile storage: they had short-term temporary storage, such as mercury delay lines or storage CRTs, and read data from offline, non-direct-access, often non-rewritable means, such as punched cards or paper tape.
2nd generation
Hard disks came along, in about 1953, commercially available in 1957: the IBM RAMAC...
https://en.wikipedia.org/wiki/IBM_305_RAMAC
Now, there were 2 distinct types of directly-accessible storage: electronic (including core store for the sake of argument) and magnetic.
A relatively small amount of volatile storage, in which the processor can directly work on data, and a large amount of read-write volatile storage but which must be transferred into volatile storage for processing. You can't add 2 values in 2 disk blocks without transferring them into memory first.
This is one of the fundamental basic models of computer architecture. However, it has been _the_ single standard architecture for many decades. We've forgotten there was ever anything else.
[[
Aside:
There was a reversion to machines with no directly-accessible storage in the late 1970s and early-to-mid 1980s, in the form of 8-bit micros with only cassette storage.
The storage was not under computer control, and was unidirectional: you could load a block, or change the tape and save a block, but in normal use for most people except the rather wealthy, the computer operated solely on the contents of its RAM and ROM.
Note: no filesystems.
(
Trying to forestall an obvious objection:
Later machines, such as the ZX Spectrum 128 and Amstrad PCW, had RAMdisks, and therefore very primitive filesystems, but that was mainly a temporary stage due to processors that couldn't access >64kB of RAM and the inability to modify their ROMs to support widespread bank-switching, because it would have broken backwards-compatibility.)
)
]]
Once all machines have this 2-level-store model, note that the 2 stores are managed differently.
Volatile store is not structured as a filesystem as it is dynamically constructed on the fly every boot. It has little to no metadata.
Permanent store needs to have metadata as well as data. The computer is regularly rebooted, and then, it needs to be able to find its way through the non-volatile storage. Thus, increasingly elaborate systems of indexing.
But the important thing is that filesystems were a solution to a technology issue: managing all that non-volatile storage.
Over the decades it has been overloaded with other functionality: data-sharing between apps, security between users, things like that. It's important to remember that these are secondary functions.
It is near-universal, but that is an artefact of technological limitations. That the fast, processor-local storage was volatile, and non-volatile storage was slow and large enough that it had to be non-local. Nonvolatile storage is managed via APIs and discrete hardware controllers, whose main job was transferring blocks of data from volatile to non-volatile storage and back again.
And that distinction is going away.
The technology is rapidly evolving to the point where we have fast, processor-local storage, in memory slots, appearing directly in the CPUs' memory map, which is non-volatile.
Example -- Flash memory DIMMs:
https://www.theregister.co.uk/2015/11/10/micron_brings_out_a_flash_dimm/
Now, the non-volatile electronic storage is increasing rapidly in speed and decreasing in price.
Example -- Intel XPoint:
https://arstechnica.com/information-technology/2017/02/specs-for-first-intel-3d-xpoint-ssd-so-so-transfer-speed-awesome-random-io/
Note the specs:
Reads as fast as Flash.
Writes nearly the same speed as reads.
Half the latency of Flash.
100x the write lifetime of Flash.
And this is the very first shipping product.
Intel is promising "1,000 times faster than NAND flash, 10 times denser than (volatile) DRAM, and with 1,000 times the endurance of NAND".
This is a game-changer.
What we are looking at is a new type of computer.
3rd generation
No distinction between volatile and non-volatile storage. All storage appears directly in the CPUs' memory map. There are no discrete "drives" of any kind as standard. Why would you? You can have 500GB or 1TB of RAM, but if you turn the machine off, then a day later turn it back on, it carries on exactly where it was.
(Yes there will be some caching and there will need to be a bit of cleverness involving flushing them, or ACID writes, or something.)
It ships to the user with an OS in that memory.
You turn it on. It doesn't boot.
What is booting? Transferring OS code from non-volatile storage into volatile storage so it can be run. There's no need. It's in the processor's memory the moment it's turned on.
It doesn't boot. It never boots. It never shuts down, either. You may have tell it you're turning it off, but it flushes its caches and it's done. Power off.
No hibernation: it doesn't need to. The OS and all state data will be there when you come back. No sleep states: just power off.
What is installing an OS or an app? That means transferring from slow non-volatile storage to fast non-volatile storage. There is no slow or fast non-volatile storage. There's just storage. All of it that the programmer can see is non-volatile.
This is profoundly different to everything since 1957 or so.
It's also profoundly different from those 1980s 8-bits with their BASIC or Forth or monitor in ROM, because it's all writable.
That is the big change.
In current machines, nobody structures RAM as a filesystem. (I welcome corrections on this point!) Filesystems are for drives. Doesn't matter what kind of drive: hard disk, SSD, SD card, CD, DVD, DVD-RW, whatever. Different filesystems, but all need transfers from them to and from volatile storage to function.
That's going away. The writing is on the wall. The early tech is shipping right now.
What I am asking is, how will it change OS design?
All I am getting back is, "don't be stupid, it won't, OS design is great already, why would we change?"
This is the same attitude the DOS and CP/M app vendors had to Windows.
WordStar and dBase survived the transition from CP/M to MS-DOS.
They didn't survive the far bigger one from MS-DOS to Windows.
WordPerfect Corp and Lotus Corp tried. They still failed and died.
A bigger transition is about to happen. Let's talk about it instead of denying it's coming.
So I am trying yet again, by attempting to to rephrase this in a historical context:
There have been a number of fundamental transitions in computing over the years.
1st generation
The very early machines didn't have fixed nonvolatile storage: they had short-term temporary storage, such as mercury delay lines or storage CRTs, and read data from offline, non-direct-access, often non-rewritable means, such as punched cards or paper tape.
2nd generation
Hard disks came along, in about 1953, commercially available in 1957: the IBM RAMAC...
https://en.wikipedia.org/wiki/IBM_305_RAMAC
Now, there were 2 distinct types of directly-accessible storage: electronic (including core store for the sake of argument) and magnetic.
A relatively small amount of volatile storage, in which the processor can directly work on data, and a large amount of read-write volatile storage but which must be transferred into volatile storage for processing. You can't add 2 values in 2 disk blocks without transferring them into memory first.
This is one of the fundamental basic models of computer architecture. However, it has been _the_ single standard architecture for many decades. We've forgotten there was ever anything else.
[[
Aside:
There was a reversion to machines with no directly-accessible storage in the late 1970s and early-to-mid 1980s, in the form of 8-bit micros with only cassette storage.
The storage was not under computer control, and was unidirectional: you could load a block, or change the tape and save a block, but in normal use for most people except the rather wealthy, the computer operated solely on the contents of its RAM and ROM.
Note: no filesystems.
(
Trying to forestall an obvious objection:
Later machines, such as the ZX Spectrum 128 and Amstrad PCW, had RAMdisks, and therefore very primitive filesystems, but that was mainly a temporary stage due to processors that couldn't access >64kB of RAM and the inability to modify their ROMs to support widespread bank-switching, because it would have broken backwards-compatibility.)
)
]]
Once all machines have this 2-level-store model, note that the 2 stores are managed differently.
Volatile store is not structured as a filesystem as it is dynamically constructed on the fly every boot. It has little to no metadata.
Permanent store needs to have metadata as well as data. The computer is regularly rebooted, and then, it needs to be able to find its way through the non-volatile storage. Thus, increasingly elaborate systems of indexing.
But the important thing is that filesystems were a solution to a technology issue: managing all that non-volatile storage.
Over the decades it has been overloaded with other functionality: data-sharing between apps, security between users, things like that. It's important to remember that these are secondary functions.
It is near-universal, but that is an artefact of technological limitations. That the fast, processor-local storage was volatile, and non-volatile storage was slow and large enough that it had to be non-local. Nonvolatile storage is managed via APIs and discrete hardware controllers, whose main job was transferring blocks of data from volatile to non-volatile storage and back again.
And that distinction is going away.
The technology is rapidly evolving to the point where we have fast, processor-local storage, in memory slots, appearing directly in the CPUs' memory map, which is non-volatile.
Example -- Flash memory DIMMs:
https://www.theregister.co.uk/2015/11/10/micron_brings_out_a_flash_dimm/
Now, the non-volatile electronic storage is increasing rapidly in speed and decreasing in price.
Example -- Intel XPoint:
https://arstechnica.com/information-technology/2017/02/specs-for-first-intel-3d-xpoint-ssd-so-so-transfer-speed-awesome-random-io/
Note the specs:
Reads as fast as Flash.
Writes nearly the same speed as reads.
Half the latency of Flash.
100x the write lifetime of Flash.
And this is the very first shipping product.
Intel is promising "1,000 times faster than NAND flash, 10 times denser than (volatile) DRAM, and with 1,000 times the endurance of NAND".
This is a game-changer.
What we are looking at is a new type of computer.
3rd generation
No distinction between volatile and non-volatile storage. All storage appears directly in the CPUs' memory map. There are no discrete "drives" of any kind as standard. Why would you? You can have 500GB or 1TB of RAM, but if you turn the machine off, then a day later turn it back on, it carries on exactly where it was.
(Yes there will be some caching and there will need to be a bit of cleverness involving flushing them, or ACID writes, or something.)
It ships to the user with an OS in that memory.
You turn it on. It doesn't boot.
What is booting? Transferring OS code from non-volatile storage into volatile storage so it can be run. There's no need. It's in the processor's memory the moment it's turned on.
It doesn't boot. It never boots. It never shuts down, either. You may have tell it you're turning it off, but it flushes its caches and it's done. Power off.
No hibernation: it doesn't need to. The OS and all state data will be there when you come back. No sleep states: just power off.
What is installing an OS or an app? That means transferring from slow non-volatile storage to fast non-volatile storage. There is no slow or fast non-volatile storage. There's just storage. All of it that the programmer can see is non-volatile.
This is profoundly different to everything since 1957 or so.
It's also profoundly different from those 1980s 8-bits with their BASIC or Forth or monitor in ROM, because it's all writable.
That is the big change.
In current machines, nobody structures RAM as a filesystem. (I welcome corrections on this point!) Filesystems are for drives. Doesn't matter what kind of drive: hard disk, SSD, SD card, CD, DVD, DVD-RW, whatever. Different filesystems, but all need transfers from them to and from volatile storage to function.
That's going away. The writing is on the wall. The early tech is shipping right now.
What I am asking is, how will it change OS design?
All I am getting back is, "don't be stupid, it won't, OS design is great already, why would we change?"
This is the same attitude the DOS and CP/M app vendors had to Windows.
WordStar and dBase survived the transition from CP/M to MS-DOS.
They didn't survive the far bigger one from MS-DOS to Windows.
WordPerfect Corp and Lotus Corp tried. They still failed and died.
A bigger transition is about to happen. Let's talk about it instead of denying it's coming.