Consolidated similar filesedmDecember 27 2010, 05:39:45 UTC
If you want zip files, and are using something with FUSE support (Linux, OS X?, others?), then fuse-zip might be worth a look. (I've not used it, but it has all the right buzzwords.) There are a bunch of other similar FUSE file systems, of which archivemount might be worth a look if you want tar files (I've not used that either).
Historically "library" files have been used for this purpose (eg, ".a" on unix systems), which were uncompressed (and optionally indexed), but most of the modern uses (JAR, etc) are based on zip. However, that's probably too generic to be something to search for. (I found the above links searching for "zip fuse" and "tar fuse" respectively, but that's one of those "know what you're looking for" situations.)
And yes I'd assume that both sufficient caching and efficient multiple I/O outstanding would minimise the benefits. Such things used to be really useful back in the days of tiny files and especially slow media (floppy drives!).
Solid state drives don't allow you to read multiple files at once any more (or less) than spinning disks do. Only one bit can be on the wire between the drive and controller at a time.
What solid state disks *do* do for you is reduce latency to zero, or next to it. ( no more head seek times or waiting for the disk to spin around ) That means multiple files, fragmented files, a monolithic contiguous file, it's all the same as far as the I/O goes.
zipping the files into a single archive may or may not gain you anything... depending on how much you're willing to load into ram and leave there, and whether you're looking at read-only disk access or read-write.
Comments 3
Historically "library" files have been used for this purpose (eg, ".a" on unix systems), which were uncompressed (and optionally indexed), but most of the modern uses (JAR, etc) are based on zip. However, that's probably too generic to be something to search for. (I found the above links searching for "zip fuse" and "tar fuse" respectively, but that's one of those "know what you're looking for" situations.)
And yes I'd assume that both sufficient caching and efficient multiple I/O outstanding would minimise the benefits. Such things used to be really useful back in the days of tiny files and especially slow media (floppy drives!).
Ewen
Reply
Reply
What solid state disks *do* do for you is reduce latency to zero, or next to it. ( no more head seek times or waiting for the disk to spin around ) That means multiple files, fragmented files, a monolithic contiguous file, it's all the same as far as the I/O goes.
zipping the files into a single archive may or may not gain you anything... depending on how much you're willing to load into ram and leave there, and whether you're looking at read-only disk access or read-write.
Reply
Leave a comment