VM granularity and data reduction
At VMworld I heard this idea a couple times, once from an analyst and once from somebody with interest in vVols:
With per-VM controls, an administrator could turn off compression and deduplication for VMs that don't benefit from it, and thus improve overall system performance.
In my mind, this is something that "sounds good" but falls apart based on closer examination.
1. Users generally don't do a good job telling whether compression and dedupe are effective on a VM.
One example provided in support of the idea is applications which consist of already-compressed data like video. Or, perhaps the VM is running with guest-level full disk encryption turned on. The OS could also be doing compression and dedupe within its own file system.
But, at least two of these examples are flawed. While compressed video files are unlikely to be compressed further, the VM contains OS files, file system metadata, and even video metadata, all of which compress well. Even the video portion may be duplicated if the guest OS performs any swapping, runs a log-structured file system, or contains redundant copies of the same video. A whole-VM or even whole-disk decision could abandon significant data reduction.
I would also argue that the guest OS generally should not be configured to do its own compression and deduplication. You will get worse behavior from turning on compression on all the individual VMs rather than letting the storage compress and dedupe globally, unless the storage controller is so short on CPU power that it can't do a good job, which brings us to the next point:
2. Poor inline compression and dedupe does not significantly affect storage throughput and latency.
I can tell you from personal experience at Tintri that we had far more problems with data that has high compression and dedup than data with low ratios! Data that doesn't compress nor dedupe is easy to accommodate; it doesn't take any more CPU cycles to fail to compress than it does to compress. But data that is extensively deduped requires more metadata operations than normal writes. Data that compresses well may leave a flash segment unfilled when the non-compressed version has exhausted non-volatile storage. (We don't compress data on the way to NVRAM, in part to keep latency low. This means that dedupe and compression don't have any first-order affects on write latency anyway--- only second-order effects based on resource consumption later in the write pipeline.)
Consider what turning off inline data reduction means. It does mean we would get some CPU savings from not compressing a block--- but this cost already has to be low to afford to do it inline, tens of thousands of times per second. There is no savings in checksumming the block, writing it to flash, modifying file system metadata, or garbage collecting the block later. If the system is properly engineered to perform inline data reduction, "turning it off" affects just a small part of the cost, one that may not even be the bottleneck for a particular workload.
However, if compression and dedupe are the bottleneck, the user is unlikely to be sure this is the case, which is my final point:
3. If there is a benefit to be gained, the storage system should do it automatically.
Tintri's goal is to build storage that "sees, learns, and adapts." If a particular VM does not compress or dedupe well, and there would be an overall system benefit in turning compression off for that VM, then the storage system should self-tune to achieve the benefit. Compare:
A. Administrator migrates a problem VM onto her storage
B. The problem VM causes bad storage behavior, which the storage system displays to the administrator in some comprehensible fashion (if she's lucky)
C. The administrator notices the problem, correctly diagnoses the underlying cause, and performs the correct configuration change without error (most of the time.)
D. Repeat it all over again for the next VM. Or the next 100 VMs. Or the next 1000 VMs across multiple storage appliances. If she can dedicate the time, compared to all the other things on her priority list.
with
A. The storage notices that a VM has unusual behavior and tailors its treatment of the VM to match.
Which world would you rather live in?
With per-VM controls, an administrator could turn off compression and deduplication for VMs that don't benefit from it, and thus improve overall system performance.
In my mind, this is something that "sounds good" but falls apart based on closer examination.
1. Users generally don't do a good job telling whether compression and dedupe are effective on a VM.
One example provided in support of the idea is applications which consist of already-compressed data like video. Or, perhaps the VM is running with guest-level full disk encryption turned on. The OS could also be doing compression and dedupe within its own file system.
But, at least two of these examples are flawed. While compressed video files are unlikely to be compressed further, the VM contains OS files, file system metadata, and even video metadata, all of which compress well. Even the video portion may be duplicated if the guest OS performs any swapping, runs a log-structured file system, or contains redundant copies of the same video. A whole-VM or even whole-disk decision could abandon significant data reduction.
I would also argue that the guest OS generally should not be configured to do its own compression and deduplication. You will get worse behavior from turning on compression on all the individual VMs rather than letting the storage compress and dedupe globally, unless the storage controller is so short on CPU power that it can't do a good job, which brings us to the next point:
2. Poor inline compression and dedupe does not significantly affect storage throughput and latency.
I can tell you from personal experience at Tintri that we had far more problems with data that has high compression and dedup than data with low ratios! Data that doesn't compress nor dedupe is easy to accommodate; it doesn't take any more CPU cycles to fail to compress than it does to compress. But data that is extensively deduped requires more metadata operations than normal writes. Data that compresses well may leave a flash segment unfilled when the non-compressed version has exhausted non-volatile storage. (We don't compress data on the way to NVRAM, in part to keep latency low. This means that dedupe and compression don't have any first-order affects on write latency anyway--- only second-order effects based on resource consumption later in the write pipeline.)
Consider what turning off inline data reduction means. It does mean we would get some CPU savings from not compressing a block--- but this cost already has to be low to afford to do it inline, tens of thousands of times per second. There is no savings in checksumming the block, writing it to flash, modifying file system metadata, or garbage collecting the block later. If the system is properly engineered to perform inline data reduction, "turning it off" affects just a small part of the cost, one that may not even be the bottleneck for a particular workload.
However, if compression and dedupe are the bottleneck, the user is unlikely to be sure this is the case, which is my final point:
3. If there is a benefit to be gained, the storage system should do it automatically.
Tintri's goal is to build storage that "sees, learns, and adapts." If a particular VM does not compress or dedupe well, and there would be an overall system benefit in turning compression off for that VM, then the storage system should self-tune to achieve the benefit. Compare:
A. Administrator migrates a problem VM onto her storage
B. The problem VM causes bad storage behavior, which the storage system displays to the administrator in some comprehensible fashion (if she's lucky)
C. The administrator notices the problem, correctly diagnoses the underlying cause, and performs the correct configuration change without error (most of the time.)
D. Repeat it all over again for the next VM. Or the next 100 VMs. Or the next 1000 VMs across multiple storage appliances. If she can dedicate the time, compared to all the other things on her priority list.
with
A. The storage notices that a VM has unusual behavior and tailors its treatment of the VM to match.
Which world would you rather live in?