vSphere, OpenFiler, and Linux kernel >= 2.6.29
I've been messing around with VMware's vSphere 4.0 (VMware ESX/ESXi 4.0) these past couple of weeks, and I've also been messing around with OpenFiler as a SAN/NAS storage solution, both for the home network as well as some production servers. So far, I've been pleased with both, and of course, the next step would be to put OpenFiler on a VM. The kernel version that comes with the stock OF 2.3 install is a branch of 2.6.26, and the VMware Tools package (which includes, among other things, the new vmxnet3 network driver code) compiles and builds just fine with the standard tarball and vmware-install.pl script (provided that you make sure to delete any rogue VMware modules that may be present in the OF modules directory). Unfortunately, after running an update on the OpenFiler VM, you end up with kernel 2.6.29-X, and now VMware Tools only partially compiles. In particular, the HGFS, VMXNET, and VMXNET3 modules don't build. That sucks.
I Googled around for a solution and couldn't find one, but I did find someone on the OpenFiler forum who was having the same problem. After digging into some of the kernel source, I found that there were some differences in some of the task scheduling structs between 2.6.26 and 2.6.29. Also found some differences in the netdev_struct which led me to wonder why VMware's developers used "dev->priv" in a bunch of places when they could have just used the netdev_priv() macro. Of course, I don't know when the netdev_priv() macro got introduced - but it is in both 2.6.26 and 2.6.29.
Anyway, I've patched the three modules to work with OpenFiler's latest 2.6.29.x kernel, and you can download that patch from here: http://cauchydeadbunny.com/vmware-tools-2.6.29-patches.tgz.
I suspect that this patchset will work with any other Linux distro that uses kernel 2.6.26 or higher, but I haven't tested it with anything except the latest OpenFiler kernel (2.6.29.5-0.2) and the stock VMware 4.0.0 tools build 164009. When VMware comes out with the next version of VMware tools, odds are that this patchset will either need to be updated, or maybe it will just be unnecessary.
I Googled around for a solution and couldn't find one, but I did find someone on the OpenFiler forum who was having the same problem. After digging into some of the kernel source, I found that there were some differences in some of the task scheduling structs between 2.6.26 and 2.6.29. Also found some differences in the netdev_struct which led me to wonder why VMware's developers used "dev->priv" in a bunch of places when they could have just used the netdev_priv() macro. Of course, I don't know when the netdev_priv() macro got introduced - but it is in both 2.6.26 and 2.6.29.
Anyway, I've patched the three modules to work with OpenFiler's latest 2.6.29.x kernel, and you can download that patch from here: http://cauchydeadbunny.com/vmware-tools-2.6.29-patches.tgz.
I suspect that this patchset will work with any other Linux distro that uses kernel 2.6.26 or higher, but I haven't tested it with anything except the latest OpenFiler kernel (2.6.29.5-0.2) and the stock VMware 4.0.0 tools build 164009. When VMware comes out with the next version of VMware tools, odds are that this patchset will either need to be updated, or maybe it will just be unnecessary.