Ceph: RBD resizing workarounds, Python API example
Some mis-reading the help for 'rbd resize' recently lead somebody to making an RBD volume extremely large, if the units are bytes, this command makes sense:
rbd resize --size $((155*1024*1024*1024)) $RBD_VOL_NAME
But, as it happens, the units are actually in MiB.
# rbd info $RBD_VOL_NAME rbd image '$RBD_VOL_NAME': size 155 PB in 41607495680 objects ... Oops, that's a bit too big, and I doubt your cluster has that much space for a single volume, even if it's sparsely allocated.
There is the --allow-shrink option to rbd resize, however on a volume this large, at least in the Hammer release, it will pretty much never return (it eventually times out).
Future work for RBD resize might include a much more intelligent resize command, that checks the highest non-null block, perhaps a reverse iterator looking for keys named at most the size of the device.
Making resize ask for confirmation with output of the new size, or have a dry-run option would also be helpful to avoid this problem.
So, as a workaround, I give you the following useful script, that manually changes the size omap field directly. It is wildly unsafe if you are already using anything past the target size, but it's fine otherwise.
You need to ensure nothing is using the volume when you apply this...
# Prerequisite: From Infernalis source or newer, src/pybind/rbd.py, src/pybind/rados.py # Make sure you have this fix: https://github.com/ceph/ceph/pull/6220 from rados import Rados, ReadOpCtx, WriteOpCtx import rbd import struct import ctypes RBD_NAME = '...' NEWSIZE = 155 * 1024 * 1024 * 1024 # 155 GiB with Rados(conffile='/etc/ceph/ceph.conf') as r: print 'RADOS object', r print 'RADOS version', r.version() print 'Cluster Stats', r.get_cluster_stats() with r.open_ioctx('rbd') as ioctx: with rbd.Image(ioctx, RBD_NAME, read_only=True) as img: imgstat = img.stat() print imgstat header_name = imgstat['block_name_prefix'].replace('rbd_data.', 'rbd_header.', 1) with ReadOpCtx(ioctx) as read_op: kv_it = ioctx.get_omap_vals(read_op, "", "", 500) ioctx.operate_read_op(read_op, header_name) kv = dict(kv_it[0]) print 'OMAP of', header_name, kv newkeys = ('size', ) newvals = (struct.pack('