Hi every one,
In the course of playing with RBD, I noticed a few things:
* The RBD images are so thin-provisioned, you can create arbitrarily
large ones.
On my 0.72.1 freshly-installed empty 200TB cluster, I was able to
create a 1PB image:
$ rbd create --image-format 2 --size 1073741824 test_img
This command is successful, and I can check the image status:
$ rbd info test
rbd image 'test':
size 1024 TB in 268435456 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.19f76b8b4567
format: 2
features: layering
* Such an oversized image seems unmountable on my 3.2.46 kernel.
However the error message is not very explicit:
$ rbd map test_img
rbd: add failed: (2) No such file or directory
There is no error or explanation to be seen anywhere in the logs.
dmesg reports the connection to the cluster through RBD as usual, and
that's it.
Using the exact same commands with image size 32GB will successfully
map the device.
* Such an oversize image takes an awfully long time to shrink or remove.
However, the image has just been created and is empty.
In RADOS, I only see the corresponding rbd_id and rbd_header, but no
data object at all.
Still, removing the 1PB image takes roughly 8 hours.
Cluster config:
3 mons, 8nodes * 72osds, about 4800pgs (2400pgs in pool "rbd")
cluster and public network are 10GbE, each node has 8 cores and 64GB mem
Oh, so my questions:
- why is it possible to create an image five times the size of the
cluster without warning?
- where could this "No such file" error come from?
- why does it take long to shrink/delete a
large-but-empty-and-thin-provisioned image?
I know that 1PB is oversized ("No such file" when trying to map), and
32GB is not, so I am currently looking for the oversize threshold. More
info coming soon.
Best regards,
Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
