[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned
On Fri, Aug 13, 2021 at 9:45 AM Boris Behrens wrote: > > Hi Janne, > thanks for the hint. I was aware of that, but it is goot to add that > knowledge to the question for further googlesearcher. > > Hi Ilya, > that fixed it. Do we know why the discard does not work when the partition > table is not aligned? We provide OS templates to our customer, but they can > also create and attach an empty block device, and they will certainly not > check if the partitions are aligned correctly. Hi Boris, Not the partition table but the partitions themselves. If the partition isn't aligned to rbd object size (but still aligned to something reasonable such as 1M), discard would work but *appear* to be ineffective because for large discard requests instead of just removing whole objects which is immediately visible in "rbd du" output it would have to resort to truncating and punching holes in those objects. To see some of the effect one would need to run "rbd du --exact" which is slow. Some of because "rbd du --exact" would basically sum up the remaining truncated object sizes but since holes don't consume space that figure would still be inaccurate (bigger). To see the real effect, one would need to look at "ceph osd df" DATA column before running discard (here removing the file) and after. And what Janne mentioned, of course. "-o discard" isn't guaranteed to actually free up space in all scenarios. This is why I said "for discard to work the same way as without partitioning" in my previous email -- because irrespective of partitioning online discards wouldn't always have the desired behaviour (on any block device, not just rbd). Thanks, Ilya > > Cheers > Boris > > > Am Fr., 13. Aug. 2021 um 08:44 Uhr schrieb Janne Johansson < > icepic...@gmail.com>: > > > Den tors 12 aug. 2021 kl 17:04 skrev Boris Behrens : > > > Hi everybody, > > > we just stumbled over a problem where the rbd image does not shrink, when > > > files are removed. > > > This only happenes when the rbd image is partitioned. > > > > > > * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition > > table > > > (/boot and /) > > > * the kvm device is virtio-scsi-pci with krbd > > > * Mount option discard is set > > > * command to create large file: dd if=/dev/zero of=testfile bs=64M > > > count=1000 > > > * the image grows in the size we expect > > > * when we remove the testfile the rbd image stays at the size > > > * we wen recreate the deleted file with the command the rbd image grows > > > further > > > > Just a small nit on this single point, regardless of if trim/discard > > works or not: > > There is no guarantee that writing a file, removing it and then > > re-writing a file > > will ever end up in the same spot again. In fact, most modern filesystems > > will > > probably make sure to NOT place things at the same spot again. > > Since the second write ends up in a different place, it will once again > > expand > > your sparse/thin image by the amount of written bytes, this is very much > > to be expected. > > > > I'm sorry if you already knew this and I am just stating the obvious to > > you, but > > your text came over as if you expected the second write to not increase the > > image since that "space" was already blown up on the first write. > > > > Trim/discard should still be investigated so you can make it shrink back > > again somehow, just wanted to point this out for the records. > > > > > > -- > > May the most significant bit of your life be positive. > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned
Hi Janne, thanks for the hint. I was aware of that, but it is goot to add that knowledge to the question for further googlesearcher. Hi Ilya, that fixed it. Do we know why the discard does not work when the partition table is not aligned? We provide OS templates to our customer, but they can also create and attach an empty block device, and they will certainly not check if the partitions are aligned correctly. Cheers Boris Am Fr., 13. Aug. 2021 um 08:44 Uhr schrieb Janne Johansson < icepic...@gmail.com>: > Den tors 12 aug. 2021 kl 17:04 skrev Boris Behrens : > > Hi everybody, > > we just stumbled over a problem where the rbd image does not shrink, when > > files are removed. > > This only happenes when the rbd image is partitioned. > > > > * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition > table > > (/boot and /) > > * the kvm device is virtio-scsi-pci with krbd > > * Mount option discard is set > > * command to create large file: dd if=/dev/zero of=testfile bs=64M > > count=1000 > > * the image grows in the size we expect > > * when we remove the testfile the rbd image stays at the size > > * we wen recreate the deleted file with the command the rbd image grows > > further > > Just a small nit on this single point, regardless of if trim/discard > works or not: > There is no guarantee that writing a file, removing it and then > re-writing a file > will ever end up in the same spot again. In fact, most modern filesystems > will > probably make sure to NOT place things at the same spot again. > Since the second write ends up in a different place, it will once again > expand > your sparse/thin image by the amount of written bytes, this is very much > to be expected. > > I'm sorry if you already knew this and I am just stating the obvious to > you, but > your text came over as if you expected the second write to not increase the > image since that "space" was already blown up on the first write. > > Trim/discard should still be investigated so you can make it shrink back > again somehow, just wanted to point this out for the records. > > > -- > May the most significant bit of your life be positive. > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned
Den tors 12 aug. 2021 kl 17:04 skrev Boris Behrens : > Hi everybody, > we just stumbled over a problem where the rbd image does not shrink, when > files are removed. > This only happenes when the rbd image is partitioned. > > * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition table > (/boot and /) > * the kvm device is virtio-scsi-pci with krbd > * Mount option discard is set > * command to create large file: dd if=/dev/zero of=testfile bs=64M > count=1000 > * the image grows in the size we expect > * when we remove the testfile the rbd image stays at the size > * we wen recreate the deleted file with the command the rbd image grows > further Just a small nit on this single point, regardless of if trim/discard works or not: There is no guarantee that writing a file, removing it and then re-writing a file will ever end up in the same spot again. In fact, most modern filesystems will probably make sure to NOT place things at the same spot again. Since the second write ends up in a different place, it will once again expand your sparse/thin image by the amount of written bytes, this is very much to be expected. I'm sorry if you already knew this and I am just stating the obvious to you, but your text came over as if you expected the second write to not increase the image since that "space" was already blown up on the first write. Trim/discard should still be investigated so you can make it shrink back again somehow, just wanted to point this out for the records. -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned
On Thu, Aug 12, 2021 at 5:03 PM Boris Behrens wrote: > > Hi everybody, > > we just stumbled over a problem where the rbd image does not shrink, when > files are removed. > This only happenes when the rbd image is partitioned. > > * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition table > (/boot and /) > * the kvm device is virtio-scsi-pci with krbd > * Mount option discard is set > * command to create large file: dd if=/dev/zero of=testfile bs=64M > count=1000 > * the image grows in the size we expect > * when we remove the testfile the rbd image stays at the size > * we wen recreate the deleted file with the command the rbd image grows > further > * using fstrim does not work > * adding a new disk and initialize the ext4 directly on the disk (wihtout > partitioning) the trim does work and the rbd image shrinks back to a couple > GB > * we use ceph 14.2.21 > > Does anybody experienced the same issue and maybe know how to solve the > problem? Hi Boris, For discard to work the same way as without partitioning, you need to make sure that your partitions start at rbd object size boundaries (see "order" in "rbd info" output). The default object size is 4M (8192 512-byte sectors). Here is an fdisk output for a 512M /boot partition and the remainder used for /: Command (m for help): p Disk /dev/rbd0: 100 GiB, 107374182400 bytes, 209715200 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 65536 bytes / 65536 bytes Disklabel type: gpt Disk identifier: 77553519-6CA1-DF48-8242-97F45B9E10C9 DeviceStart End Sectors Size Type /dev/rbd0p18192 1056767 1048576 512M Linux filesystem /dev/rbd0p2 1056768 209715166 208658399 99.5G Linux filesystem Note that "Start" values are wholly divisible by 8192. Thanks, Ilya ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Discard / Trim does not shrink rbd image size when disk is partitioned
Hi, have you checked ‚rbd sparsify‘ to reclaim unused space? Zitat von Boris Behrens : Hi everybody, we just stumbled over a problem where the rbd image does not shrink, when files are removed. This only happenes when the rbd image is partitioned. * We tested it with centos8/ubuntu20.04 with ext4 and a gpt partition table (/boot and /) * the kvm device is virtio-scsi-pci with krbd * Mount option discard is set * command to create large file: dd if=/dev/zero of=testfile bs=64M count=1000 * the image grows in the size we expect * when we remove the testfile the rbd image stays at the size * we wen recreate the deleted file with the command the rbd image grows further * using fstrim does not work * adding a new disk and initialize the ext4 directly on the disk (wihtout partitioning) the trim does work and the rbd image shrinks back to a couple GB * we use ceph 14.2.21 Does anybody experienced the same issue and maybe know how to solve the problem? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
