Thanks. Yes, it turns out this was not an issue with Ceph, but rather an issue with XenServer. Starting in version 7, Xenserver changed how they manage LVM by adding a VHD layer on top of it. They did it to handle live migrations but ironically broke live migrations when using any iSCSI including iSCSI to Ceph via lrbd. It works just fine with NFS based storage repositories but not block storage. Doesn't look like they are going to fix it since they are moving on to using glusterfs instead with an experimental version of it starting in XenServer 7.5
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ronny Aasen Sent: Monday, July 23, 2018 6:13 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Reclaim free space on RBD images that use Bluestore????? On 23.07.2018 22:18, Sean Bolding wrote: I have XenServers that connect via iSCSI to Ceph gateway servers that use lrbd and targetcli. On my ceph cluster the RBD images I create are used as storage repositories in Xenserver for the virtual machine vdisks. Whenever I delete a virtual machine, XenServer shows that the repository size has decreased. This also happens when I mount a virtual drive in Xenserver as a virtual drive in a Windows guest. If I delete a large file, such as an exported VM, it shows as deleted and space available. However; when check in Ceph using ceph -s or ceph df it still shows the space being used. I checked everywhere and it seems there was a reference to it here https://github.com/ceph/ceph/pull/14727 but not sure if a way to trim or discard freed blocks was ever implemented. The only way I have found is to play musical chairs and move the VMs to different repositories and then completely remove the old RBD images in ceph. This is not exactly easy to do. Is there a way to reclaim free space on RBD images that use Bluestore????? What commands do I use and where do I use this from? If such command exist do I run them on the ceph cluster or do I run them from XenServer? Please help. Sean I am not familiar with Xen, but it does sounds like you have a rbd mounted with a filesystem on the xen server. in that case it is the same as for other filesystems. Deleted files are just deleted in the file allocation table, and the RBD space is "reclaimed" when the filesystem zeroes out the now unused blocks. in many filesystems you would run the fstrim command to overwrite free'd blocks with zeroes, optionally mount the fs with the the discard option. in xenserver >6.5 this should be a button in xencenter to reclaim freed space. kind regards Ronny Aasen <http://t.sidekickopen08.com/e1t/o/5/f18dQhb0S7kC8dDMPbW2n0x6l2B9gXrN7sKj6v5 KRN6W56jV0P7dSBj2W5vbH2n6yGGzjf197v5Y04?si=7000000001238989&pi=37da8a12-5013 -4034-91ea-a306cca5c995>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com