First off, I have very little hope, you'll be able to recover your data working 
at gluster level...

And then there is a lot of information missing between the lines: I guess you 
are using a 3 node HCI setup and were adding new disks (/dev/sdb) on all three 
nodes and trying to move the glusterfs to those new bigger disks?

Resizing/moving/adding or removing disks are "natural" operations for Gluster. 
But oVirt isn't "gluster native" and may not be so forgiving if you just swap 
device paths on bricks.

<rant>
Practical guides on how to replace the storage without down time (after all 
this is a HA solution, right?) are somehow missing from the oVirt 
documentation, and if I was a rich man, perhaps I'd get myself an RHV support 
contract and see if RHEL engineers would say anything but "not supported".
</rant>

The first thing I'd recommend is to create some temporary space. I found using 
an extra disk as NFS storage on one of the hosts was a good way to gain some 
maneuvering room e.g. for backups.

You can try to attach the disk of the broken VM as a secondary to another good 
VM to see if the data can be salvaged from there. But before you attach it (and 
perhaps an automatic fsck ruins it for you), you can perhaps create a copy to 
the NFS export/backup (domain).

If you weren't out of space, you'd just create a local copy and work with that. 
You can also try exporting the disk image, but there is a lot of untested or 
slow code in that operation from my experience.

If that image happens to be empty (I've seen that happen) or the data on it 
cannot be recovered, there is little to be gained, by trying to work at the 
GlusterFS level. The logical disk image file will be chunked into 64MB bits and 
their order is buried deep either in GlusterFS or in oVirt and perhaps your 
business is the better place to invest your energy.

But there is a good chance the data portion of that disk image still has your 
data. The fact that oVirt/KVM generally pauses VMs when it has issues with the 
storage, tends to preserve and protect your data rather better than what 
happens when physical hosts suffer brown outs or power glitches.

I guess you'll have learned that oVirt doesn't protect you from making 
mistakes, it only tries to offer some resilience against faults.

It's good and valuable to report these things, because it helps others to 
learn, too.

I sincerely hope you'll make do!
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XSCXBSKCI3RKM5FUH57WG6JCMHOE7PMB/

Reply via email to