[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Sorry here, after some further investigation, this is not using a Gluster share. These particular VMs were set up using an NFS share on a NAS device we have. I don’t know why this was used for these particular devices. Maybe it was a temporary thing and the idea was to move these into the gluster setup eventually. Not sure. At any rate, the paused VMs don’t appear to be using gluster mounts for storage. From: jean-lo...@dupond.be (Jean-Louis Dupond) Sent: Monday, September 13, 2021 1:52 AM To: Strahil Nikolov ; marcel d'heureuse ; bob.fran...@mdaemon.com; users@ovirt.org Subject: Re: [ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error Which Qemu version are you using? Cause it might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1994494 Jean-Louis On 10/09/2021 19:59, Strahil Nikolov via Users wrote: Can you provide the output from all nodes: gluster pool list gluster peer status gluster volume status Best Regards, Strahil Nikolov On Fri, Sep 10, 2021 at 0:50, marcel d'heureuse <mailto:mar...@deheureu.se> wrote: ___ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-le...@ovirt.org <mailto:users-le...@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZNXYN2N2EAVKDTLJJJYGJLGG4ZAMZBEG/ ___ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-le...@ovirt.org <mailto:users-le...@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O3AO7BB6NGEBHJPV322RSIY3EDNANMYE/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YVGFTM5ATR6HBBOGDTG2KWIULE5ACQ3P/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Incidentally, I also did find this in the OVIRT logs: MainProcess|jsonrpc/7::DEBUG::2021-09-09 15:36:32,791::commands::219::root::(execCmd) FAILED: = 'mount.nfs: No route to host\n'; = 32 MainProcess|jsonrpc/7::DEBUG::2021-09-09 15:36:32,792::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 125 seconds (max pending: 0) MainProcess|jsonrpc/7::ERROR::2021-09-09 15:36:32,793::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in mount Not sure what it means. ‘No route to host’ is not accurate as the host the NFS mount is hosted on was completely accessible from a network perspective. Not sure if this is relevant or not, but thought I would include it. From: mar...@deheureu.se (marcel d'heureuse) Sent: Thursday, September 9, 2021 4:49 PM To: bob.fran...@mdaemon.com; users@ovirt.org Subject: Re: [ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error Moin, Normally you can use virsh to start vm which are in pause. https://www.google.com/amp/s/www.cyberciti.biz/faq/linux-list-a-kvm-vm-guest-using-virsh-command/ You have to use saslpasswd2 -a libvirt username to get access on comandline. It is importand that you Resum that vm on the same host where it was paused. Good luck... Br Marcel Am 9. September 2021 22:10:52 MESZ schrieb "bob.franzke--- via Users" mailto:users@ovirt.org> >: OK looking into this further, it seems that I was incorrect about the Gluster situation on these hosts. There are three hosts which form a replicated arbitrated gluster volume between them. There is a second gluster volume which is distributed but only seems to exist on the 4 host (the one the failed VMs are running on. I am not sure why this was set up this way. After all that I looked closer and realized there is an NFS share which these VMs disks are located on. I can access and write to this NFS share from the host itself so I am not sure what the story is as to why I cannot get this VM to come up. Seems like everything is there for it to do so. The error I get when trying to start up the VM is the following: has been paused due to storage I/O problem. How can I determine what this I/O problem actually is? IS the disk file corrupted somehow? Both VMs that won;t start are using this NFS share. Others that are running also use it so I am ot sure what the problem here is or where to start looking for an answer. Thanks in advance for your help. _ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-le...@ovirt.org <mailto:users-le...@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EX7TAYPEKKVEXQ275UNRRA473CBHB6JI/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2IACJWBMH7URUCM62MCYTDGPZWPUUGFM/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Thanks for the reply. I ended up moving the disk for another VM that was also paused and using the same NFS share for storage. I moved the disk to a different share, and then moved it back. Then to my surprise, the VM would start up. I then tried to start the VM this thread opened on and it also started up. So this appears to be due to some transient network error not being able to access the NFS share the VM was stored on. I am not sure what sort of checks Ovirt does to determine of the share is accessible, but we have seen this before with our OVIRT setup. Rebooting switches for maintenance for example would pause VMs in our setup. Rebooting the manager server after the network is stabilized seems to fix everything. I looked through the network logs to see what would have caused this but didn’t see any issue with the devices hosting this NFS share. So couldn’t be sure what caused this. It seems OVIRT is very intolerant of ANY network issues whatsoever, even ones designed to improve network stability and availability (spanning tree, failover mechanisms, etc.). In the event of a network situation, why wouldn’t OVIRT try accessing the share again once the network was stabilized. Perhaps this is simply due to poor design of our particular OVIRT installation, but it seems really finicky with ANYTHING changing on the network. If we were to design this again, I wouldn’t rely on any network storage to run VMs and put everything on gluster mounts local to the hosts. would appreciate any ideas on why this might have happened. Thanks for the reply. I’ll keep this link handy for the future on running VMs via command line. Thanks. Bob From: mar...@deheureu.se (marcel d'heureuse) Sent: Thursday, September 9, 2021 4:49 PM To: bob.fran...@mdaemon.com; users@ovirt.org Subject: Re: [ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error Moin, Normally you can use virsh to start vm which are in pause. https://www.google.com/amp/s/www.cyberciti.biz/faq/linux-list-a-kvm-vm-guest-using-virsh-command/ You have to use saslpasswd2 -a libvirt username to get access on comandline. It is importand that you Resum that vm on the same host where it was paused. Good luck... Br Marcel Am 9. September 2021 22:10:52 MESZ schrieb "bob.franzke--- via Users" mailto:users@ovirt.org> >: OK looking into this further, it seems that I was incorrect about the Gluster situation on these hosts. There are three hosts which form a replicated arbitrated gluster volume between them. There is a second gluster volume which is distributed but only seems to exist on the 4 host (the one the failed VMs are running on. I am not sure why this was set up this way. After all that I looked closer and realized there is an NFS share which these VMs disks are located on. I can access and write to this NFS share from the host itself so I am not sure what the story is as to why I cannot get this VM to come up. Seems like everything is there for it to do so. The error I get when trying to start up the VM is the following: has been paused due to storage I/O problem. How can I determine what this I/O problem actually is? IS the disk file corrupted somehow? Both VMs that won;t start are using this NFS share. Others that are running also use it so I am ot sure what the problem here is or where to start looking for an answer. Thanks in advance for your help. _ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-le...@ovirt.org <mailto:users-le...@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EX7TAYPEKKVEXQ275UNRRA473CBHB6JI/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VG433NBFHFY2YZ4DAX57IEEJ4SW3GM67/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Which Qemu version are you using? Cause it might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1994494 Jean-Louis On 10/09/2021 19:59, Strahil Nikolov via Users wrote: Can you provide the output from all nodes: gluster pool list gluster peer status gluster volume status Best Regards, Strahil Nikolov On Fri, Sep 10, 2021 at 0:50, marcel d'heureuse wrote: ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZNXYN2N2EAVKDTLJJJYGJLGG4ZAMZBEG/ ___ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-le...@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/O3AO7BB6NGEBHJPV322RSIY3EDNANMYE/___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PRSBLTW7WXJ7O2YBN47OLCW3P4TPTYTT/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Can you provide the output from all nodes: gluster pool listgluster peer statusgluster volume status Best Regards,Strahil Nikolov On Fri, Sep 10, 2021 at 0:50, marcel d'heureuse wrote: ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZNXYN2N2EAVKDTLJJJYGJLGG4ZAMZBEG/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O3AO7BB6NGEBHJPV322RSIY3EDNANMYE/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
Moin, Normally you can use virsh to start vm which are in pause. https://www.google.com/amp/s/www.cyberciti.biz/faq/linux-list-a-kvm-vm-guest-using-virsh-command/ You have to use saslpasswd2 -a libvirt username to get access on comandline. It is importand that you Resum that vm on the same host where it was paused. Good luck... Br Marcel Am 9. September 2021 22:10:52 MESZ schrieb "bob.franzke--- via Users" : >OK looking into this further, it seems that I was incorrect about the Gluster >situation on these hosts. There are three hosts which form a replicated >arbitrated gluster volume between them. There is a second gluster volume which >is distributed but only seems to exist on the 4 host (the one the failed VMs >are running on. I am not sure why this was set up this way. > >After all that I looked closer and realized there is an NFS share which these >VMs disks are located on. I can access and write to this NFS share from the >host itself so I am not sure what the story is as to why I cannot get this VM >to come up. Seems like everything is there for it to do so. The error I get >when trying to start up the VM is the following: > > has been paused due to storage I/O problem. > >How can I determine what this I/O problem actually is? IS the disk file >corrupted somehow? Both VMs that won;t start are using this NFS share. Others >that are running also use it so I am ot sure what the problem here is or >where to start looking for an answer. Thanks in advance for your help. >___ >Users mailing list -- users@ovirt.org >To unsubscribe send an email to users-le...@ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/EX7TAYPEKKVEXQ275UNRRA473CBHB6JI/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZNXYN2N2EAVKDTLJJJYGJLGG4ZAMZBEG/
[ovirt-users] Re: Cannot Start VM After Pausing due to Storage I/O Error
OK looking into this further, it seems that I was incorrect about the Gluster situation on these hosts. There are three hosts which form a replicated arbitrated gluster volume between them. There is a second gluster volume which is distributed but only seems to exist on the 4 host (the one the failed VMs are running on. I am not sure why this was set up this way. After all that I looked closer and realized there is an NFS share which these VMs disks are located on. I can access and write to this NFS share from the host itself so I am not sure what the story is as to why I cannot get this VM to come up. Seems like everything is there for it to do so. The error I get when trying to start up the VM is the following: has been paused due to storage I/O problem. How can I determine what this I/O problem actually is? IS the disk file corrupted somehow? Both VMs that won;t start are using this NFS share. Others that are running also use it so I am ot sure what the problem here is or where to start looking for an answer. Thanks in advance for your help. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EX7TAYPEKKVEXQ275UNRRA473CBHB6JI/
