Re: [ovirt-users] Non-responsive host, VM's are still running - how to resolve?
16', u'swap_in': '0', > u'swap_total': '786428', u'pageflt': '4346', u'mem_total': '3922564', > u'mem_unused': '1900456'}, 'session': 'Unknown', 'netIfaces': [], > 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} > Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, > ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' > args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'username': > 'Unknown', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': > '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', > u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', > u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', > u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], > 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} > Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, > ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' > args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': > 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': > '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', > u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', > u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', > u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], > 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Above logs say that there is no connection from the engine so events won't be sent. Can you share engine logs? > > On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <bu...@onholyground.com> > wrote: >> >> Try restarting vdsmd from the shell, “systemctl restart vdsmd”. >> >> >> >> From: Artem Tambovskiy <artem.tambovs...@gmail.com> >> Subject: [ovirt-users] Non-responsive host, VM's are still running - how >> to resolve? >> Date: November 14, 2017 at 11:23:32 AM CST >> To: users >> >> Apparently, i lost the host which was running hosted-engine and another 4 >> VM's exactly during migration of second host from bare-metal to second host >> in the cluster. For some reason first host entered the "Non reponsive" >> state. The interesting thing is that hosted-engine and all other VM's up and >> running, so its like a communication problem between hosted-engine and host. >> >> The engine.log at hosted-engine is full of following messages: >> >> 2017-11-14 17:06:43,158Z INFO >> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) >> [] Connecting to ovirt2/80.239.162.106 >> 2017-11-14 17:06:43,159Z ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] >> (DefaultQuartzScheduler9) [50938c3] Command >> 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, >> VdsIdVDSCommandParametersBase:{runAsync='true', >> hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: >> java.net.NoRouteToHostException: No route to host >> 2017-11-14 17:06:43,159Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] >> (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host >> 'ovirt2.telia.ru' - skipping VMs monitoring. >> 2017-11-14 17:06:45,929Z INFO >> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) >> [] Connecting to ovirt2/80.239.162.106 >> 2017-11-14 17:06:45,930Z ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >> (DefaultQuartzScheduler2) [6080f1cc] Command >> 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, >> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >> hostId='3970247c-69eb-4bd8-b263-9100703a8243', >> vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' >> execution failed: java.net.NoRouteToHostException: No route to host >> 2017-11-14 17:06:45,930Z ERROR >> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] >> (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host >> 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to >> host >> 2017-11-14 17:06:48,933Z INFO >> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) >> [] Connecting to ovirt2/80.239.162.106 >> 2017-11-14 17:06:48,934Z ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >> (DefaultQuartzScheduler6) [1a64dfea] Command >> 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, >> VdsIdAndVdsVDSCom
Re: [ovirt-users] Non-responsive host, VM's are still running - how to resolve?
', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <bu...@onholyground.com> wrote: > Try restarting vdsmd from the shell, “systemctl restart vdsmd”. > > > -- > *From:* Artem Tambovskiy <artem.tambovs...@gmail.com> > *Subject:* [ovirt-users] Non-responsive host, VM's are still running - > how to resolve? > *Date:* November 14, 2017 at 11:23:32 AM CST > *To:* users > > Apparently, i lost the host which was running hosted-engine and another 4 > VM's exactly during migration of second host from bare-metal to second host > in the cluster. For some reason first host entered the "Non reponsive" > state. The interesting thing is that hosted-engine and all other VM's up > and running, so its like a communication problem between hosted-engine and > host. > > The engine.log at hosted-engine is full of following messages: > > 2017-11-14 17:06:43,158Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] > (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 > 2017-11-14 17:06:43,159Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName > = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: > java.net.NoRouteToHostException: No route to host > 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core. > vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) > [50938c3] Failed to fetch vms info for host 'ovirt2.telia.ru' - skipping > VMs monitoring. > 2017-11-14 17:06:45,929Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] > (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 > 2017-11-14 17:06:45,930Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler2) [6080f1cc] Command > 'GetCapabilitiesVDSCommand(HostName > = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, > 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: > java.net.NoRouteToHostException: > No route to host > 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core. > vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] > Failure to refresh host 'ovirt2.telia.ru' runtime info: > java.net.NoRouteToHostException: > No route to host > 2017-11-14 17:06:48,933Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] > (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 > 2017-11-14 17:06:48,934Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler6) [1a64dfea] Command > 'GetCapabilitiesVDSCommand(HostName > = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, > 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: > java.net.NoRouteToHostException: > No route to host > 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core. > vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] > Failure to refresh host 'ovirt2.telia.ru' runtime info: > java.net.NoRouteToHostException: > No route to host > 2017-11-14 17:06:50,931Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] > (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 > 2017-11-14 17:06:50,932Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] > (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName > = ovirt2.telia.ru, SpmStatusVDSCom
Re: [ovirt-users] Non-responsive host, VM's are still running - how to resolve?
Try restarting vdsmd from the shell, “systemctl restart vdsmd”. > From: Artem Tambovskiy <artem.tambovs...@gmail.com> > Subject: [ovirt-users] Non-responsive host, VM's are still running - how to > resolve? > Date: November 14, 2017 at 11:23:32 AM CST > To: users > > Apparently, i lost the host which was running hosted-engine and another 4 > VM's exactly during migration of second host from bare-metal to second host > in the cluster. For some reason first host entered the "Non reponsive" state. > The interesting thing is that hosted-engine and all other VM's up and > running, so its like a communication problem between hosted-engine and host. > > The engine.log at hosted-engine is full of following messages: > > 2017-11-14 17:06:43,158Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] > Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> > 2017-11-14 17:06:43,159Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName > = ovirt2.telia.ru <http://ovirt2.telia.ru/>, > VdsIdVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: > java.net.NoRouteToHostException: No route to host > 2017-11-14 17:06:43,159Z INFO > [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] > (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host > 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' - skipping VMs monitoring. > 2017-11-14 17:06:45,929Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] > Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> > 2017-11-14 17:06:45,930Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler2) [6080f1cc] Command > 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru > <http://ovirt2.telia.ru/>, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru > <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution > failed: java.net.NoRouteToHostException: No route to host > 2017-11-14 17:06:45,930Z ERROR > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] > (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host 'ovirt2.telia.ru > <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No > route to host > 2017-11-14 17:06:48,933Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] > Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> > 2017-11-14 17:06:48,934Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler6) [1a64dfea] Command > 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru > <http://ovirt2.telia.ru/>, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru > <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution > failed: java.net.NoRouteToHostException: No route to host > 2017-11-14 17:06:48,934Z ERROR > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] > (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host 'ovirt2.telia.ru > <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No > route to host > 2017-11-14 17:06:50,931Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] > Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> > 2017-11-14 17:06:50,932Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] > (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = > ovirt2.telia.ru <http://ovirt2.telia.ru/>, > SpmStatusVDSCommandParameters:{runAsync='true', > hostId='3970247c-69eb-4bd8-b263-9100703a8243', > storagePoolId='5a044257-02ec-0382-0243-01f2'})' execution failed: > java.net.NoRouteToHostException: No route to host > 2017-11-14 17:06:50,939Z INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] > Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> > 2017-11-14 17:06:50,940Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] > (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS > 2017-11-14 17:06:50,940Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] > (DefaultQuartzScheduler4) [6b19d168] Command 'Ge
[ovirt-users] Non-responsive host, VM's are still running - how to resolve?
Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host. The engine.log at hosted-engine is full of following messages: 2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host ' ovirt2.telia.ru' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-01f2'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-01f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14