Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Tue, Jun 28, 2016 at 06:06:39PM +0200, Simone Tiraboschi wrote: > On Tue, Jun 28, 2016 at 5:24 PM, Dan Kenigsberg wrote: > > On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote: > >> On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg wrote: > >> > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: > >> >> Hi! > >> > > >> > > >> > Thanks for the detailed logging! > >> > > >> >> > >> >> The broker error is: > >> >> > >> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > >> >> MainThread::INFO::2016-06-27 > >> >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > >> >> Success, id 140293563619152 > >> >> > >> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > >> >> Thread-25::ERROR::2016-06-27 > >> >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > >> >> Error while serving connection > >> >> Traceback (most recent call last): > >> >> File > >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > >> >> line 166, in handle > >> >> data) > >> >> File > >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > >> >> line 299, in _dispatch > >> >> .set_storage_domain(client, sd_type, **options) > >> >> File > >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > >> >> line 66, in set_storage_domain > >> >> self._backends[client].connect() > >> >> File > >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > >> >> line 400, in connect > >> >> volUUID=volume.volume_uuid > >> >> File > >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > >> >> line 245, in _get_volume_path > >> >> volUUID > >> > > >> > We have two issues here. First is that > >> > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 > >> > is still using vdscli to contact vdsm, instead of the preferred > >> > jsonrpccli. > >> > >> We already have this one: > >> https://bugzilla.redhat.com/show_bug.cgi?id=1101554 > >> > > Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient > > > > It does not seem to be the same thing; here I'm referring to > > ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one > > condition it's the soon-to-be-deprecated xmlrpc. > > ovirt-hosted-engine-setup is already using just API via jsonrpc. > The issue is that ovirt-hosted-engine-ha somewhere uses vdsClient > which AFAIK internally still uses xmlrpc It's not a general "somewhere", I gave the specific link ;-) https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
HI to All!! Il 28/06/2016 17.24, Stefano Danzi ha scritto: [CUT] We have two issues here. First is that https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 is still using vdscli to contact vdsm, instead of the preferred jsonrpccli. The second is that vdscli.connect's heuristic ends up reading the local server address from vdsm config, where it finds the default ipv6-local address of "::". Please try setting [addresses] management_ip='0.0.0.0' in your /etc/vdsm/vdsm.conf instead of the crontab hacks. this solve the issue, but I still to haven't default gateway on ovirtmgmt inteface. Using this configuration all work, but every 50 minutes I receive 3 email from broker: - ovirt-hosted-engine state transition StartState-ReinitializeFSM - ovirt-hosted-engine state transition ReinitializeFSM-EngineStarting - ovirt-hosted-engine state transition EngineStarting-EngineUp the uptime of engine is about 15 hours, so really the engine VM is not rebooted. Agent log: MainThread::INFO::2016-06-29 08:46:10,453::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current sta te ReinitializeFSM (score: 0) MainThread::INFO::2016-06-29 08:46:20,547::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1467182780. 55 type=state_transition detail=ReinitializeFSM-EngineStarting hostname='ovirt01.hawai.lan' MainThread::INFO::2016-06-29 08:46:20,732::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of st ate_transition (ReinitializeFSM-EngineStarting) sent? sent MainThread::INFO::2016-06-29 08:46:20,733::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializin g VDSM MainThread::INFO::2016-06-29 08:46:24,430::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) C onnecting the storage MainThread::INFO::2016-06-29 08:46:24,431::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Conn ecting storage server MainThread::INFO::2016-06-29 08:46:31,764::storage_server::225::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Conn ecting storage server MainThread::INFO::2016-06-29 08:46:31,780::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refr eshing the storage domain MainThread::INFO::2016-06-29 08:46:31,945::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) P reparing images MainThread::INFO::2016-06-29 08:46:31,946::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2016-06-29 08:46:35,895::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) R eloading vm.conf from the shared storage domain MainThread::INFO::2016-06-29 08:46:35,896::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2016-06-29 08:46:39,621::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9 MainThread::INFO::2016-06-29 08:46:39,667::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, volUUID:3c477b06-063e-4f01-bd05-84c7d467742b MainThread::INFO::2016-06-29 08:46:39,760::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2016-06-29 08:46:39,761::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b MainThread::INFO::2016-06-29 08:46:39,772::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Found an OVF for HE VM, trying to convert MainThread::INFO::2016-06-29 08:46:39,774::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Got vm.conf from OVF_STORE MainThread::INFO::2016-06-29 08:46:43,489::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2016-06-29 08:46:53,605::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while tra
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Tue, Jun 28, 2016 at 5:24 PM, Dan Kenigsberg wrote: > On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote: >> On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg wrote: >> > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: >> >> Hi! >> > >> > >> > Thanks for the detailed logging! >> > >> >> >> >> The broker error is: >> >> >> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <== >> >> MainThread::INFO::2016-06-27 >> >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) >> >> Success, id 140293563619152 >> >> >> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <== >> >> Thread-25::ERROR::2016-06-27 >> >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) >> >> Error while serving connection >> >> Traceback (most recent call last): >> >> File >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", >> >> line 166, in handle >> >> data) >> >> File >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", >> >> line 299, in _dispatch >> >> .set_storage_domain(client, sd_type, **options) >> >> File >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", >> >> line 66, in set_storage_domain >> >> self._backends[client].connect() >> >> File >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", >> >> line 400, in connect >> >> volUUID=volume.volume_uuid >> >> File >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", >> >> line 245, in _get_volume_path >> >> volUUID >> > >> > We have two issues here. First is that >> > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 >> > is still using vdscli to contact vdsm, instead of the preferred >> > jsonrpccli. >> >> We already have this one: >> https://bugzilla.redhat.com/show_bug.cgi?id=1101554 >> > Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient > > It does not seem to be the same thing; here I'm referring to > ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one > condition it's the soon-to-be-deprecated xmlrpc. ovirt-hosted-engine-setup is already using just API via jsonrpc. The issue is that ovirt-hosted-engine-ha somewhere uses vdsClient which AFAIK internally still uses xmlrpc ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote: > On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg wrote: > > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: > >> Hi! > > > > > > Thanks for the detailed logging! > > > >> > >> The broker error is: > >> > >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > >> MainThread::INFO::2016-06-27 > >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > >> Success, id 140293563619152 > >> > >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > >> Thread-25::ERROR::2016-06-27 > >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > >> Error while serving connection > >> Traceback (most recent call last): > >> File > >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > >> line 166, in handle > >> data) > >> File > >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > >> line 299, in _dispatch > >> .set_storage_domain(client, sd_type, **options) > >> File > >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > >> line 66, in set_storage_domain > >> self._backends[client].connect() > >> File > >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > >> line 400, in connect > >> volUUID=volume.volume_uuid > >> File > >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > >> line 245, in _get_volume_path > >> volUUID > > > > We have two issues here. First is that > > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 > > is still using vdscli to contact vdsm, instead of the preferred > > jsonrpccli. > > We already have this one: > https://bugzilla.redhat.com/show_bug.cgi?id=1101554 > Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient It does not seem to be the same thing; here I'm referring to ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one condition it's the soon-to-be-deprecated xmlrpc. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
Il 28/06/2016 15.02, Dan Kenigsberg ha scritto: On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: Hi! Thanks for the detailed logging! The broker error is: ==> /var/log/ovirt-hosted-engine-ha/agent.log <== MainThread::INFO::2016-06-27 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140293563619152 ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-25::ERROR::2016-06-27 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID We have two issues here. First is that https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 is still using vdscli to contact vdsm, instead of the preferred jsonrpccli. The second is that vdscli.connect's heuristic ends up reading the local server address from vdsm config, where it finds the default ipv6-local address of "::". Please try setting [addresses] management_ip='0.0.0.0' in your /etc/vdsm/vdsm.conf instead of the crontab hacks. this solve the issue, but I still to haven't default gateway on ovirtmgmt inteface. Would you please open a bug about the two issues (ovirt-hosted-engine-ha and vdsm networking)? Here: https://bugzilla.redhat.com/show_bug.cgi?id=1350883 Would you report the output of `netstats -nltp` on your host, as I do not completely understand why no interface (not even the loopback one) was listening on ipv6? Here: [root@ovirt01 ~]# netstat -nltp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:39373 0.0.0.0:* LISTEN 2200/rpc.statd tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 969/rpcbind tcp0 0 0.0.0.0:54322 0.0.0.0:* LISTEN 943/python tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1571/sshd tcp0 0 0.0.0.0:858 0.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 1929/glusterfsd tcp0 0 0.0.0.0:49153 0.0.0.0:* LISTEN 1968/glusterfsd tcp0 0 0.0.0.0:20490.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:16514 0.0.0.0:* LISTEN 1603/libvirtd tcp0 0 0.0.0.0:38468 0.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:38469 0.0.0.0:* LISTEN 1946/glusterfs tcp0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 1585/glusterd tcp6 0 0 :::54321:::* LISTEN 1893/python tcp6 0 0 :::22 :::* LISTEN 1571/sshd tcp6 0 0 :::16514:::* LISTEN 1603/libvirtd Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg wrote: > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: >> Hi! > > > Thanks for the detailed logging! > >> >> The broker error is: >> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <== >> MainThread::INFO::2016-06-27 >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) >> Success, id 140293563619152 >> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <== >> Thread-25::ERROR::2016-06-27 >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) >> Error while serving connection >> Traceback (most recent call last): >> File >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", >> line 166, in handle >> data) >> File >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", >> line 299, in _dispatch >> .set_storage_domain(client, sd_type, **options) >> File >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", >> line 66, in set_storage_domain >> self._backends[client].connect() >> File >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", >> line 400, in connect >> volUUID=volume.volume_uuid >> File >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", >> line 245, in _get_volume_path >> volUUID > > We have two issues here. First is that > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 > is still using vdscli to contact vdsm, instead of the preferred > jsonrpccli. We already have this one: https://bugzilla.redhat.com/show_bug.cgi?id=1101554 > The second is that vdscli.connect's heuristic ends up reading the local > server address from vdsm config, where it finds the default ipv6-local > address of "::". > > Please try setting > > [addresses] > management_ip='0.0.0.0' > > in your /etc/vdsm/vdsm.conf instead of the crontab hacks. > > > Would you please open a bug about the two issues > (ovirt-hosted-engine-ha and vdsm networking)? > > Would you report the output of `netstats -nltp` on your host, as I do > not completely understand why no interface (not even the loopback one) > was listening on ipv6? > > Regards, > Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote: > Hi! Thanks for the detailed logging! > > The broker error is: > > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > MainThread::INFO::2016-06-27 > 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140293563619152 > > ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > Thread-25::ERROR::2016-06-27 > 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Error while serving connection > Traceback (most recent call last): > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 166, in handle > data) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 299, in _dispatch > .set_storage_domain(client, sd_type, **options) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > line 66, in set_storage_domain > self._backends[client].connect() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 400, in connect > volUUID=volume.volume_uuid > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 245, in _get_volume_path > volUUID We have two issues here. First is that https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 is still using vdscli to contact vdsm, instead of the preferred jsonrpccli. The second is that vdscli.connect's heuristic ends up reading the local server address from vdsm config, where it finds the default ipv6-local address of "::". Please try setting [addresses] management_ip='0.0.0.0' in your /etc/vdsm/vdsm.conf instead of the crontab hacks. Would you please open a bug about the two issues (ovirt-hosted-engine-ha and vdsm networking)? Would you report the output of `netstats -nltp` on your host, as I do not completely understand why no interface (not even the loopback one) was listening on ipv6? Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
Hi! The broker error is: ==> /var/log/ovirt-hosted-engine-ha/agent.log <== MainThread::INFO::2016-06-27 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140293563619152 ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-25::ERROR::2016-06-27 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable Broker PID is: [root@ovirt01 ~]# ps -ef | grep bro vdsm 925 1 0 09:23 ?00:00:01 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker --no-daemon root 3558 3405 0 09:28 pts/000:00:00 grep --color=auto bro [root@ovirt01 ~]# strace -f -p 925 [..] [pid 3748] munmap(0x7f98bded, 4096) = 0 _*[pid 3748] socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 8*__* *__*[pid 3748] connect(8, {sa_family=AF_INET6, sin6_port=htons(54321), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)*_ [pid 3748] close(8)= 0 [pid 3748] select(0, NULL, NULL, NULL, {1, 0} [pid 3742] <... recvfrom resumed> "set-storage-domain VdsmBackend h"..., 8192, 0, NULL, NULL) = 676 [pid 3742] fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) [pid 3742] fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 [pid 3742] openat(AT_FDCWD, "/rhev/data-center/mnt", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8 [pid 3742] getdents(8, /* 7 entries */, 32768) = 312 [pid 3742] getdents(8, /* 0 entries */, 32768) = 0 [pid 3742] close(8)= 0 [pid 3742] access("/rhev/data-center/mnt/ovirt01.hawai.lan:_home_storage-ovirtShe/46f55a31-f35f-465c-b3e2-df45c05e06a7", F_OK) = -1 ENOENT (No such file or directory) [pid 3742] access("/rhev/data-center/mnt/glusterSD/46f55a31-f35f-465c-b3e2-df45c05e06a7", F_OK) = -1 ENOENT (No such file or directory) [pid 3742] access("/rhev/data-center/mnt/ovirtbk-mount.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7", F_OK) = -1 ENOENT (No such file or directory) [pid 3742] access("/rhev/data-center/mnt/ovirtbk-sheng.hawai.lan:_var_lib_exports_iso/46f55a31-f35f-465c-b3e2-df45c05e06a7", F_OK) = -1 ENOENT (No such file or directory) [pid 3742] access("/rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7", F_OK) = 0 [pid 3742] access("/etc/pki/vdsm/keys/vdsmkey.pem", R_OK) = 0 [pid 3742] access("/etc/pki/vdsm/certs/vdsmcert.pem", R_OK) = 0 [pid 3742] access("/etc/pki/vdsm/certs/cacert.pem", R_OK) = 0 [pid 3742] open("/etc/pki/vdsm/certs/vdsmcert.pem", O_RDONLY) = 8 [pid 3742] fstat(8, {st_mode=S_IFREG|0644, st_size=1574, ...}) = 0 [pid 3742] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f98bded [pid 3742] read(8, "-BEGIN CERTIFICATE-\nMIIE"..., 4096) = 1574 [pid 3742] read(8, "", 4096) = 0 [pid 3742] close(8)= 0 [pid 3742] munmap(0x7f98bded, 4096) = 0 [pid 3742] open("/etc/pki/vdsm/keys/vdsmkey.pem", O_RDONLY) = 8 [pid 3742] fstat(8, {st_mode=S_IFREG|0440, st_size=
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Fri, Jun 24, 2016 at 06:45:24PM +0200, Stefano Danzi wrote: > HI!! > > I found a workaround > > the brocker process try to connect to vdsm to IPV4 host address using an > IPV6 connection > (I noticed that doing a strace to the process), > but ipv6 is not intialized at boot. (why connect to IPV4 address using > IPV6?) Acutally, we take an effort to disable ipv6 on ovirt host networks. Keeping them open without explicit request was deemed a security issue. Can you share your strace line and the relevant lines in vdsm.log? I don't understand what is the issue that you are reporting. > > I added the following lines to crontab: > > @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6' | > /usr/bin/at now+1 minutes > @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/ovirtmgmt/disable_ipv6' | > /usr/bin/at now+1 minutes > @reboot echo '/usr/sbin/route add default gw 192.168.1.254' | /usr/bin/at > now+1 minutes > > > > Il 24/06/2016 12.36, Stefano Danzi ha scritto: > > How I can change self hosted engine configuration to mount directly > > gluster storage without pass through gluster NFS? > > > > Maybe this solve > > > > Il 24/06/2016 10.16, Stefano Danzi ha scritto: > > > After an additional yum clean all && yum update was updated some > > > other rpms. > > > > > > Something changed. > > > My setup has engine storage on gluster, but mounted with NFS. > > > Now gluster daemon don't automatically start at boot. After starting > > > manually gluster the error is the same: > > > > > > ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > > > Thread-19::ERROR::2016-06-24 > > > 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > > > Error while serving connection > > > Traceback (most recent call last): > > > File > > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > > > line 166, in handle > > > data) > > > File > > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > > > line 299, in _dispatch > > > .set_storage_domain(client, sd_type, **options) > > > File > > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > > > line 66, in set_storage_domain > > > self._backends[client].connect() > > > File > > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > > > line 400, in connect > > > volUUID=volume.volume_uuid > > > File > > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > > > line 245, in _get_volume_path > > > volUUID > > > File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ > > > return self.__send(self.__name, args) > > > File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request > > > verbose=self.__verbose > > > File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request > > > return self.single_request(host, handler, request_body, verbose) > > > File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request > > > self.send_content(h, request_body) > > > File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content > > > connection.endheaders(request_body) > > > File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders > > > self._send_output(message_body) > > > File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output > > > self.send(msg) > > > File "/usr/lib64/python2.7/httplib.py", line 797, in send > > > self.connect() > > > File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line > > > 203, in connect > > > sock = socket.create_connection((self.host, self.port), self.timeout) > > > File "/usr/lib64/python2.7/socket.py", line 571, in create_connection > > > raise err > > > error: [Errno 101] Network is unreachable > > > > > > > > > VDSM.log > > > > > > jsonrpc.Executor/5::DEBUG::2016-06-24 > > > 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) > > > Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0 > > > aborting False > > > jsonrpc.Executor/5::WARNING::2016-06-24 > > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) > > > Following parameters ['type'] were not recogn > > > ized > > > jsonrpc.Executor/5::WARNING::2016-06-24 > > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) > > > Provided value "2" not defined in DiskType en > > > um for Volume.getInfo > > > jsonrpc.Executor/5::WARNING::2016-06-24 > > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) > > > Parameter capacity is not uint type > > > jsonrpc.Executor/5::WARNING::2016-06-24 > > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) > > > Required property allocType is not provided w > > > hen calling Volume.getInfo > > > jsonrpc.Executor/5::WARNING::2016-06-24 > > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) > > > Parameter mtime
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
HI!! I found a workaround the brocker process try to connect to vdsm to IPV4 host address using an IPV6 connection (I noticed that doing a strace to the process), but ipv6 is not intialized at boot. (why connect to IPV4 address using IPV6?) I added the following lines to crontab: @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6' | /usr/bin/at now+1 minutes @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/ovirtmgmt/disable_ipv6' | /usr/bin/at now+1 minutes @reboot echo '/usr/sbin/route add default gw 192.168.1.254' | /usr/bin/at now+1 minutes Il 24/06/2016 12.36, Stefano Danzi ha scritto: How I can change self hosted engine configuration to mount directly gluster storage without pass through gluster NFS? Maybe this solve Il 24/06/2016 10.16, Stefano Danzi ha scritto: After an additional yum clean all && yum update was updated some other rpms. Something changed. My setup has engine storage on gluster, but mounted with NFS. Now gluster daemon don't automatically start at boot. After starting manually gluster the error is the same: ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-19::ERROR::2016-06-24 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable VDSM.log jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0 aborting False jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Following parameters ['type'] were not recogn ized jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Provided value "2" not defined in DiskType en um for Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter capacity is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Required property allocType is not provided w hen calling Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter mtime is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter ctime is not int type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter truesize is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter apparentsize is not uint type jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Volume.getInfo' in bridge with {'sta tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': '--00 00--', 'format': 'RAW', 'image': '6838c974-7656-4b40-87cc-f562ff0b2a4c', '
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
How I can change self hosted engine configuration to mount directly gluster storage without pass through gluster NFS? Maybe this solve Il 24/06/2016 10.16, Stefano Danzi ha scritto: After an additional yum clean all && yum update was updated some other rpms. Something changed. My setup has engine storage on gluster, but mounted with NFS. Now gluster daemon don't automatically start at boot. After starting manually gluster the error is the same: ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-19::ERROR::2016-06-24 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable VDSM.log jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0 aborting False jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Following parameters ['type'] were not recogn ized jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Provided value "2" not defined in DiskType en um for Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter capacity is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Required property allocType is not provided w hen calling Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter mtime is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter ctime is not int type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter truesize is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter apparentsize is not uint type jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Volume.getInfo' in bridge with {'sta tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': '--00 00--', 'format': 'RAW', 'image': '6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 'truesize ': '1048576', 'type': 'PREALLOCATED'} JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,807::betterAsyncore::132::vds.dispatcher
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
After an additional yum clean all && yum update was updated some other rpms. Something changed. My setup has engine storage on gluster, but mounted with NFS. Now gluster daemon don't automatically start at boot. After starting manually gluster the error is the same: ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-19::ERROR::2016-06-24 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable VDSM.log jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0 aborting False jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Following parameters ['type'] were not recogn ized jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Provided value "2" not defined in DiskType en um for Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter capacity is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Required property allocType is not provided w hen calling Volume.getInfo jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter mtime is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter ctime is not int type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter truesize is not uint type jsonrpc.Executor/5::WARNING::2016-06-24 10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) Parameter apparentsize is not uint type jsonrpc.Executor/5::DEBUG::2016-06-24 10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Volume.getInfo' in bridge with {'sta tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': '--00 00--', 'format': 'RAW', 'image': '6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 'truesize ': '1048576', 'type': 'PREALLOCATED'} JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::WARNING::2016-06-24 10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled close event JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,807::betterAsyncore::132::vds.dispatcher::(send) SSL error during sending data: (104, 'Connection r eset by peer') JsonRpc (StompReactor)::ERROR::2016-06-24 10:10:43,959::betterAsyncore::113::vds.dispatcher::(recv) SSL error du
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Thu, Jun 23, 2016 at 11:46 PM, Stefano Danzi wrote: > Hi! > After cleanin metadata yum do an update of vdsm: > > [root@ovirt01 ~]# rpm -qva | grep vdsm > vdsm-yajsonrpc-4.18.4.1-0.el7.centos.noarch > vdsm-infra-4.18.4.1-0.el7.centos.noarch > vdsm-cli-4.18.4.1-0.el7.centos.noarch > vdsm-python-4.18.4.1-0.el7.centos.noarch > vdsm-hook-vmfex-dev-4.18.4.1-0.el7.centos.noarch > vdsm-xmlrpc-4.18.4.1-0.el7.centos.noarch > vdsm-4.18.4.1-0.el7.centos.x86_64 > vdsm-api-4.18.4.1-0.el7.centos.noarch > vdsm-gluster-4.18.4.1-0.el7.centos.noarch > vdsm-jsonrpc-4.18.4.1-0.el7.centos.noarch > > But this not solve the issue. > > - Host haven't default gateway after a reboot > - Self Hosted engine don't start. > Martin, Dan, can you please look into this? Stefano, can you please share a full sos report from the host? > > vdsm.log: > > https://drive.google.com/file/d/0ByMG4sDqvlZcVEJ5YVI1UWxrdE0/view?usp=sharing > > > Il 2016-06-23 21:41 Sandro Bonazzola ha scritto: > >> On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi >> wrote: >> >> Hi! >>> I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start >>> the self hosted engine. >>> >> >> Hi Stefano, can you please try "yum clean metadata" "yum update" >> again? >> You should get vdsm 4.18.4.1, please let us know if this solve your >> issue. >> > -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
Hi! After cleanin metadata yum do an update of vdsm: [root@ovirt01 ~]# rpm -qva | grep vdsm vdsm-yajsonrpc-4.18.4.1-0.el7.centos.noarch vdsm-infra-4.18.4.1-0.el7.centos.noarch vdsm-cli-4.18.4.1-0.el7.centos.noarch vdsm-python-4.18.4.1-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.18.4.1-0.el7.centos.noarch vdsm-xmlrpc-4.18.4.1-0.el7.centos.noarch vdsm-4.18.4.1-0.el7.centos.x86_64 vdsm-api-4.18.4.1-0.el7.centos.noarch vdsm-gluster-4.18.4.1-0.el7.centos.noarch vdsm-jsonrpc-4.18.4.1-0.el7.centos.noarch But this not solve the issue. - Host haven't default gateway after a reboot - Self Hosted engine don't start. vdsm.log: https://drive.google.com/file/d/0ByMG4sDqvlZcVEJ5YVI1UWxrdE0/view?usp=sharing Il 2016-06-23 21:41 Sandro Bonazzola ha scritto: On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi wrote: Hi! I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self hosted engine. Hi Stefano, can you please try "yum clean metadata" "yum update" again? You should get vdsm 4.18.4.1, please let us know if this solve your issue. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi wrote: > > Hi! > I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self > hosted engine. > Hi Stefano, can you please try "yum clean metadata" "yum update" again? You should get vdsm 4.18.4.1, please let us know if this solve your issue. > > first thing is that the host network lose the degaut gateway > configuration. But this is not the problem. > > Logs: > > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > MainThread::INFO::2016-06-23 > 18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) > Reloading vm.conf from the shared storage domain > MainThread::INFO::2016-06-23 > 18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Trying to get a fresher copy of vm configuration from the OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, > volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9 > MainThread::INFO::2016-06-23 > 18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, > volUUID:3c477b06-063e-4f01-bd05-84c7d467742b > MainThread::INFO::2016-06-23 > 18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) > Extracting Engine VM OVF from the OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) > OVF_STORE volume path: > /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b > MainThread::INFO::2016-06-23 > 18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Found an OVF for HE VM, trying to convert > MainThread::INFO::2016-06-23 > 18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Got vm.conf from OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > Initializing ha-broker connection > MainThread::INFO::2016-06-23 > 18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor ping, options {'addr': '192.168.1.254'} > > ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > Thread-25::ERROR::2016-06-23 > 18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Error while serving connection > Traceback (most recent call last): > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 166, in handle > data) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 299, in _dispatch > .set_storage_domain(client, sd_type, **options) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > line 66, in set_storage_domain > self._backends[client].connect() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 400, in connect > volUUID=volume.volume_uuid > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 245, in _get_volume_path > volUUID > File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ > return self.__send(self.__name, args) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request > verbose=self.__verbose > File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request > return self.single_request(host, handler, request_body, verbose) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request > self.send_content(h, request_body) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content > connection.endheaders(request_body) > File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders > self._send_output(message_body) > File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output > self.send(msg) > File "/usr/lib64/python2.7/httplib.py", line 797, in send > self.connect() > File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in > connect > sock = socket.create_connection((self.host, self.port), self.timeout) > File "/usr/lib64/python2.7/socket.py", line 571, in create_connection > raise err > error: [Errno 101] Network is unreachable > > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > MainThread::INFO::2016-06-23 > 18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > Failed set the stor
Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
> On 23 Jun 2016, at 18:36, Stefano Danzi wrote: > > > Hi! > I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self > hosted engine. > > first thing is that the host network lose the degaut gateway configuration. > But this is not the problem. > > Logs: > > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > MainThread::INFO::2016-06-23 > 18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) > Reloading vm.conf from the shared storage domain > MainThread::INFO::2016-06-23 > 18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Trying to get a fresher copy of vm configuration from the OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, > volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9 > MainThread::INFO::2016-06-23 > 18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) > Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, > volUUID:3c477b06-063e-4f01-bd05-84c7d467742b > MainThread::INFO::2016-06-23 > 18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) > Extracting Engine VM OVF from the OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) > OVF_STORE volume path: > /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b > MainThread::INFO::2016-06-23 > 18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Found an OVF for HE VM, trying to convert > MainThread::INFO::2016-06-23 > 18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) > Got vm.conf from OVF_STORE > MainThread::INFO::2016-06-23 > 18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > Initializing ha-broker connection > MainThread::INFO::2016-06-23 > 18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor ping, options {'addr': '192.168.1.254'} > > ==> /var/log/ovirt-hosted-engine-ha/broker.log <== > Thread-25::ERROR::2016-06-23 > 18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Error while serving connection > Traceback (most recent call last): > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 166, in handle >data) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > line 299, in _dispatch >.set_storage_domain(client, sd_type, **options) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > line 66, in set_storage_domain >self._backends[client].connect() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 400, in connect >volUUID=volume.volume_uuid > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", > line 245, in _get_volume_path >volUUID > File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ >return self.__send(self.__name, args) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request >verbose=self.__verbose > File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request >return self.single_request(host, handler, request_body, verbose) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request >self.send_content(h, request_body) > File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content >connection.endheaders(request_body) > File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders >self._send_output(message_body) > File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output >self.send(msg) > File "/usr/lib64/python2.7/httplib.py", line 797, in send >self.connect() > File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in > connect >sock = socket.create_connection((self.host, self.port), self.timeout) > File "/usr/lib64/python2.7/socket.py", line 571, in create_connection >raise err > error: [Errno 101] Network is unreachable > > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > MainThread::INFO::2016-06-23 > 18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) > Failed set the storage domain: 'Failed to set storage domain VdsmBackend, > options {'hosted-engine.lockspace': > '7B22696D6167655F75756964223A2022656631313
[ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0
Hi! I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self hosted engine. first thing is that the host network lose the degaut gateway configuration. But this is not the problem. Logs: ==> /var/log/ovirt-hosted-engine-ha/agent.log <== MainThread::INFO::2016-06-23 18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2016-06-23 18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2016-06-23 18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9 MainThread::INFO::2016-06-23 18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, volUUID:3c477b06-063e-4f01-bd05-84c7d467742b MainThread::INFO::2016-06-23 18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2016-06-23 18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b MainThread::INFO::2016-06-23 18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Found an OVF for HE VM, trying to convert MainThread::INFO::2016-06-23 18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Got vm.conf from OVF_STORE MainThread::INFO::2016-06-23 18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2016-06-23 18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.1.254'} ==> /var/log/ovirt-hosted-engine-ha/broker.log <== Thread-25::ERROR::2016-06-23 18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error while serving connection Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 400, in connect volUUID=volume.volume_uuid File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 245, in _get_volume_path volUUID File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect sock = socket.create_connection((self.host, self.port), self.timeout) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 101] Network is unreachable ==> /var/log/ovirt-hosted-engine-ha/agent.log <== MainThread::INFO::2016-06-23 18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed set the storage domain: 'Failed to set storage domain VdsmBackend, options {'hosted-engine.lockspace': '7B22696D6167655F75756964223A202265663131373139322D623564662D346534362D383939622D6262663862663135222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202230613363393433652D633032392D343134372D623364342D396366353364663161356262227D', 'sp_uuid': '0