Re: [ovirt-users] Hosted engine install failed; vdsm upset about broker

[knarra]

On 04/21/2017 06:34 PM, Jamie Lawrence wrote:
On Apr 20, 2017, at 10:36 PM, knarra <kna...@redhat.com> wrote:
The installer claimed it did, but I believe it didn’t. Below the error from my 
original email, there’s the below (apologies for not including it earlier; I 
missed it). Note: 04ff4cf1-135a-4918-9a1f-8023322f89a3 is the HE - I’m pretty 
sure it is complaining about itself. (In any case, I verified that there are no 
other VMs running with both virsh and vdsClient.)
^^^
2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:128 Stage 
late_setup METHOD otopi.plugins.gr_he_setup.vm.runvm.Plugin._late_setup
2017-04-19 12:27:02 DEBUG otopi.plugins.gr_he_setup.vm.runvm 
runvm._late_setup:83 {'status': {'message': 'Done', 'code': 0}, 'items': 
[u'04ff4cf1-135a-4918-9a1f-8023322f89a3']}
2017-04-19 12:27:02 ERROR otopi.plugins.gr_he_setup.vm.runvm 
runvm._late_setup:91 The following VMs have been found: 
04ff4cf1-135a-4918-9a1f-8023322f89a3
2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:142 method 
exception
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in 
_executeMethod
     method['method']()
   File 
"/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/vm/runvm.py",
 line 95, in _late_setup
     _('Cannot setup Hosted Engine with other VMs running')
RuntimeError: Cannot setup Hosted Engine with other VMs running
2017-04-19 12:27:02 ERROR otopi.context context._executeMethod:151 Failed to 
execute stage 'Environment setup': Cannot setup Hosted Engine with other VMs 
running
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT 
DUMP - BEGIN
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV 
BASE/error=bool:'True'
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV 
BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Cannot 
setup Hosted Engine with other VMs running',), <traceback object at 0x4358098>)]'
2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT 
DUMP - END
James, generally this issue happens when the setup failed once and you tried re 
running it again.  Can you clean it and deploy it again?  HE should come up 
successfully. Below are the steps for cleaning it up.
Knarra,

I realize that. However, that is not the situation in my case. See above, at 
the mark - the UUID it is complaining about is the UUID of the hosted-engine it 
just installed. From the answers file generated from the run (whole thing 
below):

OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
Also see the WARNs I mentioned previously, quoted below. Excerpt:

Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: 
/var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm
 already removed
Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: 
/var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0
 already removed
Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm 
ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to 
broker, the number of errors has exceeded the limit (1)
I’m not clear on what it is attempting to do there, but it seems relevant.
I remember that you said HE vm was not started when the installation was 
successful. Is Local Maintenance enabled on that host?

can you please check if the services 'ovirt-ha-agent' and 
'ovirt-ha-broker' running fine and try to restart them once ?



I know there is no failed install left on the gluster volume, because when I 
attempt an install, part of my scripted prep process is deleting and recreating 
the Gluster volume. The below instructions are more or less what I’m doing 
already in a script[1]. (the gluster portion of the script process is: stop the 
volume, delete the volume, remove the mount point directory to avoid Gluster’s 
xattr problem with recycling directories, recreate the directory, change perms, 
create the volume, start the volume, set Ovirt-recc’ed volume options.)

-j

[1] We have a requirement for automated setup of all production resources, so 
all of this ends up being scripted.

1) vdsClient -s 0 list table | awk '{print $1}' | xargs vdsClient -s 0 destroy

2) stop the volume and delete all the information inside the bricks from all 
the hosts

3) try to umount storage from /rhev/data-center/mnt/ - umount -f 
/rhev/data-center/mnt/  if it is mounted

4) remove all dirs from /rhev/data-center/mnt/ - rm -rf /rhev/data-center/mnt/*

5) start  volume again and start the deployment.

Thanks
kasturi

If I start it manually, the default DC is down, the default cluster has the 
installation host in the cluster,  there is no storage, and the VM doesn’t show 
up in the GUI. In this install run, I have not yet started the engine manually.
you wont be seeing HE vm until HE storage is imported into the UI. HE storage 
will be automatically imported into the UI (which will import HE vm too )once a 
master domain is present .
Sure; I’m just attempting to provide context.

I assume this is related to the errors in ovirt-hosted-engine-setup.log, below. 
(The timestamps are confusing; it looks like the Python errors are logged some 
time after they’re captured or something.) The HA broker and agent logs just 
show them looping in the sequence below.

Is there a decent way to pick this up and continue? If not, how do I make this 
work?
Can you please check the following things.

1) is glusterd running on all the nodes ? 'systemctl status glistered’
2) Are you able to connect to your storage server which is ovirt_engine in your 
case.
3) Can you check if all the brick process in the volume is up ?
1) Verified that glusterd is running on all three nodes.

2)
[root@sc5-thing-1]# mount -tglusterfs sc5-gluster-1:/ovirt_engine 
/mnt/ovirt_engine
[root@sc5-thing-1]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
[…]
sc5-gluster-1:/ovirt_engine         300G  2.6G  298G   1% /mnt/ovirt_engine


3)
[root@sc5-gluster-1 jlawrence]# gluster volume status
Status of volume: ovirt_engine
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick sc5-gluster-1:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1                        49217     0          Y       22102
Brick sc5-gluster-2:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1                        49157     0          Y       37842
Brick sc5-gluster-3:/gluster-bricks/ovirt_e
ngine/ovirt_engine-1                        49157     0          Y       112018
Self-heal Daemon on localhost               N/A       N/A        Y       113364
Self-heal Daemon on sc5-gluster-2.squaretra
de.com                                      N/A       N/A        Y       19344
Self-heal Daemon on sc5-gluster-3.squaretra
de.com                                      N/A       N/A        Y       37956

Task Status of Volume ovirt_engine
------------------------------------------------------------------------------
There are no active volume tasks

Volume ovirt_images is not started




Thanks
kasturi.
Thank you!

-j


Thanks,

-j

- - - - ovirt-hosted-engine-setup.log snippet: - - - -

2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage 
late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_setup
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd 
systemd.status:90 check service vdsmd status
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd 
plugin.executeRaw:813 execute: ('/bin/systemctl', 'status', 'vdsmd.service'), 
executable='None', cwd='None', env=None
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd 
plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status', 
'vdsmd.service'), rc=0
2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd 
plugin.execute:921 execute-output: ('/bin/systemctl', 'status', 
'vdsmd.service') stdout:
● vdsmd.service - Virtual Desktop Server Manager
    Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor 
preset: enabled)
    Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago
   Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh 
--post-stop (code=exited, status=0/SUCCESS)
   Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)
  Main PID: 70062 (vdsm)
    CGroup: /system.slice/vdsmd.service
            └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm

Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm 
ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to 
broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed 
to retrieve Hosted Engine HA info
                                                          Traceback (most 
recent call last):
                                                            File 
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
                                                              stats = 
instance.get_all_stats()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", 
line 102, in get_all_stats
                                                              with 
broker.connection(self._retries, self._wait):
                                                            File 
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
                                                              return 
self.gen.next()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 99, in connection
                                                              
self.connect(retries, wait)
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 78, in connect
                                                              raise 
BrokerConnectionError(error_msg)
                                                          
BrokerConnectionError: Failed to connect to broker, the number of errors has 
exceeded the limit (1)
Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm 
ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to 
broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed 
to retrieve Hosted Engine HA info
                                                          Traceback (most 
recent call last):
                                                            File 
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
                                                              stats = 
instance.get_all_stats()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", 
line 102, in get_all_stats
                                                              with 
broker.connection(self._retries, self._wait):
                                                            File 
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
                                                              return 
self.gen.next()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 99, in connection
                                                              
self.connect(retries, wait)
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 78, in connect
                                                              raise 
BrokerConnectionError(error_msg)
                                                          
BrokerConnectionError: Failed to connect to broker, the number of errors has 
exceeded the limit (1)
Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: 
/var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm
 already removed
Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN File: 
/var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0
 already removed
Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm 
ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to 
broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed 
to retrieve Hosted Engine HA info
                                                          Traceback (most 
recent call last):
                                                            File 
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
                                                              stats = 
instance.get_all_stats()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", 
line 102, in get_all_stats
                                                              with 
broker.connection(self._retries, self._wait):
                                                            File 
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
                                                              return 
self.gen.next()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 99, in connection
                                                              
self.connect(retries, wait)
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 78, in connect
                                                              raise 
BrokerConnectionError(error_msg)
                                                          
BrokerConnectionError: Failed to connect to broker, the number of errors has 
exceeded the limit (1)
Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm 
ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect to 
broker, the number of errors has exceeded the limit (1)
Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR failed 
to retrieve Hosted Engine HA info
                                                          Traceback (most 
recent call last):
                                                            File 
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
                                                              stats = 
instance.get_all_stats()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", 
line 102, in get_all_stats
                                                              with 
broker.connection(self._retries, self._wait):
                                                            File 
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
                                                              return 
self.gen.next()
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 99, in connection
                                                              
self.connect(retries, wait)
                                                            File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 78, in connect
                                                              raise 
BrokerConnectionError(error_msg)
                                                          
BrokerConnectionError: Failed to connect to broker, the number of errors has 
exceeded the limit (1)

2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd 
plugin.execute:926 execute-output: ('/bin/systemctl', 'status', 
'vdsmd.service') stderr:


2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv 
util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0}, 
u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942', 
u'systemFamily': u'PowerEdge', u'systemUUID': 
u'4C4C4544-0054-4810-8035-C7C04F393432', u'systemManufacturer': u'Dell Inc.'}
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT 
DUMP - BEGIN
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV 
OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at 0x34a8290>'
2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774 ENVIRONMENT 
DUMP - END
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage 
late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134 condition 
False
2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage 
late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki 
plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text', 
'-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None', 
cwd='None', env=None
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki 
plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout', 
'-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0
2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki 
plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout', '-text', 
'-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout:

- - - - broker.log: - - - -

Thread-2970::INFO::2017-04-19 
14:29:51,591::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load)
 System load total=0.0117, engine=0.0000, non-engine=0.0117
Thread-2976::INFO::2017-04-19 
14:29:56,347::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2976::INFO::2017-04-19 
14:29:56,361::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2977::INFO::2017-04-19 
14:29:57,628::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2977::INFO::2017-04-19 
14:29:57,629::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Starting submonitor ping
Thread-2977::INFO::2017-04-19 
14:29:57,631::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Started submonitor ping, id 140603614328464
Thread-2977::INFO::2017-04-19 
14:29:57,662::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Starting submonitor mgmt-bridge
Thread-2977::INFO::2017-04-19 
14:29:57,667::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Started submonitor mgmt-bridge, id 140603614326992
Thread-2977::INFO::2017-04-19 
14:29:57,688::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Starting submonitor mem-free
Thread-2977::INFO::2017-04-19 
14:29:57,735::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Started submonitor mem-free, id 140604548742096
Thread-2977::INFO::2017-04-19 
14:29:57,756::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Starting submonitor cpu-load-no-engine
Thread-2977::INFO::2017-04-19 
14:29:57,798::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Started submonitor cpu-load-no-engine, id 43463568
Thread-2977::INFO::2017-04-19 
14:29:57,841::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Starting submonitor engine-health
Thread-2977::INFO::2017-04-19 
14:29:57,863::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
 Started submonitor engine-health, id 140604881915728
Thread-2983::INFO::2017-04-19 
14:29:58,153::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2984::INFO::2017-04-19 
14:30:02,179::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2977::INFO::2017-04-19 
14:30:02,249::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
 Cleaning up stale LV link 
'/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.lockspace'
Thread-2977::INFO::2017-04-19 
14:30:02,321::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
 Cleaning up stale LV link 
'/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.metadata'
Thread-2983::INFO::2017-04-19 
14:30:02,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2984::INFO::2017-04-19 
14:30:02,340::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2969::INFO::2017-04-19 
14:30:02,343::mem_free::50::mem_free.MemFree::(action) memFree: 254533
Thread-2980::INFO::2017-04-19 
14:30:02,382::mem_free::50::mem_free.MemFree::(action) memFree: 254530
Thread-2985::INFO::2017-04-19 
14:30:07,991::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2985::INFO::2017-04-19 
14:30:08,082::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2969::INFO::2017-04-19 
14:30:08,129::mem_free::50::mem_free.MemFree::(action) memFree: 254534
Thread-2966::INFO::2017-04-19 
14:30:09,719::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2966::INFO::2017-04-19 
14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopping submonitor ping, id 140603546975120
Thread-2966::INFO::2017-04-19 
14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopped submonitor ping, id 140603546975120
Thread-2966::INFO::2017-04-19 
14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopping submonitor mgmt-bridge, id 140603546973136
Thread-2966::INFO::2017-04-19 
14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopped submonitor mgmt-bridge, id 140603546973136
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopping submonitor mem-free, id 140603546973264
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopped submonitor mem-free, id 140603546973264
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopping submonitor cpu-load-no-engine, id 140603546976144
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopped submonitor cpu-load-no-engine, id 140603546976144
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopping submonitor engine-health, id 140603546974096
Thread-2966::INFO::2017-04-19 
14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
 Stopped submonitor engine-health, id 140603546974096
Thread-2979::INFO::2017-04-19 
14:30:11,145::mgmt_bridge::55::mgmt_bridge.MgmtBridge::(action) Found bridge 
ovirtmgmt with ports
Thread-2986::INFO::2017-04-19 
14:30:11,462::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2986::INFO::2017-04-19 
14:30:11,480::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2987::INFO::2017-04-19 
14:30:11,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
 Connection established
Thread-2987::INFO::2017-04-19 
14:30:11,488::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Connection closed
Thread-2980::INFO::2017-04-19 
14:30:11,502::mem_free::50::mem_free.MemFree::(action) memFree: 254535

- - - - ha agent.log: - - - -

MainThread::WARNING::2017-04-19 
14:31:16,352::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
 Restarting agent, attempt '6'
MainThread::INFO::2017-04-19 
14:31:16,383::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
 Found certificate common name: sc5-ovirt-2.squaretrade.com
MainThread::INFO::2017-04-19 
14:31:16,385::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
 Initializing VDSM
MainThread::INFO::2017-04-19 
14:31:18,863::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
 Connecting the storage
MainThread::INFO::2017-04-19 
14:31:18,864::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
 Connecting storage server
MainThread::INFO::2017-04-19 
14:31:23,672::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
 Connecting storage server
MainThread::INFO::2017-04-19 
14:31:23,689::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
 Refreshing the storage domain
MainThread::INFO::2017-04-19 
14:31:23,956::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
 Preparing images
MainThread::INFO::2017-04-19 
14:31:23,957::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
 Preparing images
MainThread::INFO::2017-04-19 
14:31:26,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
 Refreshing vm.conf
MainThread::INFO::2017-04-19 
14:31:26,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
 Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-04-19 
14:31:26,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
 Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::WARNING::2017-04-19 
14:31:29,548::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
 Unable to find OVF_STORE
MainThread::ERROR::2017-04-19 
14:31:29,551::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
 Unable to identify the OVF_STORE volume, falling back to initial vm.conf. 
Please ensure you already added your first data domain for regular VMs
MainThread::INFO::2017-04-19 
14:31:29,576::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Initializing ha-broker connection
MainThread::INFO::2017-04-19 
14:31:29,577::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor ping, options {'addr': '10.181.26.1'}
MainThread::INFO::2017-04-19 
14:31:29,596::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Success, id 140604881712208
MainThread::INFO::2017-04-19 
14:31:29,596::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 
'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-04-19 
14:31:29,600::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Success, id 140604881712016
MainThread::INFO::2017-04-19 
14:31:29,600::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-04-19 
14:31:29,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Success, id 140604881711568
MainThread::INFO::2017-04-19 
14:31:29,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': 
'04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 
14:31:29,605::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Success, id 140604881711312
MainThread::INFO::2017-04-19 
14:31:29,605::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': 
'04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
MainThread::INFO::2017-04-19 
14:31:29,608::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Success, id 140604881714896
MainThread::INFO::2017-04-19 
14:31:34,326::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain)
 Success, id 140604881713168
MainThread::INFO::2017-04-19 
14:31:34,327::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Broker initialized, all submonitors started
MainThread::ERROR::2017-04-19 
14:31:34,372::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
 Service sanlock is not running and the admin is responsible for starting it. 
Waiting…


- - - - answers.conf - - - -

[environment:default]
OVEHOSTED_CORE/rollbackProceed=none:None
OVEHOSTED_CORE/screenProceed=none:None
OVEHOSTED_CORE/deployProceed=bool:True
OVEHOSTED_CORE/upgradeProceed=none:None
OVEHOSTED_CORE/confirmSettings=bool:True
OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
OVEHOSTED_NETWORK/firewallManager=none:None
OVEHOSTED_NETWORK/gateway=str:10.181.26.1
OVEHOSTED_ENGINE/insecureSSL=none:None
OVEHOSTED_ENGINE/clusterName=str:Default
OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
OVEHOSTED_STORAGE/domainType=str:glusterfs
OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311
OVEHOSTED_STORAGE/LunID=none:None
OVEHOSTED_STORAGE/imgSizeGB=str:50
OVEHOSTED_STORAGE/mntOptions=none:None
OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-b95698e58b18
OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab
OVEHOSTED_STORAGE/iSCSITargetName=none:None
OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-39770fb2e821
OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-5c81f735e43b
OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6
OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc
OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000
OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-d4f0e46e150f
OVEHOSTED_ENGINE/enableHcGlusterService=bool:False
OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
OVEHOSTED_STORAGE/iSCSIPortal=none:None
OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d
OVEHOSTED_STORAGE/vgUUID=none:None
OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7
OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine
OVEHOSTED_STORAGE/iSCSIPortalUser=none:None
OVEHOSTED_VDSM/consoleType=str:qxl
OVEHOSTED_VM/vmMemSizeMB=int:16384
OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2
OVEHOSTED_VM/emulatedMachine=str:pc
OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78
OVEHOSTED_VM/vmVCpus=str:4
OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f
OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22
OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova
OVEHOSTED_VM/vmCDRom=none:None
OVEHOSTED_VM/automateVMShutdown=bool:True
OVEHOSTED_VM/cloudInitISO=str:generate
OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com
OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.squaretrade.com
OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip]
OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip]
OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles
OVEHOSTED_VM/rootSshAccess=str:yes
OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False
OVEHOSTED_VM/cloudinitVMDNS=[snip]
OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, 
CN=sc5-ovirt-2.squaretrade.com
OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX
OVEHOSTED_NOTIF/smtpPort=str:25
OVEHOSTED_NOTIF/smtpServer=[snip]
OVEHOSTED_NOTIF/sourceEmail=[snip]
OVEHOSTED_NOTIF/destEmail=[snip]
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users