On Thu, Apr 20, 2017 at 2:14 AM, Jamie Lawrence <jlawre...@squaretrade.com> wrote:
> > So, tracing this further, I’m pretty sure this is something about sanlock. > > As best I can tell this[1] seems to be the failure that is blocking > importing the pool, creating storage domains, importing the HE, etc. > Contrary to the log, sanlock is running; I verified it starts on > system-boot and restarts just fine. > > I found one reference to someone having a similar problem in 3.6, but that > appeared to have been a permission issue I’m not afflicted with. > > How can I move past this? > Could you please share the output of sudo -u vdsm sudo service sanlock status ? > > TIA, > > -j > > > [1] agent.log: > MainThread::WARNING::2017-04-19 17:07:13,537::agent::209:: > ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, > attempt '6' > MainThread::INFO::2017-04-19 17:07:13,567::hosted_engine:: > 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > Found certificate common name: sc5-ovirt-2.squaretrade.com > MainThread::INFO::2017-04-19 17:07:13,569::hosted_engine:: > 604::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_vdsm) Initializing VDSM > MainThread::INFO::2017-04-19 17:07:16,044::hosted_engine:: > 630::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Connecting the storage > MainThread::INFO::2017-04-19 17:07:16,045::storage_server:: > 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > MainThread::INFO::2017-04-19 17:07:20,876::storage_server:: > 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > MainThread::INFO::2017-04-19 17:07:20,893::storage_server:: > 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Refreshing the storage domain > MainThread::INFO::2017-04-19 17:07:21,160::hosted_engine:: > 657::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Preparing images > MainThread::INFO::2017-04-19 17:07:21,160::image::126:: > ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images > MainThread::INFO::2017-04-19 17:07:23,954::hosted_engine:: > 660::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Refreshing vm.conf > MainThread::INFO::2017-04-19 17:07:23,955::config::485:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) > Reloading vm.conf from the shared storage domain > MainThread::INFO::2017-04-19 17:07:23,955::config::412:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine. > config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher > copy of vm configuration from the OVF_STORE > MainThread::WARNING::2017-04-19 17:07:26,741::ovf_store::107:: > ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find > OVF_STORE > MainThread::ERROR::2017-04-19 17:07:26,744::config::450:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine. > config::(_get_vm_conf_content_from_ovf_store) Unable to identify the > OVF_STORE volume, falling back to initial vm.conf. Please ensure you > already added your first data domain for regular VMs > MainThread::INFO::2017-04-19 17:07:26,770::hosted_engine:: > 509::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_broker) Initializing ha-broker connection > MainThread::INFO::2017-04-19 17:07:26,771::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor ping, options {'addr': '10.181.26.1'} > MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140621269798096 > MainThread::INFO::2017-04-19 17:07:26,774::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': > 'ovirtmgmt', 'address': '0'} > MainThread::INFO::2017-04-19 17:07:26,791::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140621269798544 > MainThread::INFO::2017-04-19 17:07:26,792::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} > MainThread::INFO::2017-04-19 17:07:26,793::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140621269798224 > MainThread::INFO::2017-04-19 17:07:26,794::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': > '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} > MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140621269796816 > MainThread::INFO::2017-04-19 17:07:26,796::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': > '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} > MainThread::INFO::2017-04-19 17:07:26,798::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140621269797840 > MainThread::INFO::2017-04-19 17:07:31,513::brokerlink::179: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) > Success, id 140621269798480 > MainThread::INFO::2017-04-19 17:07:31,513::hosted_engine:: > 601::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_broker) Broker initialized, all submonitors > started > MainThread::ERROR::2017-04-19 17:07:31,556::agent::196:: > ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is > not running and the admin is responsible for starting it. Waiting... > MainThread::WARNING::2017-04-19 17:07:36,562::agent::209:: > ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, > attempt '7' > MainThread::INFO::2017-04-19 17:07:36,592::hosted_engine:: > 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > Found certificate common name: sc5-ovirt-2.squaretrade.com > > > > > - - - prior message below - - - > > I trialed installing the hosted engine, following the instructions at > http://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_ > Engine/ . This is using Gluster as the backend storage subsystem. > > Answer file at the end. > > Per the docs, > > "When the hosted-engine deployment script completes successfully, the > oVirt Engine is configured and running on your host. The Engine has already > configured the data center, cluster, host, the Engine virtual machine, and > a shared storage domain dedicated to the Engine virtual machine.” > > In my case, this is false. The installation claims success, but the > hosted engine VM stays stopped, unless I start it manually. If I start it > manually, the default DC is down, the default cluster has the installation > host in the cluster, there is no storage, and the VM doesn’t show up in > the GUI. In this install run, I have not yet started the engine manually. > > I assume this is related to the errors in ovirt-hosted-engine-setup.log, > below. (The timestamps are confusing; it looks like the Python errors are > logged some time after they’re captured or something.) The HA broker and > agent logs just show them looping in the sequence below. > > Is there a decent way to pick this up and continue? If not, how do I make > this work? > > Thanks, > > -j > > - - - - ovirt-hosted-engine-setup.log snippet: - - - - > > 2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage > late_setup METHOD otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_ > setup > 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd > systemd.status:90 check service vdsmd status > 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd > plugin.executeRaw:813 execute: ('/bin/systemctl', 'status', > 'vdsmd.service'), executable='None', cwd='None', env=None > 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd > plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status', > 'vdsmd.service'), rc=0 > 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd > plugin.execute:921 execute-output: ('/bin/systemctl', 'status', > 'vdsmd.service') stdout: > ● vdsmd.service - Virtual Desktop Server Manager > Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor > preset: enabled) > Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s ago > Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh > --post-stop (code=exited, status=0/SUCCESS) > Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh > --pre-start (code=exited, status=0/SUCCESS) > Main PID: 70062 (vdsm) > CGroup: /system.slice/vdsmd.service > └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm > > Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm > ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect > to broker, the number of errors has exceeded the limit (1) > Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR > failed to retrieve Hosted Engine HA info > Traceback (most > recent call last): > File > "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in > _getHaInfo > stats = > instance.get_all_stats() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", > line 102, in get_all_stats > with > broker.connection(self._retries, self._wait): > File > "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > return > self.gen.next() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 99, in connection > > self.connect(retries, wait) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 78, in connect > raise > BrokerConnectionError(error_msg) > > BrokerConnectionError: Failed to connect to broker, the number of errors > has exceeded the limit (1) > Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm > ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect > to broker, the number of errors has exceeded the limit (1) > Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR > failed to retrieve Hosted Engine HA info > Traceback (most > recent call last): > File > "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in > _getHaInfo > stats = > instance.get_all_stats() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", > line 102, in get_all_stats > with > broker.connection(self._retries, self._wait): > File > "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > return > self.gen.next() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 99, in connection > > self.connect(retries, wait) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 78, in connect > raise > BrokerConnectionError(error_msg) > > BrokerConnectionError: Failed to connect to broker, the number of errors > has exceeded the limit (1) > Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN > File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918- > 9a1f-8023322f89a3.com.redhat.rhevm.vdsm already removed > Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN > File: /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918- > 9a1f-8023322f89a3.org.qemu.guest_agent.0 already removed > Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm > ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect > to broker, the number of errors has exceeded the limit (1) > Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR > failed to retrieve Hosted Engine HA info > Traceback (most > recent call last): > File > "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in > _getHaInfo > stats = > instance.get_all_stats() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", > line 102, in get_all_stats > with > broker.connection(self._retries, self._wait): > File > "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > return > self.gen.next() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 99, in connection > > self.connect(retries, wait) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 78, in connect > raise > BrokerConnectionError(error_msg) > > BrokerConnectionError: Failed to connect to broker, the number of errors > has exceeded the limit (1) > Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm > ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect > to broker, the number of errors has exceeded the limit (1) > Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR > failed to retrieve Hosted Engine HA info > Traceback (most > recent call last): > File > "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in > _getHaInfo > stats = > instance.get_all_stats() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", > line 102, in get_all_stats > with > broker.connection(self._retries, self._wait): > File > "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ > return > self.gen.next() > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 99, in connection > > self.connect(retries, wait) > File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 78, in connect > raise > BrokerConnectionError(error_msg) > > BrokerConnectionError: Failed to connect to broker, the number of errors > has exceeded the limit (1) > > 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd > plugin.execute:926 execute-output: ('/bin/systemctl', 'status', > 'vdsmd.service') stderr: > > > 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv > util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code': 0}, > u'systemProductName': u'PowerEdge R630', u'systemSerialNumber': u'GTH5942', > u'systemFamily': u'PowerEdge', u'systemUUID': > u'4C4C4544-0054-4810-8035-C7C04F393432', > u'systemManufacturer': u'Dell Inc.'} > 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760 > ENVIRONMENT DUMP - BEGIN > 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV > OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at > 0x34a8290>' > 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774 > ENVIRONMENT DUMP - END > 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage > late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup > 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134 > condition False > 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage > late_setup METHOD otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup > 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki > plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout', '-text', > '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), executable='None', > cwd='None', env=None > 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki > plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout', > '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0 > 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki > plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout', > '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout: > > - - - - broker.log: - - - - > > Thread-2970::INFO::2017-04-19 14:29:51,591::cpu_load_no_ > engine::113::cpu_load_no_engine.EngineHealth::(calculate_load) System > load total=0.0117, engine=0.0000, non-engine=0.0117 > Thread-2976::INFO::2017-04-19 14:29:56,347::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2976::INFO::2017-04-19 14:29:56,361::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2977::INFO::2017-04-19 14:29:57,628::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2977::INFO::2017-04-19 14:29:57,629::monitor::66:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) > Starting submonitor ping > Thread-2977::INFO::2017-04-19 14:29:57,631::monitor::79:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started > submonitor ping, id 140603614328464 > Thread-2977::INFO::2017-04-19 14:29:57,662::monitor::66:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) > Starting submonitor mgmt-bridge > Thread-2977::INFO::2017-04-19 14:29:57,667::monitor::79:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started > submonitor mgmt-bridge, id 140603614326992 > Thread-2977::INFO::2017-04-19 14:29:57,688::monitor::66:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) > Starting submonitor mem-free > Thread-2977::INFO::2017-04-19 14:29:57,735::monitor::79:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started > submonitor mem-free, id 140604548742096 > Thread-2977::INFO::2017-04-19 14:29:57,756::monitor::66:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) > Starting submonitor cpu-load-no-engine > Thread-2977::INFO::2017-04-19 14:29:57,798::monitor::79:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started > submonitor cpu-load-no-engine, id 43463568 > Thread-2977::INFO::2017-04-19 14:29:57,841::monitor::66:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) > Starting submonitor engine-health > Thread-2977::INFO::2017-04-19 14:29:57,863::monitor::79:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor) Started > submonitor engine-health, id 140604881915728 > Thread-2983::INFO::2017-04-19 14:29:58,153::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2984::INFO::2017-04-19 14:30:02,179::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2977::INFO::2017-04-19 14:30:02,249::storage_ > backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) > Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ > ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_ > agent/hosted-engine.lockspace' > Thread-2977::INFO::2017-04-19 14:30:02,321::storage_ > backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks) > Cleaning up stale LV link '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ > ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_ > agent/hosted-engine.metadata' > Thread-2983::INFO::2017-04-19 14:30:02,334::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2984::INFO::2017-04-19 14:30:02,340::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2969::INFO::2017-04-19 > 14:30:02,343::mem_free::50::mem_free.MemFree::(action) > memFree: 254533 > Thread-2980::INFO::2017-04-19 > 14:30:02,382::mem_free::50::mem_free.MemFree::(action) > memFree: 254530 > Thread-2985::INFO::2017-04-19 14:30:07,991::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2985::INFO::2017-04-19 14:30:08,082::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2969::INFO::2017-04-19 > 14:30:08,129::mem_free::50::mem_free.MemFree::(action) > memFree: 254534 > Thread-2966::INFO::2017-04-19 14:30:09,719::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping > submonitor ping, id 140603546975120 > Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped > submonitor ping, id 140603546975120 > Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::90:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping > submonitor mgmt-bridge, id 140603546973136 > Thread-2966::INFO::2017-04-19 14:30:09,720::monitor::99:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped > submonitor mgmt-bridge, id 140603546973136 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping > submonitor mem-free, id 140603546973264 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped > submonitor mem-free, id 140603546973264 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping > submonitor cpu-load-no-engine, id 140603546976144 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped > submonitor cpu-load-no-engine, id 140603546976144 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::90:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopping > submonitor engine-health, id 140603546974096 > Thread-2966::INFO::2017-04-19 14:30:09,721::monitor::99:: > ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor) Stopped > submonitor engine-health, id 140603546974096 > Thread-2979::INFO::2017-04-19 14:30:11,145::mgmt_bridge::55: > :mgmt_bridge.MgmtBridge::(action) Found bridge ovirtmgmt with ports > Thread-2986::INFO::2017-04-19 14:30:11,462::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2986::INFO::2017-04-19 14:30:11,480::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2987::INFO::2017-04-19 14:30:11,481::listener::134:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) > Connection established > Thread-2987::INFO::2017-04-19 14:30:11,488::listener::186:: > ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) > Connection closed > Thread-2980::INFO::2017-04-19 > 14:30:11,502::mem_free::50::mem_free.MemFree::(action) > memFree: 254535 > > - - - - ha agent.log: - - - - > > MainThread::WARNING::2017-04-19 14:31:16,352::agent::209:: > ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, > attempt '6' > MainThread::INFO::2017-04-19 14:31:16,383::hosted_engine:: > 242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > Found certificate common name: sc5-ovirt-2.squaretrade.com > MainThread::INFO::2017-04-19 14:31:16,385::hosted_engine:: > 604::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_vdsm) Initializing VDSM > MainThread::INFO::2017-04-19 14:31:18,863::hosted_engine:: > 630::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Connecting the storage > MainThread::INFO::2017-04-19 14:31:18,864::storage_server:: > 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > MainThread::INFO::2017-04-19 14:31:23,672::storage_server:: > 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Connecting storage server > MainThread::INFO::2017-04-19 14:31:23,689::storage_server:: > 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) > Refreshing the storage domain > MainThread::INFO::2017-04-19 14:31:23,956::hosted_engine:: > 657::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Preparing images > MainThread::INFO::2017-04-19 14:31:23,957::image::126:: > ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images > MainThread::INFO::2017-04-19 14:31:26,768::hosted_engine:: > 660::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_storage_images) Refreshing vm.conf > MainThread::INFO::2017-04-19 14:31:26,769::config::485:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) > Reloading vm.conf from the shared storage domain > MainThread::INFO::2017-04-19 14:31:26,769::config::412:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine. > config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher > copy of vm configuration from the OVF_STORE > MainThread::WARNING::2017-04-19 14:31:29,548::ovf_store::107:: > ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find > OVF_STORE > MainThread::ERROR::2017-04-19 14:31:29,551::config::450:: > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine. > config::(_get_vm_conf_content_from_ovf_store) Unable to identify the > OVF_STORE volume, falling back to initial vm.conf. Please ensure you > already added your first data domain for regular VMs > MainThread::INFO::2017-04-19 14:31:29,576::hosted_engine:: > 509::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_broker) Initializing ha-broker connection > MainThread::INFO::2017-04-19 14:31:29,577::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor ping, options {'addr': '10.181.26.1'} > MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140604881712208 > MainThread::INFO::2017-04-19 14:31:29,596::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': > 'ovirtmgmt', 'address': '0'} > MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140604881712016 > MainThread::INFO::2017-04-19 14:31:29,600::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} > MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140604881711568 > MainThread::INFO::2017-04-19 14:31:29,602::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': > '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} > MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140604881711312 > MainThread::INFO::2017-04-19 14:31:29,605::brokerlink::130: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': > '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'} > MainThread::INFO::2017-04-19 14:31:29,608::brokerlink::141: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) > Success, id 140604881714896 > MainThread::INFO::2017-04-19 14:31:34,326::brokerlink::179: > :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) > Success, id 140604881713168 > MainThread::INFO::2017-04-19 14:31:34,327::hosted_engine:: > 601::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(_initialize_broker) Broker initialized, all submonitors > started > MainThread::ERROR::2017-04-19 14:31:34,372::agent::196:: > ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Service sanlock is > not running and the admin is responsible for starting it. Waiting… > > > - - - - answers.conf - - - - > > [environment:default] > OVEHOSTED_CORE/rollbackProceed=none:None > OVEHOSTED_CORE/screenProceed=none:None > OVEHOSTED_CORE/deployProceed=bool:True > OVEHOSTED_CORE/upgradeProceed=none:None > OVEHOSTED_CORE/confirmSettings=bool:True > OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com > OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt > OVEHOSTED_NETWORK/firewallManager=none:None > OVEHOSTED_NETWORK/gateway=str:10.181.26.1 > OVEHOSTED_ENGINE/insecureSSL=none:None > OVEHOSTED_ENGINE/clusterName=str:Default > OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter > OVEHOSTED_STORAGE/domainType=str:glusterfs > OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311 > OVEHOSTED_STORAGE/LunID=none:None > OVEHOSTED_STORAGE/imgSizeGB=str:50 > OVEHOSTED_STORAGE/mntOptions=none:None > OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None > OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940- > b95698e58b18 > OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab > OVEHOSTED_STORAGE/iSCSITargetName=none:None > OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8- > 39770fb2e821 > OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e- > 5c81f735e43b > OVEHOSTED_STORAGE/iSCSIPortalPort=none:None > OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6 > OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc > OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000 > OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091- > d4f0e46e150f > OVEHOSTED_ENGINE/enableHcGlusterService=bool:False > OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage > OVEHOSTED_STORAGE/iSCSIPortal=none:None > OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d > OVEHOSTED_STORAGE/vgUUID=none:None > OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7 > OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine > OVEHOSTED_STORAGE/iSCSIPortalUser=none:None > OVEHOSTED_VDSM/consoleType=str:qxl > OVEHOSTED_VM/vmMemSizeMB=int:16384 > OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3 > OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2 > OVEHOSTED_VM/emulatedMachine=str:pc > OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78 > OVEHOSTED_VM/vmVCpus=str:4 > OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f > OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22 > OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine- > appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova > OVEHOSTED_VM/vmCDRom=none:None > OVEHOSTED_VM/automateVMShutdown=bool:True > OVEHOSTED_VM/cloudInitISO=str:generate > OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com > OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1. > squaretrade.com > OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip] > OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True > OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip] > OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles > OVEHOSTED_VM/rootSshAccess=str:yes > OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False > OVEHOSTED_VM/cloudinitVMDNS=[snip] > OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com, CN= > sc5-ovirt-2.squaretrade.com > OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test > OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA > OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX > OVEHOSTED_NOTIF/smtpPort=str:25 > OVEHOSTED_NOTIF/smtpServer=[snip] > OVEHOSTED_NOTIF/sourceEmail=[snip] > OVEHOSTED_NOTIF/destEmail=[snip] > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users