Hello,
I'm running into some weird issues with vdsm and my storage domains
after a reboot or a shutdown. I can't manage to figure out what's
going on...
Currently, my cluster (4.0.5 with hosted engine) is composed of one
main node. (and another inactive one but unrelated to this issue).
It has local storage exposed to oVirt via 3 NFS exports (one specific
for the hosted engine vm) reachable from my local network.
When I wan't to shutdown or reboot my main host (and so the whole
cluster), I use a custom script :
1. Shutdown all VM
2. Shutdown engine VM
3. Stop HA agent and broker
4. Stop vdsmd
5. Release the sanlock on the hosted engine SD
6. Shutdown / Reboot
It works just fine, but at the next boot, VDSM takes at least 10-15
minutes to find storage domains, except the hosted engine one. The
engine loops trying to reconstruct the SPM.
During this time, vdsClient getConnectedStoragePoolsList returns nothing.
getStorageDomainsList returns only the hosted engine domain.
NFS exports are mountable from another server.
But when I restart vdsm manually after the boot, it seems to detect
immediately the storage domains.
Is there some kind of staled storage data used by vdsm and a timeout
to invalidate them ?
Am I missing something on the vdsm side in my shutdown procedure ?
Thanks !
Engine and vdsm logs are attached.
--
Yoann Laissus
---------------------------------------------------------------------------------------------------------------------------------
engine :
2016-12-03 15:58:53,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value '
TaskStatusListReturnForXmlRpc:{status='StatusForXmlRpc [code=654, message=Not SPM: ()]'}
'
2016-12-03 15:58:53,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] HostName = ovirt-host1
2016-12-03 15:58:53,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c7d02baa-7c47-484e-8146-ba01b36b49d6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2016-12-03 15:58:53,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] FINISH, SpmStopVDSCommand, log id: 5f207339
2016-12-03 15:58:53,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] FINISH, ResetIrsVDSCommand, log id: dbff776
2016-12-03 15:58:53,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] START, DisconnectStoragePoolVDSCommand(HostName = ovirt-host1, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='c7d02baa-7c47-484e-8146-ba01b36b49d6', storagePoolId='58317bfd-02fd-030e-01a3-000000000181', vds_spm_id='1'}), log id: 26e25174
2016-12-03 15:58:54,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] FINISH, DisconnectStoragePoolVDSCommand, log id: 26e25174
2016-12-03 15:58:54,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] START, ReconstructMasterVDSCommand(HostName = ovirt-host1, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='c7d02baa-7c47-484e-8146-ba01b36b49d6', vdsSpmId='1', storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storagePoolName='Default', masterDomainId='a403cebb-243d-41f0-83c1-25cb9f911f30', masterVersion='698', domainsList='[StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='02d9ee81-6db1-494b-b32b-d54236717aa0'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='e1140160-8578-436f-a69b-6fee6cc7c894'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='a403cebb-243d-41f0-83c1-25cb9f911f30'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='47bec5f5-e44f-4cd3-9249-38266a5f2ab9'}', status='Inactive'}]'}), log id: 606cb02b
2016-12-03 15:58:55,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Failed in 'ReconstructMasterVDS' method
2016-12-03 15:58:55,670 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM ovirt-host1 command failed: Storage domain does not exist: (u'a403cebb-243d-41f0-83c1-25cb9f911f30',)
2016-12-03 15:58:55,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=358, message=Storage domain does not exist: (u'a403cebb-243d-41f0-83c1-25cb9f911f30',)]]'
2016-12-03 15:58:55,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] HostName = ovirt-host1
2016-12-03 15:58:55,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command 'ReconstructMasterVDSCommand(HostName = ovirt-host1, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='c7d02baa-7c47-484e-8146-ba01b36b49d6', vdsSpmId='1', storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storagePoolName='Default', masterDomainId='a403cebb-243d-41f0-83c1-25cb9f911f30', masterVersion='698', domainsList='[StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='02d9ee81-6db1-494b-b32b-d54236717aa0'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='e1140160-8578-436f-a69b-6fee6cc7c894'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='a403cebb-243d-41f0-83c1-25cb9f911f30'}', status='Unknown'}, StoragePoolIsoMap:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='47bec5f5-e44f-4cd3-9249-38266a5f2ab9'}', status='Inactive'}]'})' execution failed: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: (u'a403cebb-243d-41f0-83c1-25cb9f911f30',), code = 358
2016-12-03 15:58:55,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] FINISH, ReconstructMasterVDSCommand, log id: 606cb02b
2016-12-03 15:58:55,671 ERROR [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command 'org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: (u'a403cebb-243d-41f0-83c1-25cb9f911f30',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
2016-12-03 15:58:55,679 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command [id=128f2e84-8ca5-4fad-82f3-010257d5f21a]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='StoragePoolIsoMapId:{storagePoolId='58317bfd-02fd-030e-01a3-000000000181', storageId='47bec5f5-e44f-4cd3-9249-38266a5f2ab9'}', status='Unknown'}.
2016-12-03 15:58:55,682 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command [id=128f2e84-8ca5-4fad-82f3-010257d5f21a]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic:{name='SSD_shared', id='47bec5f5-e44f-4cd3-9249-38266a5f2ab9'}.
2016-12-03 15:58:55,683 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Command [id=128f2e84-8ca5-4fad-82f3-010257d5f21a]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic:{name='Reds_shared', id='a403cebb-243d-41f0-83c1-25cb9f911f30'}.
2016-12-03 15:58:55,698 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [82a36a5] Correlation ID: 82a36a5, Job ID: 0f1efa4e-e40e-46e8-ac92-8129f73666ee, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
2016-12-03 15:58:55,707 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-4) [4a8b07a3] Finished reconstruct for pool '58317bfd-02fd-030e-01a3-000000000181'. Clearing event queue
---------------------------------------------------------------------------------------------------------------------------------
vdsm :
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,321::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`96520959-a711-4054-b62a-a1849eba9869`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '986' at 'connectStoragePool'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,321::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,321::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,321::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`96520959-a711-4054-b62a-a1849eba9869`::Granted request
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,321::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.58317bfd-02fd-030e-01a3-000000000181`ReqID=`1c6a1c52-3748-48a3-b245-d604699c90d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1034' at '_connectStoragePool'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.58317bfd-02fd-030e-01a3-000000000181' for lock type 'exclusive'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.58317bfd-02fd-030e-01a3-000000000181' is free. Now locking as 'exclusive' (1 active user)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.58317bfd-02fd-030e-01a3-000000000181`ReqID=`1c6a1c52-3748-48a3-b245-d604699c90d7`::Granted request
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::spbackends::499::Storage.StoragePoolMemoryBackend::(updateVersionAndDomains) updating domain version to 697 and domains map to {u'a403cebb-243d-41f0-83c1-25cb9f911f30': u'active', u'47bec5f5-e44f-4cd3-9249-38266a5f2ab9': u'active', u'02d9ee81-6db1-494b-b32b-d54236717aa0': u'active', u'e1140160-8578-436f-a69b-6fee6cc7c894': u'active'}
jsonrpc.Executor/7::INFO::2016-12-03 15:58:51,322::spbackends::447::Storage.StoragePoolMemoryBackend::(setDomainsMap) new storage pool master version 697 and domains map {u'02d9ee81-6db1-494b-b32b-d54236717aa0': u'Active', u'47bec5f5-e44f-4cd3-9249-38266a5f2ab9': u'Active', u'a403cebb-243d-41f0-83c1-25cb9f911f30': u'Active', u'e1140160-8578-436f-a69b-6fee6cc7c894': u'Active'}
jsonrpc.Executor/7::INFO::2016-12-03 15:58:51,322::sp::131::Storage.StoragePool::(setBackend) updating pool 58317bfd-02fd-030e-01a3-000000000181 backend from type NoneType instance 0x7f64cde5bf00 to type StoragePoolMemoryBackend instance 0x29d54c8
jsonrpc.Executor/7::INFO::2016-12-03 15:58:51,322::sp::634::Storage.StoragePool::(connect) Connect host #1 to the storage pool 58317bfd-02fd-030e-01a3-000000000181 with master domain: 47bec5f5-e44f-4cd3-9249-38266a5f2ab9 (ver = 697)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::sp::656::Storage.StoragePool::(_startWatchingDomainsState) Start watching domains state
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,322::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,323::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,323::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,323::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R (cwd None)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,335::misc::566::Storage.SamplingMethod::(__call__) Returning last result
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,336::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (vdsm.storage.hba.rescan)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,336::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,336::hba::58::Storage.HBA::(rescan) Starting scan
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,383::hba::64::Storage.HBA::(rescan) Scan finished
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,383::misc::566::Storage.SamplingMethod::(__call__) Returning last result
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,384::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-15 /sbin/udevadm settle --timeout=5 (cwd None)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,395::commands::86::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,396::misc::566::Storage.SamplingMethod::(__call__) Returning last result
jsonrpc.Executor/7::ERROR::2016-12-03 15:58:51,396::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 47bec5f5-e44f-4cd3-9249-38266a5f2ab9
jsonrpc.Executor/7::ERROR::2016-12-03 15:58:51,396::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 47bec5f5-e44f-4cd3-9249-38266a5f2ab9
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,397::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 47bec5f5-e44f-4cd3-9249-38266a5f2ab9 (cwd None)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,422::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "47bec5f5-e44f-4cd3-9249-38266a5f2ab9" not found\n Cannot process volume group 47bec5f5-e44f-4cd3-9249-38266a5f2ab9\n WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 5
jsonrpc.Executor/7::WARNING::2016-12-03 15:58:51,422::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "47bec5f5-e44f-4cd3-9249-38266a5f2ab9" not found', ' Cannot process volume group 47bec5f5-e44f-4cd3-9249-38266a5f2ab9', ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!']
jsonrpc.Executor/7::ERROR::2016-12-03 15:58:51,426::sdc::146::Storage.StorageDomainCache::(_findDomain) domain 47bec5f5-e44f-4cd3-9249-38266a5f2ab9 not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'47bec5f5-e44f-4cd3-9249-38266a5f2ab9',)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::sp::662::Storage.StoragePool::(_stopWatchingDomainsState) Stop watching domains state
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.58317bfd-02fd-030e-01a3-000000000181'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.58317bfd-02fd-030e-01a3-000000000181' (0 active users)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.58317bfd-02fd-030e-01a3-000000000181' is free, finding out if anyone is waiting for it.
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.58317bfd-02fd-030e-01a3-000000000181', Clearing records.
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,427::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
jsonrpc.Executor/7::ERROR::2016-12-03 15:58:51,427::task::868::Storage.TaskManager.Task::(_setError) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 875, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 988, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1053, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 646, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
self.setMasterDomain(msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1430, in setMasterDomain
raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: u'spUUID=58317bfd-02fd-030e-01a3-000000000181, msdUUID=47bec5f5-e44f-4cd3-9249-38266a5f2ab9'
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::887::Storage.TaskManager.Task::(_run) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::Task._run: 45b7c7bc-3c85-46bb-abc6-09b993663a94 (u'58317bfd-02fd-030e-01a3-000000000181', 1, u'47bec5f5-e44f-4cd3-9249-38266a5f2ab9', 697, {u'a403cebb-243d-41f0-83c1-25cb9f911f30': u'active', u'47bec5f5-e44f-4cd3-9249-38266a5f2ab9': u'active', u'02d9ee81-6db1-494b-b32b-d54236717aa0': u'active', u'e1140160-8578-436f-a69b-6fee6cc7c894': u'active'}) {} failed - stopping task
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::1248::Storage.TaskManager.Task::(stop) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::stopping in state preparing (force False)
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::995::Storage.TaskManager.Task::(_decref) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::ref 1 aborting True
jsonrpc.Executor/7::INFO::2016-12-03 15:58:51,428::task::1173::Storage.TaskManager.Task::(prepare) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::aborting: Task is aborted: 'Cannot find master domain' - code 304
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::1178::Storage.TaskManager.Task::(prepare) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::Prepare: aborted: Cannot find master domain
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::995::Storage.TaskManager.Task::(_decref) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::ref 0 aborting True
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,428::task::930::Storage.TaskManager.Task::(_doAbort) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::Task._doAbort: force False
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::task::597::Storage.TaskManager.Task::(_updateState) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::moving from state preparing -> state aborting
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::_aborting: recover policy none
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::task::597::Storage.TaskManager.Task::(_updateState) Task=`45b7c7bc-3c85-46bb-abc6-09b993663a94`::moving from state aborting -> state failed
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
jsonrpc.Executor/7::DEBUG::2016-12-03 15:58:51,429::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
jsonrpc.Executor/7::ERROR::2016-12-03 15:58:51,429::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: u'spUUID=58317bfd-02fd-030e-01a3-000000000181, msdUUID=47bec5f5-e44f-4cd3-9249-38266a5f2ab9'", 'code': 304}}
jsonrpc.Executor/7::INFO::2016-12-03 15:58:51,429::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.connect failed (error 304) in 0.11 seconds
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
