The issue was found. We are not looking for a directory with the exact name of the Storage Domain UUID, but we look for ALL directories that contain the Storage Domain UUID.
For example: [root@ovirt3 gluster1:_data__fast4]# ll общо 1 drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484 drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484-NEW [root@ovirt3 gluster1:_data__fast]# ll общо 1 drwxrwxrwx. 6 vdsm kvm 59 11 апр 6,30 396604d9-2a9e-49cd-9563-fdc79981f67b drwxr-xr-x. 5 vdsm kvm 48 19 ное 21,32 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD As 578bca3d-6540-41cd-8e0e-9e3047026484-NEW & 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD are not valid SDuuids - the activation fails. I have opened a bug for it (not sure on the correct Product/Component) : https://bugzilla.redhat.com/show_bug.cgi?id=1823033 Hey Sandro, is this expected, as I never saw it as a requirement ? If it is expected (already in the docs), please mark 1823033 "notabug". Best Regards, Strahil Nikolov В вторник, 7 април 2020 г., 23:13:43 Гринуич+3, Strahil Nikolov <hunter86...@yahoo.com> написа: Hi Shani, thanks for the asitance. First atempt to activate storage domain "data_fast" (for today) which is ontop gluster volume "data_fast": So Far I see the following when I filter "INFO" on the engine: 2020-04-07 22:44:55,760+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54 0] Failed in 'ActivateStorageDomainVDS' method 2020-04-07 22:44:55,781+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:55,781+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] IrsBroker::F ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:44:56,223+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:44:58,086+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag eDomainVDS' method 2020-04-07 22:44:58,106+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:58,106+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] IrsBroker::Failed::ActivateStorageDomain VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:44:58,345+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE _POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:45:00,222+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage DomainVDS' method 2020-04-07 22:45:00,240+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:45:00,241+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] IrsBroker::Failed::ActivateStorageDomainV DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:45:00,241+03 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command 'org.ovirt.engine.core .bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to Activa teStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2020-04-07 22:45:00,298+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE _DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by admin@internal-authz Second attempt looks the same: 2020-04-07 22:46:10,899+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d 1] Failed in 'ActivateStorageDomainVDS' method 2020-04-07 22:46:10,923+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:10,923+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] IrsBroker::F ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:11,226+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE _POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:46:13,088+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage DomainVDS' method 2020-04-07 22:46:13,107+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:13,107+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] IrsBroker::Failed::ActivateStorageDomainV DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:13,358+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:46:15,131+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag eDomainVDS' method 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] IrsBroker::Failed::ActivateStorageDomain VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command 'org.ovirt.engine.cor e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to Activ ateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2020-04-07 22:46:15,192+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by admin@internal-authz SPM is ovirt1 and vdsm log shows: 2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83) 2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83) Metadata shows the following: [root@ovirt1 dom_md]# cat /rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata CLASS=Data DESCRIPTION=data_fast IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8 REMOTE_PATH=gluster1:/data_fast ROLE=Regular SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b TYPE=GLUSTERFS VERSION=4 _SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b [root@ovirt1 dom_md]# ls -l /rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata -rw-r--r--. 1 vdsm kvm 332 9 мар 2,25 /rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata [root@ovirt1 dom_md]# vdsm-client StorageDomain validate "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" vdsm-client: Command StorageDomain.validate with args {'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) [root@ovirt1 dom_md]# vdsm-client StorageDomain getStats "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" vdsm-client: Command StorageDomain.getStats with args {'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) [root@ovirt1 dom_md]# vdsm-client StorageDomain activate "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" "storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8" vdsm-client: Command StorageDomain.activate with args {'storagepoolID': 'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ & v7.1+ are causing me headaches). My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6 volumes are in maintenance mode and I cannot detach , nor to activate them :) What is the flow that the engine is implementing (what files are checked with which user, what is being read, extracted,etc) in order to activate a storage domain? I'm attaching the engine's log , SPM's log and some output from the DB. Affected storage domains are "data_fast" , "data_fast2" , "data_fast3" & "data_fast4" (gluster volumes are with the same name). Best Regards, Strahil Nikolov _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U536VB242PEVVCSK2SCYITFEDZBBFBYJ/
