I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt
"controller node" that manages.
2 problems:
It will no sync networks on hosts.
It will no launch a vm unless you go to system and choose an instance size. It
will boot and the network card has to be hot added.
I'm not sure what I'm missing,
I can give log entries showing several bad lines in diggerent modules.'
Engine log while updating hosts:
tail -n 100 -f engine.log
'
2025-01-09 02:43:15,722-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
HostName = kvm02
2025-01-09 02:43:15,722-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2025-01-09 02:43:15,729-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}),
log id: 5e145d4b
2025-01-09 02:43:15,729-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
HostName = kvm04
2025-01-09 02:43:18,057-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b
2025-01-09 02:43:18,057-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:22,344-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command 'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:43:22,344-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs
placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:22,349-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}),
log id: 28df8c85
2025-01-09 02:43:22,349-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
HostName = kvm01
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 28df8c85
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,275-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs
placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:43:25,279-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}),
log id: 20ec709f
2025-01-09 02:43:25,279-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:27,675-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:27,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
HostName = kvm02
2025-01-09 02:43:27,676-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,676-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 20ec709f
2025-01-09 02:43:27,676-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,741-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:43:27,765-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and
refresh for all hosts, last master 'true', command status 'true'
2025-01-09 02:43:27,789-05 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:43:30,154-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
START, GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 770c0b37
2025-01-09 02:43:30,362-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37
2025-01-09 02:43:30,381-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
START, AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 3b2ce207
2025-01-09 02:43:33,630-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
FINISH, AddGlusterServerVDSCommand, return: , log id: 3b2ce207
2025-01-09 02:43:33,640-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into
Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with
Transport endpoint is not connected']
2025-01-09 02:43:33,641-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
START, GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 6a99b1ab
2025-01-09 02:43:33,748-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) []
FINISH, GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: 6a99b1ab
2025-01-09 02:43:37,689-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:43:37,691-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:43:37,691-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
HostName = kvm02
2025-01-09 02:43:37,691-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2025-01-09 02:43:37,698-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
START, ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}),
log id: 201774c9
2025-01-09 02:43:37,698-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:38,381-05 INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) []
There is no host with more than 10 running guests, no balancing is needed
2025-01-09 02:43:38,400-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) []
START, GlusterTasksListVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}),
log id: 702ab66f
2025-01-09 02:43:38,515-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) []
FINISH, GlusterTasksListVDSCommand, return: [], log id: 702ab66f
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
HostName = kvm01
2025-01-09 02:43:40,598-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 201774c9
2025-01-09 02:43:40,598-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:40,606-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs
placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:43:40,611-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
START, ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}),
log id: 164d36cb
2025-01-09 02:43:40,611-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
HostName = kvm04
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 164d36cb
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,960-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:43:42,960-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs
placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:42,964-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
START, ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}),
log id: 863c329
2025-01-09 02:43:42,965-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
HostName = kvm02
2025-01-09 02:43:45,328-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 863c329
2025-01-09 02:43:45,328-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:45,388-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:43:45,412-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and
refresh for all hosts, last master 'true', command status 'true'
2025-01-09 02:43:45,436-05 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:43:48,763-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
START, GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: db1640e
2025-01-09 02:43:48,970-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e
2025-01-09 02:43:48,983-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
START, AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 717f84ae
2025-01-09 02:43:52,255-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
FINISH, AddGlusterServerVDSCommand, return: , log id: 717f84ae
2025-01-09 02:43:52,277-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into
Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with
Transport endpoint is not connected']
2025-01-09 02:43:52,278-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
START, GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 6a43f9e6
2025-01-09 02:43:52,384-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) []
FINISH, GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: 6a43f9e6
2025-01-09 02:43:55,341-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:43:55,343-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:43:55,343-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
HostName = kvm02
2025-01-09 02:43:55,343-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2025-01-09 02:43:55,350-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
START, ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}),
log id: 35cf220b
2025-01-09 02:43:55,350-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
HostName = kvm04
2025-01-09 02:43:57,661-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 35cf220b
2025-01-09 02:43:57,661-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:57,671-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:43:57,671-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs
placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:57,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
START, ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}),
log id: 2ff24c19
2025-01-09 02:43:57,676-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
HostName = kvm02
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:04,367-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:44:04,367-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs
placed on server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover
2025-01-09 02:44:04,372-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
START, ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}),
log id: 23ed7788
2025-01-09 02:44:04,372-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
HostName = kvm01
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 23ed7788
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,375-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:44:07,412-05 INFO
[org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default
task-7) [9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command:
SyncAllHostNetworksCommand internal: false. Entities affected : ID:
acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group
CONFIGURE_HOST_NETWORK with role type ADMIN
2025-01-09 02:44:07,421-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 340c8f9d
2025-01-09 02:44:07,423-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and
refresh for all hosts, last master 'true', command status 'true'
2025-01-09 02:44:07,423-05 INFO
[org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand]
(default task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand
internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd
Type: VDS
2025-01-09 02:44:07,448-05 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:44:07,460-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default
task-7) [7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1):
Applying network's changes on host kvm01. (User:
admin@ovirt@internalkeycloak-authz)
2025-01-09 02:44:07,461-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Before acquiring lock-timeout
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:07,461-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Lock-timeout acquired to object
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Running command: HostSetupNetworksCommand internal: true.
Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction
group CONFIGURE_HOST_NETWORK with role type ADMIN
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Before acquiring lock in order to prevent monitoring for
host 'kvm01' from data-center 'DDS_Data_Center'
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Lock acquired, from now a monitoring of host will be skipped
for host 'kvm01' from data-center 'DDS_Data_Center'
2025-01-09 02:44:07,498-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(default task-7) [7814c7ae] START, HostSetupNetworksVDSCommand(HostName =
kvm01,
HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]',
rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='false', bonding='false',
networkName='vlan3', vdsmName='vlan3', nicName='eno2', vlan='3',
vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP',
ipv4Address='172.50.26.252', ipv4Netmask='255.255.255.0',
ipv4Gateway='172.50.26.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'},
HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt',
vdsmName='ovirtmgmt', nicName='eno1', vlan='null', vmNetwork='true',
stp='false', properties='[]', ipv4BootProtocol='STAT
IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0',
ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null',
nameServers='[NameServer:{address='192.168.254.222'},
NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]', bonds='[]',
removedBonds='[]', clusterSwitchType='LEGACY',
managementNetworkChanged='true'}), log id: 56a27c18
2025-01-09 02:44:07,499-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(default task-7) [7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log
id: 56a27c18
2025-01-09 02:44:07,631-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d
2025-01-09 02:44:07,642-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 7c25cce3
2025-01-09 02:44:10,879-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, AddGlusterServerVDSCommand, return: , log id: 7c25cce3
2025-01-09 02:44:10,923-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into
Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with
Transport endpoint is not connected']
2025-01-09 02:44:10,923-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
START, GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: effd1e8
2025-01-09 02:44:11,033-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
FINISH, GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: effd1e8
2025-01-09 02:44:17,314-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:44:17,315-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
HostName = kvm01
2025-01-09 02:44:17,315-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01,
VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2025-01-09 02:44:17,331-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
START, ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}),
log id: bcbf59e
2025-01-09 02:44:17,331-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:18,272-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default
task-7) [7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's
following network(s) are not synchronized with their Logical Network
configuration: ovirtmgmt,vlan3.
2025-01-09 02:44:18,283-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Host setup networks finished. Lock released. Monitoring can
run now for host 'kvm01' from data-center 'DDS_Data_Center'
2025-01-09 02:44:18,285-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default
task-7) [7814c7ae] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:18,305-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default
task-7) [7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903),
(1/1): Successfully applied changes on host kvm01. (User:
admin@ovirt@internalkeycloak-authz)
2025-01-09 02:44:18,316-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default
task-7) [7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed
to sync all host kvm01 networks.
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command
ConnectStoragePoolVDS failed: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
HostName = kvm04
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: bcbf59e
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,679-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command 'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error
validating master storage domain: ('MD read error',)
2025-01-09 02:44:19,679-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs
placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:44:19,684-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
START, ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}),
log id: 251dbd22
2025-01-09 02:44:19,684-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
HostName = kvm01
2025-01-09 02:44:22,181-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 251dbd22
2025-01-09 02:44:22,181-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:22,189-05 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs
placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:44:22,198-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
START, ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}),
log id: 1f60ebb1
2025-01-09 02:44:22,198-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active,
028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:24,581-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master
domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:24,581-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
HostName = kvm02
2025-01-09 02:44:24,581-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
Command 'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:24,582-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
FINISH, ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1
2025-01-09 02:44:24,582-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:24,658-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:44:24,691-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and
refresh for all hosts, last master 'true', command status 'true'
2025-01-09 02:44:24,715-05 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
While attempting to start vm will be sent after but I think the net allocation
issue with hosts is the center of the vm issues,,
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/FESUTP32LCEA5FMPLS4ITQSXYG22KHUN/
