Both the initial messages and the additional detail in these vdsm logs show
that it can’t mount a storage volume. In a multi node environment, it’s usually
iscsi, gluster, etc, so start by looking at your network and the storage
devices and figuring out why it can’t mount them. Comparison to any remaining
running nodes may be helpful.
> On Oct 25, 2022, at 3:40 AM, eug...@knorydev.com wrote:
>
> Hi Tomas,
>
> [root@host1 vdsm]# tail -f vdsm.log
> StorageUnavailableError: Unable to get volume size for domain
> 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> 581f65d2-3db5-4fde-960f-4c4ffe27341e
> 2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm]
> (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
> 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain
> 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity:
> 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state:
> exceeded) (vm:1289)
> 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START
> sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> volDict={'newSize': 92341796864, 'domainID':
> '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*,
> extend-volume=0.00*)>, 'internal': False, 'volumeID':
> '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
> callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm
> object at 0x7fea007336d0>>) from=internal,
> task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:48)
> 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH
> sendExtendMsg return=None from=internal,
> task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54)
> 2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm)
> [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM -
> ['/usr/bin/dd',
> 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
> 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1',
> 'seek=1'] (mailbox:380)
> 2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC
> call Host.ping2 succeeded in 0.00 seconds (__init__:312)
> 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats()
> from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48)
> 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats
> error=Virtual machine does not exist: {'vmId':
> u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'} (api:129)
> 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats
> return={'status': {'message': "Virtual machine does not exist: {'vmId':
> u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code': 1}} from=::1,39184,
> vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54)
> 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC
> call VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
> 2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm]
> (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
> 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain
> 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity:
> 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state:
> exceeded) (vm:1289)
> 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START
> sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> volDict={'newSize': 92341796864, 'domainID':
> '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*,
> extend-volume=0.00*)>, 'internal': False, 'volumeID':
> '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
> callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm
> object at 0x7fea007336d0>>) from=internal,
> task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:48)
> 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH
> sendExtendMsg return=None from=internal,
> task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54)
> 2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC
> call Host.ping2 succeeded in 0.00 seconds (__init__:312)
> 2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm)
> [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM -
> ['/usr/bin/dd',
> 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
> 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1',
> 'seek=1'] (mailbox:380)
> 2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3)
> [storage.ThreadPool.WorkerThread] START task
> b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at
> 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
> <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
> 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=2.88*,
> extend-volume=2.88, refresh-volume=0.00*)>, 'internal': False, 'volumeID':
> '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208)
> 2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START
> refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
> spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
> volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal,
> task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48)
> 2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
> ['/sbin/lvm', 'vgs', '--config', 'devices {
> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3
> filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
> "r|.*|"] } global { locking_type=4 prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|',
> '--ignoreskippedcluster', '-o',
> 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
> '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with
> warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error
> at offset 30099960109568', " Couldn't read volume
> group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", '
> Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at
> 30099960109568 has invalid summary for VG.', ' Failed to read metadata
> summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to
> scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
> 2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH
> refreshVolume error=Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal,
> task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52)
> 2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run
> return fn(*args, **kargs)
> File "<string>", line 2, in refreshVolume
> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method
> ret = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in
> refreshVolume
> sdUUID=sdUUID).produceVolume(imgUUID=imgUUID,
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
> produce
> domain.getRealDomain()
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
> getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
> _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
> _findDomain
> return findMethod(sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843,
> in findDomain
> return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701,
> in findDomainPath
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
> 2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted:
> "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" -
> code 358 (task:1181)
> 2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher]
> FINISH refreshVolume error=Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
> 2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START
> getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
> spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
> volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal,
> task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48)
> 2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
> ['/sbin/lvm', 'vgs', '--config', 'devices {
> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3
> filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
> "r|.*|"] } global { locking_type=4 prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|',
> '--ignoreskippedcluster', '-o',
> 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
> '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with
> warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error
> at offset 30099960109568', " Couldn't read volume
> group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", '
> Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at
> 30099960109568 has invalid summary for VG.', ' Failed to read metadata
> summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to
> scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
> 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH
> getVolumeSize error=Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal,
> task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52)
> 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run
> return fn(*args, **kargs)
> File "<string>", line 2, in getVolumeSize
> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method
> ret = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in
> getVolumeSize
> dom = sdCache.produce(sdUUID=sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
> produce
> domain.getRealDomain()
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
> getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
> _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
> _findDomain
> return findMethod(sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843,
> in findDomain
> return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701,
> in findDomainPath
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
> 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted:
> "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" -
> code 358 (task:1181)
> 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher]
> FINISH getVolumeSize error=Storage domain does not exist:
> ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
> 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run
> return fn(*args, **kargs)
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in
> __afterVolumeExtension
> volSize = self.__verifyVolumeExtension(volInfo)
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in
> __verifyVolumeExtension
> volInfo['imageID'], volInfo['volumeID'])
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in
> _getVolumeSize
> (domainID, volumeID))
> StorageUnavailableError: Unable to get volume size for domain
> 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> 581f65d2-3db5-4fde-960f-4c4ffe27341e
> 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
> (Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted:
> u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
> volume 581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181)
> 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3)
> [storage.ThreadPool.WorkerThread] FINISH task
> b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function
> runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
> <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
> 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=3.20*,
> extend-volume=2.88, refresh-volume=0.16)>, 'internal': False, 'volumeID':
> '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line
> 209, in _processNextTask
> cmd(args)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in
> runTask
> ctask.prepare(cmd, *args)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> wrapper
> return m(self, *a, **kw)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
> prepare
> raise self.error
> StorageUnavailableError: Unable to get volume size for domain
> 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> 581f65d2-3db5-4fde-960f-4c4ffe27341e
> ^C
>
> This is what I get from vdsm when I ran tail -f.
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/QRSLEBLJUE5NY5CTNB2IJ7MTFGEZCBBG/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q6U3KM7BCJIQA6G4IK56FJVQKEGYE7FN/
