On Tue, Oct 25, 2022 at 7:38 PM Darrell Budic <bu...@onholyground.com> wrote:
>
> Both the initial messages and the additional detail in these vdsm logs show
> that it can’t mount a storage volume. In a multi node environment, it’s
> usually iscsi, gluster, etc, so start by looking at your network and the
> storage devices and figuring out why it can’t mount them. Comparison to any
> remaining running nodes may be helpful.
Indeed, but in this particular case, it also provides more details:
[' /dev/mapper/3600c0ff00051441d1
e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't
read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.",
' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000
at 30099960109568 has invalid summary for VG.', ' Failed to read
metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000',
' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000']
If this VG does work well from some other host, perhaps it's a local
issue. Otherwise, it's probably corrupted - if you have good and
recent backups, it might be best to restore them, otherwise you can
try fixing the VG - I'd start by searching the net for relevant parts
of this error message and see how people handled similar cases.
Good luck and best regards,
>
> > On Oct 25, 2022, at 3:40 AM, eug...@knorydev.com wrote:
> >
> > Hi Tomas,
> >
> > [root@host1 vdsm]# tail -f vdsm.log
> > StorageUnavailableError: Unable to get volume size for domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> > 581f65d2-3db5-4fde-960f-4c4ffe27341e
> > 2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm]
> > (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for
> > volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity:
> > 322122547200, allocated: 90872676352, physical: 91268055040
> > threshold_state: exceeded) (vm:1289)
> > 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START
> > sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> > volDict={'newSize': 92341796864, 'domainID':
> > '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> > '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*,
> > extend-volume=0.00*)>, 'internal': False, 'volumeID':
> > '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> > '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
> > callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm
> > object at 0x7fea007336d0>>) from=internal,
> > task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:48)
> > 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH
> > sendExtendMsg return=None from=internal,
> > task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54)
> > 2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm)
> > [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM -
> > ['/usr/bin/dd',
> > 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
> > 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1',
> > 'seek=1'] (mailbox:380)
> > 2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC
> > call Host.ping2 succeeded in 0.00 seconds (__init__:312)
> > 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats()
> > from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48)
> > 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats
> > error=Virtual machine does not exist: {'vmId':
> > u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'} (api:129)
> > 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats
> > return={'status': {'message': "Virtual machine does not exist: {'vmId':
> > u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code': 1}} from=::1,39184,
> > vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54)
> > 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC
> > call VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
> > 2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm]
> > (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for
> > volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity:
> > 322122547200, allocated: 90872676352, physical: 91268055040
> > threshold_state: exceeded) (vm:1289)
> > 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START
> > sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> > volDict={'newSize': 92341796864, 'domainID':
> > '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID':
> > '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*,
> > extend-volume=0.00*)>, 'internal': False, 'volumeID':
> > '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> > '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
> > callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm
> > object at 0x7fea007336d0>>) from=internal,
> > task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:48)
> > 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH
> > sendExtendMsg return=None from=internal,
> > task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54)
> > 2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC
> > call Host.ping2 succeeded in 0.00 seconds (__init__:312)
> > 2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm)
> > [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM -
> > ['/usr/bin/dd',
> > 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
> > 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1',
> > 'seek=1'] (mailbox:380)
> > 2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3)
> > [storage.ThreadPool.WorkerThread] START task
> > b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at
> > 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
> > <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
> > 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
> > 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
> > <Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal':
> > False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> > '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208)
> > 2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START
> > refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
> > spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> > imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
> > volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal,
> > task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48)
> > 2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
> > ['/sbin/lvm', 'vgs', '--config', 'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
> > "r|.*|"] } global { locking_type=4 prioritise_write_locks=1
> > wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0
> > }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|',
> > '--ignoreskippedcluster', '-o',
> > 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
> > '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
> > with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum
> > error at offset 30099960109568', " Couldn't read volume
> > group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", '
> > Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at
> > 30099960109568 has invalid summary for VG.', ' Failed to read metadata
> > summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to
> > scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
> > 2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH
> > refreshVolume error=Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal,
> > task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52)
> > 2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0')
> > Unexpected error (task:875)
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> > _run
> > return fn(*args, **kargs)
> > File "<string>", line 2, in refreshVolume
> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> > method
> > ret = func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in
> > refreshVolume
> > sdUUID=sdUUID).produceVolume(imgUUID=imgUUID,
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
> > produce
> > domain.getRealDomain()
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
> > getRealDomain
> > return self._cache._realProduce(self._sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
> > _realProduce
> > domain = self._findDomain(sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
> > _findDomain
> > return findMethod(sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line
> > 1843, in findDomain
> > return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line
> > 1701, in findDomainPath
> > raise se.StorageDomainDoesNotExist(sdUUID)
> > StorageDomainDoesNotExist: Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
> > 2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0')
> > aborting: Task is aborted: "Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
> > 2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher]
> > FINISH refreshVolume error=Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
> > 2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START
> > getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
> > spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
> > imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
> > volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None)
> > from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48)
> > 2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
> > ['/sbin/lvm', 'vgs', '--config', 'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
> > "r|.*|"] } global { locking_type=4 prioritise_write_locks=1
> > wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0
> > }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|',
> > '--ignoreskippedcluster', '-o',
> > 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
> > '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
> > with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum
> > error at offset 30099960109568', " Couldn't read volume
> > group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", '
> > Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at
> > 30099960109568 has invalid summary for VG.', ' Failed to read metadata
> > summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to
> > scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
> > 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH
> > getVolumeSize error=Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal,
> > task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52)
> > 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad')
> > Unexpected error (task:875)
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> > _run
> > return fn(*args, **kargs)
> > File "<string>", line 2, in getVolumeSize
> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> > method
> > ret = func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in
> > getVolumeSize
> > dom = sdCache.produce(sdUUID=sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
> > produce
> > domain.getRealDomain()
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
> > getRealDomain
> > return self._cache._realProduce(self._sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
> > _realProduce
> > domain = self._findDomain(sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
> > _findDomain
> > return findMethod(sdUUID)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line
> > 1843, in findDomain
> > return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line
> > 1701, in findDomainPath
> > raise se.StorageDomainDoesNotExist(sdUUID)
> > StorageDomainDoesNotExist: Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
> > 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad')
> > aborting: Task is aborted: "Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
> > 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher]
> > FINISH getVolumeSize error=Storage domain does not exist:
> > ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
> > 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114')
> > Unexpected error (task:875)
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> > _run
> > return fn(*args, **kargs)
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in
> > __afterVolumeExtension
> > volSize = self.__verifyVolumeExtension(volInfo)
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in
> > __verifyVolumeExtension
> > volInfo['imageID'], volInfo['volumeID'])
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in
> > _getVolumeSize
> > (domainID, volumeID))
> > StorageUnavailableError: Unable to get volume size for domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> > 581f65d2-3db5-4fde-960f-4c4ffe27341e
> > 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3)
> > [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114')
> > aborting: Task is aborted: u'Unable to get volume size for domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> > 581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181)
> > 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3)
> > [storage.ThreadPool.WorkerThread] FINISH task
> > b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function
> > runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension
> > of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
> > 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
> > 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
> > <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>, 'internal':
> > False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
> > '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222)
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line
> > 209, in _processNextTask
> > cmd(args)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85,
> > in runTask
> > ctask.prepare(cmd, *args)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> > wrapper
> > return m(self, *a, **kw)
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189,
> > in prepare
> > raise self.error
> > StorageUnavailableError: Unable to get volume size for domain
> > 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
> > 581f65d2-3db5-4fde-960f-4c4ffe27341e
> > ^C
> >
> > This is what I get from vdsm when I ran tail -f.
> > _______________________________________________
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/QRSLEBLJUE5NY5CTNB2IJ7MTFGEZCBBG/
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q6U3KM7BCJIQA6G4IK56FJVQKEGYE7FN/
--
Didi
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EUIWGSAMJVUK6PW7NVW6EOACO5BLVA44/
