** Project changed: nova => ceph
** Also affects: nova
Importance: Undecided
Status: New
** Changed in: nova
Importance: Undecided => High
** Changed in: nova
Status: New => Triaged
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2116852
Title:
nova-ceph-multistore job fails constantly with memory allocation error
Status in Ceph:
New
Status in OpenStack Compute (nova):
Triaged
Bug description:
https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-
multistore&skip=0
First failure was at 2025-07-11 08:33:54
https://zuul.opendev.org/t/openstack/build/3c87e32281df4101b698b01ae6b1f3d5
Since then each run fails as the test case
test_extend_attached_encrypted_volume_luksv1[id-e93243ec-7c37-4b5b-a099-ebf052c13216]
fails with
Traceback (most recent call last):
File
"/opt/stack/tempest/tempest/api/volume/admin/test_encrypted_volumes_extend.py",
line 41, in test_extend_attached_encrypted_volume_luksv1
self._test_extend_attached_volume(volume)
File "/opt/stack/tempest/tempest/api/volume/test_volumes_extend.py", line
173, in _test_extend_attached_volume
self.assertTrue(
File "/usr/lib/python3.12/unittest/case.py", line 727, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true : Unexpected compute_extend_volume result
'Error' for request req-9ec57a2c-91f8-46f6-8718-6171d653020a.
While the following log is visible in the nova-compute log:
Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [req-ed795c55-2c45-4828-98ce-40c79568e8b6
req-9ec57a2c-91f8-46f6-8718-6171d653020a service nova] [instance:
259e9dfb-5882-4de0-9672-c619cb9ea4e6] Unknown error when attempting to find the
payload_offset for LUKSv1 encrypted disk
rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder.:
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to
execute on rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder :
Unexpected error while running command.
76292 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Command:
/opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824
--cpu=30 -- env LC_ALL=C LANG=C qemu-img info
rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder --force-share
--output=json
76293 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Exit code:
-6
76294 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Stdout: ''
76295 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Stderr:
'failed to allocate memory for stack: Cannot allocate memory\n'
76296 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
Traceback (most recent call last):
76297 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2969, in
_resize_attached_encrypted_volume
76298 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
info = images.privileged_qemu_img_info(path)
76299 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
76300 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
File "/opt/stack/nova/nova/virt/images.py", line 57, in privileged_qemu_img_info
76301 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
info = nova.privsep.qemu.privileged_qemu_img_info(path, format=format)
76302 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
76303 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/priv_context.py",
line 267, in _wrap
76304 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
return self.channel.remote_call(name, args, kwargs,
76305 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
76306 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/daemon.py",
line 227, in remote_call
76307 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
raise exc_type(*result[2])
76308 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to
execute on rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder :
Unexpected error while running command.
76309 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
Command: /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit
--as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info
rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder --force-share
--output=json
76310 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Exit
code: -6
76311 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
Stdout: ''
76312 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
Stderr: 'failed to allocate memory for stack: Cannot allocate memory\n'
76313 Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR
nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]
76314 Jul 13 13:52:20.367937 np42aaac35a2c74 nova-compute[93712]: WARNING
nova.compute.manager [req-ed795c55-2c45-4828-98ce-40c79568e8b6
req-9ec57a2c-91f8-46f6-8718-6171d653020a service nova] [instance:
259e9dfb-5882-4de0-9672-c619cb9ea4e6] Extend volume failed,
volume_id=514bd4d2-0a87-427d-a69e-c03efe96ba05, reason: Disk info file is
invalid: qemu-img failed to execute on
rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder : Unexpected
error while running command.
https://zuul.opendev.org/t/openstack/build/46d5b5c8de4f4cd38bd882693c9e1b06/log/controller/logs/screen-
n-cpu.txt#76291
logsearch log --job nova-ceph-multistore --result FAILURE --file
controller/logs/screen-n-cpu.txt --days 7 'Unknown error when attempting to
find the payload_offset for LUKSv1 encrypted disk rbd:volumes.*:
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to
execute on rbd:volumes.*: Unexpected error while running command.'
Builds with matching logs 35/40:
+----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+
| uuid | finished | project
| pipeline | review | branch |
+----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+
| 46d5b5c8de4f4cd38bd882693c9e1b06 | 2025-07-13T14:14:55 | openstack/nova
| check | https://review.opendev.org/948186 | master |
| 6eb3deb7440e4048830040f10f3cd4cb | 2025-07-13T04:54:29 | openstack/devstack
| check | https://review.opendev.org/954750 | master |
| fe9ffb6424c74ef7b894aa6cd0b3fd6b | 2025-07-12T09:28:10 | openstack/nova
| check | https://review.opendev.org/940508 | master |
| 397e40dfc1b6488c8272bf425aa1041c | 2025-07-12T05:40:13 | openstack/tempest
| check | https://review.opendev.org/954404 | master |
| 6f087929b98844dfb1432b3623bba8a1 | 2025-07-11T22:59:56 | openstack/tempest
| check | https://review.opendev.org/954404 | master |
| a3c7f03f5c8145ff96f46dda94c2505a | 2025-07-11T18:53:59 | openstack/nova
| check | https://review.opendev.org/940508 | master |
| 25509218b73d44909f5aa13b17549a3e | 2025-07-11T17:52:28 | openstack/devstack
| check | https://review.opendev.org/954751 | master |
| aeb854c93e684e22b82d53f698468447 | 2025-07-11T17:08:06 | openstack/devstack
| check | https://review.opendev.org/954750 | master |
| caf13dce4ea345f18791e3355f7826ba | 2025-07-11T17:08:31 | openstack/nova
| check | https://review.opendev.org/940508 | master |
| 6409d84e7dce4beea35a08ef252a30e9 | 2025-07-11T16:55:14 | openstack/devstack
| check | https://review.opendev.org/915980 | master |
| f992cc90cc5145e29c6742f33166571d | 2025-07-11T16:56:37 | openstack/devstack
| check | https://review.opendev.org/936089 | master |
| c5ea790752424e739638dccf8c71cdd9 | 2025-07-11T15:58:40 | openstack/nova
| check | https://review.opendev.org/952666 | master |
| 7bbd100af7204665bce4152853c34e55 | 2025-07-11T15:24:39 | openstack/nova
| check | https://review.opendev.org/953436 | master |
| e3c0e66d95d94acdb1d3203d30496272 | 2025-07-11T16:21:40 | openstack/nova
| check | https://review.opendev.org/948311 | master |
| 6cb5dade8b074853a5c08523216916ba | 2025-07-11T16:00:42 | openstack/nova
| check | https://review.opendev.org/953815 | master |
| 533c09d432e14917aaf74aac4d720786 | 2025-07-11T15:39:09 | openstack/nova
| check | https://review.opendev.org/951957 | master |
| 6074e8fc93324f328d75c4f887d060b8 | 2025-07-11T16:00:11 | openstack/nova
| check | https://review.opendev.org/948087 | master |
| 6aeea098e1814ef28e19f7f0f520d3bb | 2025-07-11T15:55:59 | openstack/nova
| check | https://review.opendev.org/948187 | master |
| bfc1f8c9711c475e9a87885a902a4671 | 2025-07-11T15:52:37 | openstack/nova
| check | https://review.opendev.org/953475 | master |
| 00680c804d44472d861687b710c0a05b | 2025-07-11T16:04:51 | openstack/nova
| check | https://review.opendev.org/948450 | master |
| a1722357540a4216852e21179b9c9bef | 2025-07-11T15:57:02 | openstack/nova
| check | https://review.opendev.org/948340 | master |
| 31deee8d4e1b4ec2a21606c4092ad419 | 2025-07-11T15:58:44 | openstack/nova
| check | https://review.opendev.org/948086 | master |
| 03d3dd1627b64776b40f0592956de636 | 2025-07-11T15:35:00 | openstack/nova
| check | https://review.opendev.org/950994 | master |
| 75bb8c2efb814394ab02fd51fae70473 | 2025-07-11T15:40:12 | openstack/nova
| check | https://review.opendev.org/948188 | master |
| c87e4c615ceb40b99a3f355f0cc04758 | 2025-07-11T15:07:46 | openstack/nova
| check | https://review.opendev.org/948186 | master |
| ee8214075c4e4c90ba7623a5650cf199 | 2025-07-11T14:44:43 | openstack/nova
| check | https://review.opendev.org/953338 | master |
| 3f985e2d1c564154a3d697c010000ad1 | 2025-07-11T14:47:47 | openstack/nova
| check | https://review.opendev.org/940508 | master |
| 6b933d4362a548368ceee5ecfd5115dd | 2025-07-11T12:35:30 | openstack/nova
| check | https://review.opendev.org/953704 | master |
| c4335675bc3c41d09f4b7dc27d6bc052 | 2025-07-11T12:09:26 | openstack/nova
| check | https://review.opendev.org/902688 | master |
| bf6bf0af10bf40b3b9ed1310e99c7e60 | 2025-07-11T12:24:00 | openstack/nova
| check | https://review.opendev.org/953705 | master |
| 46b2b1a3d69c420b94f81c1be35e3da9 | 2025-07-11T12:12:51 | openstack/nova
| check | https://review.opendev.org/953703 | master |
| d5e8e47f4e574d0d8e4e2981a5f66f25 | 2025-07-11T12:15:29 | openstack/nova
| check | https://review.opendev.org/940508 | master |
| bcacd1b14bb44af4836a124ebfba230e | 2025-07-11T11:06:51 | openstack/nova
| check | https://review.opendev.org/954210 | master |
| 3269938364c143dfa4d7f40a08a7e3a6 | 2025-07-11T09:44:50 | openstack/nova
| check | https://review.opendev.org/954613 | master |
| 3c87e32281df4101b698b01ae6b1f3d5 | 2025-07-11T09:31:38 | openstack/nova
| check | https://review.opendev.org/954149 | master |
+----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/ceph/+bug/2116852/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
