Public bug reported: Hi,
I followed the guide here - https://docs.openstack.org/cinder/pike/configuration/block-storage /volume-encryption.html I also use Barbican and for that I added [barbican] auth_endpoint = http://controller:5000 to cinder.conf and nova.conf Creation of LUKS disks is successful. I also created normal disks and could easily attach them to an instance. Cinder disks are on LVM Attaching LUKS disks fails with the following trace: 017-10-05 11:44:57.445 1 INFO nova.compute.manager [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Attaching volume 5c03f92f-470a-4f15-aaca-49d9232512a8 to /dev/vdc 2017-10-05 11:44:57.835 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpKwgHpn/privsep.sock'] 2017-10-05 11:44:58.598 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Spawned new privsep daemon via rootwrap 2017-10-05 11:44:58.548 80 INFO oslo.privsep.daemon [-] privsep daemon starting 2017-10-05 11:44:58.553 80 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep daemon running as pid 80 2017-10-05 11:45:01.468 1 INFO os_brick.initiator.connectors.iscsi [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Trying to connect to iSCSI portal 10.10.245.211:3260 2017-10-05 11:45:05.762 1 WARNING os_brick.encryptors [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Use of the in tree encryptor class nova.volume.encryptors.luks.LuksEncryptor by directly referencing the implementation class will be blocked in the Queens release of os-brick. 2017-10-05 11:45:07.431 1 WARNING os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] isLuks exited abnormally (status 1): Device /dev/sdb is not a valid LUKS device. Command failed with code 22: Invalid argument : ProcessExecutionError: Unexpected error while running command. Command: cryptsetup isLuks --verbose /dev/sdb Exit code: 1 Stdout: u'' Stderr: u'Device /dev/sdb is not a valid LUKS device.\nCommand failed with code 22: Invalid argument\n' 2017-10-05 11:45:07.432 1 INFO os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] /dev/sdb is not a valid LUKS device; formatting device for first use 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Failed to attach volume at mountpoint: /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Traceback (most recent call last): 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] encryptor.attach_volume(context, **encryption) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 160, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] self._format_volume(passphrase, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 87, in _format_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] attempts=3) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] result = self.__execute(*args, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return execute_root(*cmd, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 205, in _wrap 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return self.channel.remote_call(name, args, kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] raise exc_type(*result[2]) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] ProcessExecutionError: Unexpected error while running command. 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Exit code: 5 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stdout: u'' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stderr: u'Cannot format device /dev/sdb which is still in use.\n' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] 2017-10-05 11:45:19.027 1 ERROR nova.virt.block_device [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Driver failed to attach volume 5c03f92f-470a-4f15-aaca-49d9232512a8 at /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' I have checked that /dev/sdb appears for short time (/dev/sda is unencrypted drive attached to an instance on the same host) and running cryptsetup luksFormat gives 'cannot format device...still in use' (nova-compute)[root@compute5 /]# pip freeze | grep nova nova==13.0.0.0rc4.dev8314 python-novaclient==9.1.0 (nova-compute)[root@compute5 /]# pip freeze | grep oslo oslo.cache==1.25.0 oslo.concurrency==3.21.0 oslo.config==4.11.0 oslo.context==2.17.0 oslo.db==4.25.0 oslo.i18n==3.17.0 oslo.log==3.30.0 oslo.messaging==5.30.0 oslo.middleware==3.30.0 oslo.policy==1.25.1 oslo.privsep==1.22.0 oslo.reports==1.22.0 oslo.rootwrap==5.9.0 oslo.serialization==2.20.0 oslo.service==1.25.0 oslo.utils==3.28.0 oslo.versionedobjects==1.26.0 oslo.vmware==2.24.0 (nova-compute)[root@compute5 /]# pip freeze | grep barbican python-barbicanclient==4.5.2 (nova-compute)[root@compute5 /]# rpm -qa | grep crypt libgcrypt-devel-1.5.3-14.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 cryptsetup-1.7.4-3.el7.x86_64 cryptsetup-libs-1.7.4-3.el7.x86_64 ** Affects: nova Importance: Undecided Status: New ** Description changed: Hi, I followed the guide here - https://docs.openstack.org/cinder/pike/configuration/block-storage /volume-encryption.html I also use Barbican and for that I added [barbican] auth_endpoint = - controlnode:5000 to cinder.conf and nova.conf + http://controller:5000 to cinder.conf and nova.conf Creation of LUKS disks is successful. I also created normal disks and could easily attach them to an instance. Cinder disks are on LVM Attaching LUKS disks fails with the following trace: 017-10-05 11:44:57.445 1 INFO nova.compute.manager [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Attaching volume 5c03f92f-470a-4f15-aaca-49d9232512a8 to /dev/vdc 2017-10-05 11:44:57.835 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpKwgHpn/privsep.sock'] 2017-10-05 11:44:58.598 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Spawned new privsep daemon via rootwrap 2017-10-05 11:44:58.548 80 INFO oslo.privsep.daemon [-] privsep daemon starting 2017-10-05 11:44:58.553 80 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep daemon running as pid 80 2017-10-05 11:45:01.468 1 INFO os_brick.initiator.connectors.iscsi [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Trying to connect to iSCSI portal 10.10.245.211:3260 2017-10-05 11:45:05.762 1 WARNING os_brick.encryptors [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Use of the in tree encryptor class nova.volume.encryptors.luks.LuksEncryptor by directly referencing the implementation class will be blocked in the Queens release of os-brick. 2017-10-05 11:45:07.431 1 WARNING os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] isLuks exited abnormally (status 1): Device /dev/sdb is not a valid LUKS device. Command failed with code 22: Invalid argument : ProcessExecutionError: Unexpected error while running command. Command: cryptsetup isLuks --verbose /dev/sdb Exit code: 1 Stdout: u'' Stderr: u'Device /dev/sdb is not a valid LUKS device.\nCommand failed with code 22: Invalid argument\n' 2017-10-05 11:45:07.432 1 INFO os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] /dev/sdb is not a valid LUKS device; formatting device for first use 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Failed to attach volume at mountpoint: /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Traceback (most recent call last): 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] encryptor.attach_volume(context, **encryption) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 160, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] self._format_volume(passphrase, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 87, in _format_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] attempts=3) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] result = self.__execute(*args, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return execute_root(*cmd, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 205, in _wrap 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return self.channel.remote_call(name, args, kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] raise exc_type(*result[2]) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] ProcessExecutionError: Unexpected error while running command. 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Exit code: 5 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stdout: u'' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stderr: u'Cannot format device /dev/sdb which is still in use.\n' - 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] + 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] 2017-10-05 11:45:19.027 1 ERROR nova.virt.block_device [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Driver failed to attach volume 5c03f92f-470a-4f15-aaca-49d9232512a8 at /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' - - I have checked that /dev/sdb appears for short time (/dev/sda is unencrypted drive attached to an instance on the same host) and running cryptsetup luksFormat gives 'cannot format device...still in use' - + I have checked that /dev/sdb appears for short time (/dev/sda is + unencrypted drive attached to an instance on the same host) and running + cryptsetup luksFormat gives 'cannot format device...still in use' (nova-compute)[root@compute5 /]# pip freeze | grep nova nova==13.0.0.0rc4.dev8314 python-novaclient==9.1.0 (nova-compute)[root@compute5 /]# pip freeze | grep oslo oslo.cache==1.25.0 oslo.concurrency==3.21.0 oslo.config==4.11.0 oslo.context==2.17.0 oslo.db==4.25.0 oslo.i18n==3.17.0 oslo.log==3.30.0 oslo.messaging==5.30.0 oslo.middleware==3.30.0 oslo.policy==1.25.1 oslo.privsep==1.22.0 oslo.reports==1.22.0 oslo.rootwrap==5.9.0 oslo.serialization==2.20.0 oslo.service==1.25.0 oslo.utils==3.28.0 oslo.versionedobjects==1.26.0 oslo.vmware==2.24.0 (nova-compute)[root@compute5 /]# pip freeze | grep barbican python-barbicanclient==4.5.2 - (nova-compute)[root@compute5 /]# rpm -qa | grep crypt + (nova-compute)[root@compute5 /]# rpm -qa | grep crypt libgcrypt-devel-1.5.3-14.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 cryptsetup-1.7.4-3.el7.x86_64 cryptsetup-libs-1.7.4-3.el7.x86_64 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1721522 Title: encrypted volumes: Cannot format device /dev/sdb which is still in use Status in OpenStack Compute (nova): New Bug description: Hi, I followed the guide here - https://docs.openstack.org/cinder/pike/configuration/block-storage /volume-encryption.html I also use Barbican and for that I added [barbican] auth_endpoint = http://controller:5000 to cinder.conf and nova.conf Creation of LUKS disks is successful. I also created normal disks and could easily attach them to an instance. Cinder disks are on LVM Attaching LUKS disks fails with the following trace: 017-10-05 11:44:57.445 1 INFO nova.compute.manager [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Attaching volume 5c03f92f-470a-4f15-aaca-49d9232512a8 to /dev/vdc 2017-10-05 11:44:57.835 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpKwgHpn/privsep.sock'] 2017-10-05 11:44:58.598 1 INFO oslo.privsep.daemon [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Spawned new privsep daemon via rootwrap 2017-10-05 11:44:58.548 80 INFO oslo.privsep.daemon [-] privsep daemon starting 2017-10-05 11:44:58.553 80 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2017-10-05 11:44:58.558 80 INFO oslo.privsep.daemon [-] privsep daemon running as pid 80 2017-10-05 11:45:01.468 1 INFO os_brick.initiator.connectors.iscsi [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Trying to connect to iSCSI portal 10.10.245.211:3260 2017-10-05 11:45:05.762 1 WARNING os_brick.encryptors [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] Use of the in tree encryptor class nova.volume.encryptors.luks.LuksEncryptor by directly referencing the implementation class will be blocked in the Queens release of os-brick. 2017-10-05 11:45:07.431 1 WARNING os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] isLuks exited abnormally (status 1): Device /dev/sdb is not a valid LUKS device. Command failed with code 22: Invalid argument : ProcessExecutionError: Unexpected error while running command. Command: cryptsetup isLuks --verbose /dev/sdb Exit code: 1 Stdout: u'' Stderr: u'Device /dev/sdb is not a valid LUKS device.\nCommand failed with code 22: Invalid argument\n' 2017-10-05 11:45:07.432 1 INFO os_brick.encryptors.luks [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] /dev/sdb is not a valid LUKS device; formatting device for first use 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Failed to attach volume at mountpoint: /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Traceback (most recent call last): 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1250, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] encryptor.attach_volume(context, **encryption) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 160, in attach_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] self._format_volume(passphrase, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 87, in _format_volume 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] attempts=3) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] result = self.__execute(*args, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return execute_root(*cmd, **kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 205, in _wrap 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] return self.channel.remote_call(name, args, kwargs) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] raise exc_type(*result[2]) 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] ProcessExecutionError: Unexpected error while running command. 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Exit code: 5 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stdout: u'' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Stderr: u'Cannot format device /dev/sdb which is still in use.\n' 2017-10-05 11:45:18.848 1 ERROR nova.virt.libvirt.driver [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] 2017-10-05 11:45:19.027 1 ERROR nova.virt.block_device [req-4780adcc-161e-4223-b53c-56b27a984ca1 f20698e1d05a4e2582e023778bfa5693 6a4c6d8b18714579a6e448e754d8838f - default default] [instance: 90376ab6-d553-477a-bda6-eeed8e70cc8d] Driver failed to attach volume 5c03f92f-470a-4f15-aaca-49d9232512a8 at /dev/vdc: ProcessExecutionError: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/sdb Exit code: 5 Stdout: u'' Stderr: u'Cannot format device /dev/sdb which is still in use.\n' I have checked that /dev/sdb appears for short time (/dev/sda is unencrypted drive attached to an instance on the same host) and running cryptsetup luksFormat gives 'cannot format device...still in use' (nova-compute)[root@compute5 /]# pip freeze | grep nova nova==13.0.0.0rc4.dev8314 python-novaclient==9.1.0 (nova-compute)[root@compute5 /]# pip freeze | grep oslo oslo.cache==1.25.0 oslo.concurrency==3.21.0 oslo.config==4.11.0 oslo.context==2.17.0 oslo.db==4.25.0 oslo.i18n==3.17.0 oslo.log==3.30.0 oslo.messaging==5.30.0 oslo.middleware==3.30.0 oslo.policy==1.25.1 oslo.privsep==1.22.0 oslo.reports==1.22.0 oslo.rootwrap==5.9.0 oslo.serialization==2.20.0 oslo.service==1.25.0 oslo.utils==3.28.0 oslo.versionedobjects==1.26.0 oslo.vmware==2.24.0 (nova-compute)[root@compute5 /]# pip freeze | grep barbican python-barbicanclient==4.5.2 (nova-compute)[root@compute5 /]# rpm -qa | grep crypt libgcrypt-devel-1.5.3-14.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 cryptsetup-1.7.4-3.el7.x86_64 cryptsetup-libs-1.7.4-3.el7.x86_64 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1721522/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp