Resolved by changing the no_path_retry option in multipath.conf from "queue" to "0". The issue was that that when IO was queued and the path was about to be removed it was blocked and was never removed, so the flushing of the multipath device failed because the multipath device was in-use by this stuck device.
I also changed removed the VIR_MIGRATE_TUNNELLED value from the live_migration_flag option in nova.conf by recommendation from Kashyap Chamarthy (kashyapc). To reload the multipath.conf while multipathd is running (won't stop or break your multipath devices). multipathd -k reconfigure Resolved with good help from these links: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623613 http://linux.die.net/man/5/multipath.conf http://christophe.varoqui.free.fr/refbook.html Right now, 26 live migrations and counting without any issues. Best regards ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1525802 Title: live migration with multipath cinder volumes crashes node Status in OpenStack Compute (nova): Invalid Bug description: Hello, When issuing a live migration between kvm nodes having multipath cinder volume it sometimes hangs and causes qemu-kvm to crash, the only solution is a restart of the kvm node. Sometimes when live migrating you get stuck when it tries to migrate the active RAM, you will see something like this in the nova-compute.log: http://paste.openstack.org/show/481773/ As you can see it get's nowhere. What is happening in the backgroun is that for some reason the multipath volumes when viewing with 'multipath -ll' they go into a 'faulty running' state and causes issues with the block device causing the qemu-kvm process to hang, the kvm node also tries to run blkid and kpart but all of those hang, which means you can get 100+ load just for those stuck processes. [1015086.978188] end_request: I/O error, dev sdg, sector 41942912 [1015086.978398] device-mapper: multipath: Failing path 8:96. [1015088.547034] qbr8eff45f7-ed: port 1(qvb8eff45f7-ed) entered disabled state [1015088.791695] INFO: task qemu-system-x86:19383 blocked for more than 120 seconds. [1015088.791940] Tainted: P OX 3.13.0-68-generic #111-Ubuntu [1015088.792147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1015088.792396] qemu-system-x86 D ffff88301f2f3180 0 19383 1 0x00000000 [1015088.792404] ffff8817440ada88 0000000000000086 ffff8817fa574800 ffff8817440adfd8 [1015088.792414] 0000000000013180 0000000000013180 ffff8817fa574800 ffff88301f2f3a18 [1015088.792420] 0000000000000000 ffff882ff7ab5280 0000000000000000 ffff8817fa574800 [1015088.792426] Call Trace: [1015088.792440] [<ffffffff8172880d>] io_schedule+0x9d/0x140 [1015088.792449] [<ffffffff811fc304>] do_blockdev_direct_IO+0x1ce4/0x2910 [1015088.792456] [<ffffffff811f7170>] ? I_BDEV+0x10/0x10 [1015088.792462] [<ffffffff811fcf85>] __blockdev_direct_IO+0x55/0x60 [1015088.792467] [<ffffffff811f7170>] ? I_BDEV+0x10/0x10 [1015088.792472] [<ffffffff811f7866>] blkdev_direct_IO+0x56/0x60 [1015088.792476] [<ffffffff811f7170>] ? I_BDEV+0x10/0x10 [1015088.792482] [<ffffffff81150ce1>] generic_file_direct_write+0xc1/0x180 [1015088.792487] [<ffffffff811510a5>] __generic_file_aio_write+0x305/0x3d0 [1015088.792492] [<ffffffff811f8146>] blkdev_aio_write+0x46/0x90 [1015088.792501] [<ffffffff811bdc2a>] do_sync_write+0x5a/0x90 [1015088.792507] [<ffffffff811be3b4>] vfs_write+0xb4/0x1f0 [1015088.792512] [<ffffffff811bef62>] SyS_pwrite64+0x72/0xb0 [1015088.792519] [<ffffffff81734cdd>] system_call_fastpath+0x1a/0x1f root 19410 0.0 0.0 0 0 ? D 08:12 0:00 [blkid] root 19575 0.0 0.0 0 0 ? D 08:13 0:00 [blkid] root 19584 0.0 0.0 28276 1076 ? S 08:13 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 21734 0.0 0.0 28276 1080 ? D 08:15 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 21735 0.0 0.0 28276 1076 ? S 08:15 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000ed root 22419 0.0 0.0 28276 1076 ? D 08:16 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000ed root 22420 0.0 0.0 28276 1076 ? D 08:16 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 22864 0.0 0.0 28276 1076 ? D 08:16 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000ed root 22865 0.0 0.0 28276 1076 ? D 08:16 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 23316 0.0 0.0 28276 1076 ? D 08:17 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 23317 0.0 0.0 28276 1072 ? D 08:17 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000ed root 23756 0.0 0.0 28276 1076 ? D 08:17 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 24200 0.0 0.0 28276 1076 ? D 08:18 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 24637 0.0 0.0 28276 1072 ? D 08:18 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root 25058 0.0 0.0 28276 1076 ? D 08:19 0:00 /sbin/kpartx -a -p -part /dev/mapper/36000d31000a6500000000000000000c6 root@kvm3:~# Ultimately this will cause so much issues on your kvm node that the only fix is a restart because of all the libvirt locks you won't be able to stop, restart or destroy the qemu-kvm process and issuing a kill -9 won't help you either, the only solution is a restart. What will happen is that your live migration will fail with something like this. 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [req-99771cf6-d17e-49f7-a01d-38201afbce69 212f451de64b4ae89c853f1430510037 e47ebdf3f3934025b37df3b85bdfd565 - - -] [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] Live migration failed. 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] Traceback (most recent call last): 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5108, in _do_live_migration 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] block_migration, migrate_data) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5431, in live_migration 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] migrate_data) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6071, in _live_migration 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] dom, finish_event) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5919, in _live_migration_monitor 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] dom.abortJob() 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] result = proxy_call(self._autowrap, f, *args, **kwargs) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] rv = execute(f, *args, **kwargs) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] six.reraise(c, e, tb) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] rv = meth(*args, **kwargs) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 482, in abortJob 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] if ret == -1: raise libvirtError ('virDomainAbortJob() failed', dom=self) 2015-12-14 08:19:51.577 23821 ERROR nova.compute.manager [instance: babf696c-55d1-4bde-be83-3124be2ac7f2] libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3) I'm not sure if this is the same bug as https://bugs.launchpad.net/nova/+bug/1419577 but I want to make sure to bring some light into this area since this affect our operations very much. Best regards To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1525802/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
