[openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever
Hi all. When I try to fix a bug:https://bugs.launchpad.net/nova/+bug/1242961, I get a trouble. To reproduce the bug is very easy. Live migrate a vm in block_migration mode, and then delelte the vm immediately. The reason of this bug is as follow: 1. Because live migrate costs more time, so the vm will be deleted sucessfully before live migrate complete. And then, we will get an exception while live migrating. 2. After live migrate failed, we start to rollback. But, in the rollback method we will get or modify the info of vm from db. Because the vm has been deleted already, so we will get instance_not_found exception and rollback will be faild too. I have two ways to fix the bug: i)Add check in nova-api. When try to delete a vm, we return an error message if the vm_state is LIVE_MIGRATING. This way is very simple, but need to carefully consider. I have found a related discussion: http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html, but it has no result in the discussion. ii)Before live migrate we get all the data needed by rollback method, and add a new rollback method. The new method will clean up resources at destination based on the above data(The resouces at source has been already cleaned up by deleting). I have no idea whitch one I should choose. Or, any other ideas?:) Regards, wanghong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever
Isn’t just handling the exception instance_not_found enough? By this time source would’ve been cleaned up. Destination VM resources will get cleaned up by the periodic task since the VM is not associated with this host. Am I missing something here? From: 王宏 [mailto:w.wangho...@gmail.com] Sent: 16 December 2013 11:32 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Hi all. When I try to fix a bug:https://bugs.launchpad.net/nova/+bug/1242961, I get a trouble. To reproduce the bug is very easy. Live migrate a vm in block_migration mode, and then delelte the vm immediately. The reason of this bug is as follow: 1. Because live migrate costs more time, so the vm will be deleted sucessfully before live migrate complete. And then, we will get an exception while live migrating. 2. After live migrate failed, we start to rollback. But, in the rollback method we will get or modify the info of vm from db. Because the vm has been deleted already, so we will get instance_not_found exception and rollback will be faild too. I have two ways to fix the bug: i)Add check in nova-api. When try to delete a vm, we return an error message if the vm_state is LIVE_MIGRATING. This way is very simple, but need to carefully consider. I have found a related discussion: http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html, but it has no result in the discussion. ii)Before live migrate we get all the data needed by rollback method, and add a new rollback method. The new method will clean up resources at destination based on the above data(The resouces at source has been already cleaned up by deleting). I have no idea whitch one I should choose. Or, any other ideas?:) Regards, wanghong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever
I would block it in the API or have the API cancelling the migration first. I don't see a reason why to start an operation that is meant to fail, which also has a complex chain of event, following it failure. Regardless of the above, I think that the suggested exception handling is needed in any case. Vladik - Original Message - From: Loganathan Parthipan parthi...@hp.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, 16 December, 2013 8:25:09 AM Subject: Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Isn’t just handling the exception instance_not_found enough? By this time source would’ve been cleaned up. Destination VM resources will get cleaned up by the periodic task since the VM is not associated with this host. Am I missing something here? From: 王宏 [mailto:w.wangho...@gmail.com] Sent: 16 December 2013 11:32 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Hi all. When I try to fix a bug: https://bugs.launchpad.net/nova/+bug/1242961 , I get a trouble. To reproduce the bug is very easy. Live migrate a vm in block_migration mode, and then delelte the vm immediately. The reason of this bug is as follow: 1. Because live migrate costs more time, so the vm will be deleted sucessfully before live migrate complete. And then, we will get an exception while live migrating. 2. After live migrate failed, we start to rollback. But, in the rollback method we will get or modify the info of vm from db. Because the vm has been deleted already, so we will get instance_not_found exception and rollback will be faild too. I have two ways to fix the bug: i)Add check in nova-api. When try to delete a vm, we return an error message if the vm_state is LIVE_MIGRATING. This way is very simple, but need to carefully consider. I have found a related discussion: http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html , but it has no result in the discussion. ii)Before live migrate we get all the data needed by rollback method, and add a new rollback method. The new method will clean up resources at destination based on the above data(The resouces at source has been already cleaned up by deleting). I have no idea whitch one I should choose. Or, any other ideas?:) Regards, wanghong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever
could we use Taskflow https://wiki.openstack.org/wiki/TaskFlow to manage task state and resource for this kind of tasks in Nova? Cinder has been an pilot to use Taskflow for volume backup tasks. anyone interested in this suggestion or has done some research to improve the live migration workflow? 2013/12/17 Vladik Romanovsky vladik.romanov...@enovance.com I would block it in the API or have the API cancelling the migration first. I don't see a reason why to start an operation that is meant to fail, which also has a complex chain of event, following it failure. Regardless of the above, I think that the suggested exception handling is needed in any case. Vladik - Original Message - From: Loganathan Parthipan parthi...@hp.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, 16 December, 2013 8:25:09 AM Subject: Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Isn’t just handling the exception instance_not_found enough? By this time source would’ve been cleaned up. Destination VM resources will get cleaned up by the periodic task since the VM is not associated with this host. Am I missing something here? From: 王宏 [mailto:w.wangho...@gmail.com] Sent: 16 December 2013 11:32 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Hi all. When I try to fix a bug: https://bugs.launchpad.net/nova/+bug/1242961 , I get a trouble. To reproduce the bug is very easy. Live migrate a vm in block_migration mode, and then delelte the vm immediately. The reason of this bug is as follow: 1. Because live migrate costs more time, so the vm will be deleted sucessfully before live migrate complete. And then, we will get an exception while live migrating. 2. After live migrate failed, we start to rollback. But, in the rollback method we will get or modify the info of vm from db. Because the vm has been deleted already, so we will get instance_not_found exception and rollback will be faild too. I have two ways to fix the bug: i)Add check in nova-api. When try to delete a vm, we return an error message if the vm_state is LIVE_MIGRATING. This way is very simple, but need to carefully consider. I have found a related discussion: http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html, but it has no result in the discussion. ii)Before live migrate we get all the data needed by rollback method, and add a new rollback method. The new method will clean up resources at destination based on the above data(The resouces at source has been already cleaned up by deleting). I have no idea whitch one I should choose. Or, any other ideas?:) Regards, wanghong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Tang Yaguang Canonical Ltd. | www.ubuntu.com | www.canonical.com Mobile: +86 152 1094 6968 gpg key: 0x187F664F ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever
Hi! Actually, I have already filed cancel of LiveMigration with using taskflow [0]. But my approach to blueprint was not so good at that time. So, I rewrote this blueprint from also point of russelb. I want to repush and wanted to approved. If you have any suggestions, ideas etc... I appreciate it :) Sincerely, Haruka Tanizawa [0] https://blueprints.launchpad.net/nova/+spec/feature-of-cancel 2013/12/17 Yaguang Tang yaguang.t...@canonical.com could we use Taskflow https://wiki.openstack.org/wiki/TaskFlow to manage task state and resource for this kind of tasks in Nova? Cinder has been an pilot to use Taskflow for volume backup tasks. anyone interested in this suggestion or has done some research to improve the live migration workflow? 2013/12/17 Vladik Romanovsky vladik.romanov...@enovance.com I would block it in the API or have the API cancelling the migration first. I don't see a reason why to start an operation that is meant to fail, which also has a complex chain of event, following it failure. Regardless of the above, I think that the suggested exception handling is needed in any case. Vladik - Original Message - From: Loganathan Parthipan parthi...@hp.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, 16 December, 2013 8:25:09 AM Subject: Re: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Isn’t just handling the exception instance_not_found enough? By this time source would’ve been cleaned up. Destination VM resources will get cleaned up by the periodic task since the VM is not associated with this host. Am I missing something here? From: 王宏 [mailto:w.wangho...@gmail.com] Sent: 16 December 2013 11:32 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Nova][libvirt]when deleting instance which is in migrating state, instance files can be stay in destination node forever Hi all. When I try to fix a bug: https://bugs.launchpad.net/nova/+bug/1242961 , I get a trouble. To reproduce the bug is very easy. Live migrate a vm in block_migration mode, and then delelte the vm immediately. The reason of this bug is as follow: 1. Because live migrate costs more time, so the vm will be deleted sucessfully before live migrate complete. And then, we will get an exception while live migrating. 2. After live migrate failed, we start to rollback. But, in the rollback method we will get or modify the info of vm from db. Because the vm has been deleted already, so we will get instance_not_found exception and rollback will be faild too. I have two ways to fix the bug: i)Add check in nova-api. When try to delete a vm, we return an error message if the vm_state is LIVE_MIGRATING. This way is very simple, but need to carefully consider. I have found a related discussion: http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html, but it has no result in the discussion. ii)Before live migrate we get all the data needed by rollback method, and add a new rollback method. The new method will clean up resources at destination based on the above data(The resouces at source has been already cleaned up by deleting). I have no idea whitch one I should choose. Or, any other ideas?:) Regards, wanghong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Tang Yaguang Canonical Ltd. | www.ubuntu.com | www.canonical.com Mobile: +86 152 1094 6968 gpg key: 0x187F664F ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
