Reviewed:  https://review.openstack.org/275650
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=30d5d805c10b0cc6e474fe1292b2c6549fc07d33
Submitter: Jenkins
Branch:    master

commit 30d5d805c10b0cc6e474fe1292b2c6549fc07d33
Author: ShaoHe Feng <shaohe.f...@intel.com>
Date:   Tue Feb 2 09:41:33 2016 +0000

    reset task_state after select_destinations failed.
    
    During live migration, there maybe exception when let scheduler
    select destination, and live migration will abort. But the task
    state of the instance still keep migrating, then we can not take
    any action on this instance.
    
    We need to recover the state of the task as None.
    We should also recover the vm_state.
    
    Change-Id: If1cae8f4c9037f7821554a94d4440f66d9538794
    Closes-bug: #1536916


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1536916

Title:
  nova's task_state still is migrating after live migration failed

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Environment:
  distribution:
  $ cat /etc/lsb-release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=14.04
  DISTRIB_CODENAME=trusty
  DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

  $ sudo virsh version
  Compiled against library: libvirt 1.2.2
  Using library: libvirt 1.2.2
  Using API: QEMU 1.2.2
  Running hypervisor: QEMU 2.0.0

  $ git log --oneline
  806113e Merge "Changed filter_by() to filter() during filtering instances in 
db API"
  ...

  There are two hosts in my environment.

  Host A is controller with compute node.  Host B is only as  compute
  node.

  Produce:

  1. I upgrade the nova code in Host A.  and restart  n-sch, n-cond, n-cpu.
     the nova code in Host B, and restart n-cpu.
     keep n-sch as old version.

  
  2. do live migration
  $ nova live-migration tt
  it report error. "Remote error: UnsupportedVersion Endpoint does not support 
RPC version 4.3. Attempted method: select_destinations
  $ sudo virsh list                                                             
                                                         x~~~~~~~~
   Id    Name                           State                                   
                                                                                
     x~~~~~~~~
  ----------------------------------------------------                          
                                                                                
     x~~~~~~~~
   26    instance-00000002              running                                 
                                                                                
     x~~~~~~~~

  error details as follow:
  
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------
  
----------------------------------------------------------------------------------+
  | Property                             | Value
                                                                                
    |
  
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------
  
----------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL
                                                                                
    |
  | OS-EXT-AZ:availability_zone          | nova
                                                                                
    |
  | OS-EXT-SRV-ATTR:host                 | shaohe1
                                                                                
    |
  | OS-EXT-SRV-ATTR:hostname             | tt                                   
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | shaohe1                              
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000002                    
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:kernel_id            | 89e91cc1-a40c-4c9f-bcfa-37b0e94d5f57 
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:launch_index         | 0                                    
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:ramdisk_id           | d4c09101-c0fa-40de-9c88-28b9428d03fb 
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:reservation_id       | r-p0884maw                           
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                             
                                                                                
                                                                                
                |
  | OS-EXT-SRV-ATTR:user_data            | -                                    
                                                                                
                                                                                
                |
  | OS-EXT-STS:power_state               | 1                                    
                                                                                
                                                                                
                |
  | OS-EXT-STS:task_state                | migrating                            
                                                                                
                                                                                
                |
  | OS-EXT-STS:vm_state                  | error                                
                                                                                
                                                                                
                |
  | OS-SRV-USG:launched_at               | 2015-12-18T07:41:00.000000           
                                                                                
                                                                                
                |
  | OS-SRV-USG:terminated_at             | -                                    
                                                                                
                                                                                
                |
  | accessIPv4                           |                                      
                                                                                
                                                                                
                |
  | accessIPv6                           |                                      
                                                                                
                                                                                
                |
  | config_drive                         | True                                 
                                                                                
                                                                                
                |
                                                                                
    |
  | created                              | 2015-12-18T07:40:44Z                 
                                                                                
                                                                                
                |
  | fault                                | {"message": "Remote error: 
UnsupportedVersion Endpoint does not support RPC version 4.3. Attempted method: 
select_destinations                                                             
                          |
  |                                      | [u'Traceback (most recent call 
last):\                                                                         
                                                                                
                      |
  |                                      | ', u'  File 
\"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\", 
line 142, in _d", "code": 500, "details": "  File 
\"/opt/stack/nova/nova/conductor/manager.py\", line 295, in _live_migrate |
  |                                      |     task.execute()                   
                                                                                
                                                                                
                |
  |                                      |   File 
\"/opt/stack/nova/nova/conductor/tasks/base.py\", line 27, in wrap              
                                                                                
                                              |
  |                                      |     self.rollback()                  
                                                                                
                                                                                
                |
  |                                      |   File 
\"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 204, in 
__exit__                                                                        
                                              |
  |                                      |     six.reraise(self.type_, 
self.value, self.tb)                                                            
                                                                                
                         |
  |                                      |   File 
\"/opt/stack/nova/nova/conductor/tasks/base.py\", line 24, in wrap              
                                                                                
                                              |
  |                                      |     return original(self)            
                                                                                
                                                                                
                |
  |                                      |   File 
\"/opt/stack/nova/nova/conductor/tasks/base.py\", line 42, in execute           
                                                                                
                                              |
  |                                      |     return self._execute()           
                                                                                
                                                                                
                |
  |                                      |   File 
\"/opt/stack/nova/nova/conductor/tasks/live_migrate.py\", line 58, in _execute  
                                                                                
                                              |
  |                                      |     self.destination = 
self._find_destination()                                                        
                                                                                
                              |
  |                                      |   File 
\"/opt/stack/nova/nova/conductor/tasks/live_migrate.py\", line 181, in 
_find_destination                                                               
                                                       |
  |                                      |     spec_obj)[0]['host']             
                                                                                
                                                                                
                |
  |                                      |   File 
\"/opt/stack/nova/nova/scheduler/utils.py\", line 358, in wrapped               
                                                                                
                                              |
  |                                      |     return func(*args, **kwargs)     
                                                                                
                                                                                
                |
  |                                      |   File 
\"/opt/stack/nova/nova/scheduler/client/__init__.py\", line 51, in 
select_destinations                                                             
                                                           |
  |                                      |     return 
self.queryclient.select_destinations(context, spec_obj)                         
                                                                                
                                          |
  |                                      |   File 
\"/opt/stack/nova/nova/scheduler/client/__init__.py\", line 37, in __run_method 
                                                                                
                                              |
  |                                      |     return getattr(self.instance, 
__name)(*args, **kwargs)
                                                                                
    |
  |                                      |   File 
\"/opt/stack/nova/nova/scheduler/client/query.py\", line 32, in 
select_destinations                                                             
                                                              |
  |                                      |     return 
self.scheduler_rpcapi.select_destinations(context, spec_obj)                    
                                                                                
                                          |
  |                                      |   File 
\"/opt/stack/nova/nova/scheduler/rpcapi.py\", line 121, in select_destinations  
                                                                                
                                              |
  |                                      |     return cctxt.call(ctxt, 
'select_destinations', **msg_args)                                              
                                                                                
                         |
  |                                      |   File 
\"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py\", line 
158, in call                                                                    
                                                |
  |                                      |     retry=self.retry)                
                                                                                
                                                                                
                |
  |                                      |   File 
\"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py\", line 
90, in _send                                                                    
                                                 |
  |                                      |     timeout=timeout, retry=retry)    
                                                                                
                                                                                
                |
  |                                      |   File 
\"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py\",
 line 464, in send                                                              
                                             |
  |                                      |     retry=retry)                     
                                                                                
                                                                                
                |
  |                                      |   File 
\"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py\",
 line 455, in _send                                                             
                                             |
  |                                      |     raise result                     
                                                                                
                                                                                
                |
  |                                      | ", "created": 
"2016-01-22T03:41:21Z"}                                                         
                                                                                
                                       |
  | flavor                               | m1.tiny (1)                          
                                                                                
                                                                                
                |
  | hostId                               | 
94170a2af829a4a136f9f9c9d3c3f8f10df72bc1238a5eeef5b4ffa5                        
                                                                                
                                                     |
  | id                                   | 058fc419-a8a8-4e08-b62c-a9841ef9cd3f 
                                                                                
                                                                                
                |
  | image                                | cirros-0.3.4-x86_64-uec 
(3476ee05-b80e-4a80-aac4-78350ce132a2)                                          
                                                                                
                             |
  | key_name                             | -                                    
                                                                                
                                                                                
                |
  | metadata                             | {}                                   
                                                                                
                                                                                
                |
  | name                                 | tt                                   
                                                                                
                                                                                
                |
  | os-extended-volumes:volumes_attached | []                                   
                                                                                
                                                                                
                |
  | private network                      | fdf1:3ce8:94a:0:f816:3eff:fee1:2a95, 
10.0.0.4                                                                        
                                                                                
                |
  | security_groups                      | default
                                                                                
    |
  | status                               | ERROR                                
                                                                                
                                                                                
                |
  | tenant_id                            | f5a8829cc14c4825a2728b273aa91aa1     
                                                                                
                                                                                
                |
  | updated                              | 2016-01-22T03:41:21Z                 
                                                                                
                                                                                
                |
  | user_id                              | b5450f0c30154d2bb1506968a05c6f80     
                                                                                
                                                                                
                |
  
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  3. when I do live migration again.
  $ nova live-migration tt
  ERROR (Conflict): Cannot 'os-migrateLive' instance 
058fc419-a8a8-4e08-b62c-a9841ef9cd3f while it is in vm_state error (HTTP 409) 
(Request-ID: req-2712a326-c3ab-4a6d-9afb-5001b3822047)

  4.  and stop the server.
  $ nova stop tt
  Cannot 'stop' instance 058fc419-a8a8-4e08-b62c-a9841ef9cd3f while it is in 
task_state migrating (HTTP 409) (Request-ID: 
req-580ae847-1a5d-44c8-a285-7cc31dc9e0e2)
  ERROR (CommandError): Unable to stop the specified server(s).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1536916/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to