** Also affects: nova/kilo
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1444497
Title:
Instance doesn't get an address via DHCP (nova-network) because of
issue with live migration
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) kilo series:
New
Bug description:
When instance is migrated to another compute node, it's dhcp lease is not
removed from the first compute node even after instance termination.
If a new instance got the same IP which was present in the previous instance
created on the the first compute node where dhcp lease for this IP remains,
then the dnsmasq refuse DHCP request of the IP address for a new instance with
different MAC.
Steps to reproduce:
Scenario:
1. Create cluster (CentOS, nova-network with Flat-DHCP , Ceph for
images and volumes)
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
4. Deploy the cluster
5. Create a VM
6. Wait until the VM got IP address via DHCP (in VM console log)
7. Migrate the VM to another compute node.
8. Terminate the VM.
9. Repeat stages from 5 to 8 several times (in my case - 4..6
times was enough) until a new instance stops receiving IP address via DHCP.
10. Check dnsmasq-dhcp.log (/var/log/daemon.log on the compute
node) for messages like :
=============================================
2014-11-09T20:28:29.671344+00:00 warning: not using configured address
10.0.0.2 because it is leased to fa:16:3e:65:70:be
This means that:
I. An instance was created on the compute node-1 and got a dhcp lease:
==== nova-dhcpbridge.log
2014-11-09 20:12:03.811 27360 DEBUG nova.dhcpbridge [-] Called 'add' for mac
'fa:16:3e:65:70:be' with ip '10.0.0.2' main
/usr/lib/python2.6/site-packages/nova/cmd/dhcpbridge.py:135
II. When the instance was migrating from compute node-1 to node-3,
'dhcp_release' was not performed on compute node-1, please check the time range
in the logs : 2014-11-09 20:14:36-37
==== Running.log (node-1)
2014-11-09T20:14:36.647588+00:00 debug: cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf conntrack -D -r 10.0.0.2
### But there is missing a command like: sudo nova-rootwrap
/etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be
III. On the compute node-3, DHCP lease was added and it was successfully
removed when the instance was terminated:
==== Running.log (node-3)
2014-11-09T20:15:17.250243+00:00 debug: cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be
IV. When an another instance got the same address '10.0.0.2' and was
created on node-1, it didn't get IP address via DHCP:
==== Running.log (node-1)
2014-11-09T20:28:29.671344+00:00 warning: not using configured address
10.0.0.2 because it is leased to fa:16:3e:65:70:be
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1444497/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
