Re: [Openstack] Nova doesn't release ips when terminating instances
force_dhcp_release=true should cause the ip to be released immediately, assuming the relevant optional binary from dnsmasq is installed (it is in the package dnsmasq-utils in ubuntu). The dhcp_release command does not appear to be packaged with Fedora. If it is set to false then the ips should be reclaimed after a set timeout period (ten minutes by default) via a periodic task in the network worker. If they are not being reclaimed properly then there is definitely a bug somewhere... It does not appear that ips are ever properly reclaimed. They will hang around with allocated=0 and instance_id != NULL forever, until I manually correct the database. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
On 06/26/2012 05:55 PM, Lars Kellogg-Stedman wrote: force_dhcp_release=true should cause the ip to be released immediately, assuming the relevant optional binary from dnsmasq is installed (it is in the package dnsmasq-utils in ubuntu). The dhcp_release command does not appear to be packaged with Fedora. It's in the dnsmasq-utils package on Fedora 16 and 17 The Fedora openstack packages auto install this package and enable the option to use it. Oh I see you're using CentOS 6.2. A rebuild of this would probably work: http://ftp.redhat.com/pub/redhat/linux/enterprise/6Workstation/en/os/SRPMS/dnsmasq-2.48-6.el6.src.rpm If it is set to false then the ips should be reclaimed after a set timeout period (ten minutes by default) via a periodic task in the network worker. If they are not being reclaimed properly then there is definitely a bug somewhere... It does not appear that ips are ever properly reclaimed. They will hang around with allocated=0 and instance_id != NULL forever, until I manually correct the database. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
On Jun 26, 2012, at 9:55 AM, Lars Kellogg-Stedman wrote: It does not appear that ips are ever properly reclaimed. They will hang around with allocated=0 and instance_id != NULL forever, until I manually correct the database. I just turned off force_dhcp_release on my install and it appears to be working. Do you see something like the following every sixty seconds in your network.log? 2012-06-26 17:52:38 DEBUG nova.manager [-] Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips from (pid=20993) periodic_tasks /opt/stack/nova/nova/manager.py:164 fixed_ip_disassociate_timeout defaults to 600 so ips should be reclaimed after 10 minutes unless you have changed the value of that option. Vish ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
On Tue, Jun 26, 2012 at 6:55 PM, Lars Kellogg-Stedman l...@seas.harvard.edu wrote: force_dhcp_release=true should cause the ip to be released immediately, assuming the relevant optional binary from dnsmasq is installed (it is in the package dnsmasq-utils in ubuntu). The dhcp_release command does not appear to be packaged with Fedora. It is in optional dnsmasq-utils subpackage in F17 and F16 updates https://admin.fedoraproject.org/updates/dnsmasq-2.59-4.fc16 Cheers, Alan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Do you see something like the following every sixty seconds in your
network.log?
2012-06-26 17:52:38 DEBUG nova.manager [-] Running periodic task
FlatDHCPManager._disassociate_stale_fixed_ips from (pid=20993) periodic_tasks
/opt/stack/nova/nova/manager.py:164
I do see these messages in the log (approximately once/minute, it looks like).
Here's a test:
- I boot and then deleted ('nova delete ...') an instance. This
leaves the database looking like this:
mysql select created_at,updated_at,allocated,instance_id
from fixed_ips where allocated=0 and instance_id is not null;
+-+-+---+-+
| created_at | updated_at | allocated | instance_id |
+-+-+---+-+
| 2012-06-21 20:26:45 | 2012-06-26 18:56:32 | 0 |2247 |
+-+-+---+-+
1 row in set (0.00 sec)
- I wait a while...
mysql select utc_time();
++
| utc_time() |
++
| 19:08:24 |
++
1 row in set (0.00 sec)
- But the ip still has an instance_id:
mysql select created_at,updated_at,allocated,instance_id
from fixed_ips where allocated=0 and instance_id is not null;
+-+-+---+-+
| created_at | updated_at | allocated | instance_id |
+-+-+---+-+
| 2012-06-21 20:26:45 | 2012-06-26 18:56:32 | 0 |2247 |
+-+-+---+-+
1 row in set (0.00 sec)
fixed_ip_disassociate_timeout defaults to 600 so ips should be reclaimed
after 10 minutes unless you have changed the value of that option.
That option appears to be set to the default of 600 seconds.
--
Lars Kellogg-Stedman l...@seas.harvard.edu |
Senior Technologist| http://ac.seas.harvard.edu/
Academic Computing |
http://code.seas.harvard.edu/
Harvard School of Engineering and Applied Sciences |
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
A rebuild of this would probably work: http://ftp.redhat.com/pub/redhat/linux/enterprise/6Workstation/en/os/SRPMS/dnsmasq-2.48-6.el6.src.rpm Thanks for the pointer! I'll drop that into our build system and see what comes out. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Found the issue. Fix here: https://review.openstack.org/9026 On Jun 26, 2012, at 12:13 PM, Lars Kellogg-Stedman wrote: A rebuild of this would probably work: http://ftp.redhat.com/pub/redhat/linux/enterprise/6Workstation/en/os/SRPMS/dnsmasq-2.48-6.el6.src.rpm Thanks for the pointer! I'll drop that into our build system and see what comes out. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Fix here: https://review.openstack.org/9026 That changes appears to be against nova/network/quantum/nova_ipam_lib.py. Is that also in the code path for non-Quantum users (specifically, people using the FlatDHCP model)? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Yessir. The issue (which was very annoying to track down) is the nova_ipam_lib is loaded by default, and it trickily was unsetting the timeout_fixed_ips setting of FlatDHCPManager with the (seemingly innocuous): self.net_manager.timeout_fixed_ips = not self.net_manager.DHCP Vish On Jun 26, 2012, at 1:46 PM, Lars Kellogg-Stedman wrote: Fix here: https://review.openstack.org/9026 That changes appears to be against nova/network/quantum/nova_ipam_lib.py. Is that also in the code path for non-Quantum users (specifically, people using the FlatDHCP model)? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
The issue (which was very annoying to track down) is the nova_ipam_lib is loaded by default, and it trickily was unsetting the timeout_fixed_ips setting of FlatDHCPManager with the (seemingly innocuous): self.net_manager.timeout_fixed_ips = not self.net_manager.DHCP Got it. I can confirm that it has fixed our problem with addresses not being released. Thanks! -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
force_dhcp_release=true should cause the ip to be released immediately, assuming the relevant optional binary from dnsmasq is installed (it is in the package dnsmasq-utils in ubuntu). If it is set to false then the ips should be reclaimed after a set timeout period (ten minutes by default) via a periodic task in the network worker. If they are not being reclaimed properly then there is definitely a bug somewhere, although I have not seen this happening recently myself. On Jun 23, 2012 6:14 PM, Lars Kellogg-Stedman l...@seas.harvard.edu wrote: can you try with the flag: force_dhcp_release=false It turns out I already had force_dhcp_release set to False. I've opened https://bugs.launchpad.net/nova/+bug/1017013 on this issue. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Hi, can you try with the flag: force_dhcp_release=false But first you need to: Manually set instance_id=NULL in the fixed_ips table allows things to work again. Belmiro CERN On Jun 23, 2012, at 5:11 AM, Lars Kellogg-Stedman wrote: When an instance terminates, the allocated field in the fixed_ips is set to 0, but the instance_id field remains set. Once all the addresses are in this state, new instances fail to start, and the following error is logged by nova-network: 2012-06-22 23:09:34 ERROR nova.rpc.amqp [req-1fea207d-cd65-4375-9a04-17ba1ab92e3e 22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] Returning exception Zero fixed ips available. to caller Which shows up in compute.log as: 2012-06-22 23:08:35 TRACE nova.rpc.amqp RemoteError: Remote error: NoMoreFixedIps Zero fixed ips available. Manually set instance_id=NULL in the fixed_ips table allows things to work again. We're running the 2012.1.1 release and we're using the FlatDHCP model. Is this a known bug? Thanks, -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Seems like this might be a pretty valid issue to report/fix though, no? I would assume those IPs should be unallocated/released if they are no longer attached to anything... John Postlethwait Nebula, Inc. 206-999-4492 On Saturday, June 23, 2012 at 11:06 AM, Belmiro Moreira wrote: Hi, can you try with the flag: force_dhcp_release=false But first you need to: Manually set instance_id=NULL in the fixed_ips table allows things to work again. Belmiro CERN On Jun 23, 2012, at 5:11 AM, Lars Kellogg-Stedman wrote: When an instance terminates, the allocated field in the fixed_ips is set to 0, but the instance_id field remains set. Once all the addresses are in this state, new instances fail to start, and the following error is logged by nova-network: 2012-06-22 23:09:34 ERROR nova.rpc.amqp [req-1fea207d-cd65-4375-9a04-17ba1ab92e3e 22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] Returning exception Zero fixed ips available. to caller Which shows up in compute.log as: 2012-06-22 23:08:35 TRACE nova.rpc.amqp RemoteError: Remote error: NoMoreFixedIps Zero fixed ips available. Manually set instance_id=NULL in the fixed_ips table allows things to work again. We're running the 2012.1.1 release and we're using the FlatDHCP model. Is this a known bug? Thanks, -- Lars Kellogg-Stedman l...@seas.harvard.edu (mailto:l...@seas.harvard.edu) | Senior Technologist | http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
can you try with the flag: force_dhcp_release=false I've been looking unsuccessfully for documentation on this option...so I'm not sure exactly what it does. However, if I understand https://bugs.launchpad.net/nova/+bug/953712 correctly, it requires the dhcp_release command, which is sometimes part of the dnsmasq package. I'm running on CentOS 6.2 and dhcp_release does not appear to exist. On the other hand, simple database manipulation seems to resolve the problem, so would everything work just fine if I added a no-op dhcp_release command? I'll give this a shot later tonight. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
On Sat, Jun 23, 2012 at 11:55:33AM -0700, John Postlethwait wrote: Seems like this might be a pretty valid issue to report/fix though, no? Is that a question for me or the list? I certainly think it should be fixed, but I wanted to check here before posting a bug report (in case someone said, oh, it's supposed to work that way, you need to set the nova_act_sane_please configuration option to change the behavior). Given the two responses here I will open a bug report later this evening. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Same hapenned to us. Adding de dhcp flag on false using flat injected the instance field on the fixed_ips table at the DB get NULLED ok ... Without the flags this not happen ...as the user reported On Jun 23, 2012 7:17 PM, Lars Kellogg-Stedman l...@seas.harvard.edu wrote: can you try with the flag: force_dhcp_release=false I've been looking unsuccessfully for documentation on this option...so I'm not sure exactly what it does. However, if I understand https://bugs.launchpad.net/nova/+bug/953712 correctly, it requires the dhcp_release command, which is sometimes part of the dnsmasq package. I'm running on CentOS 6.2 and dhcp_release does not appear to exist. On the other hand, simple database manipulation seems to resolve the problem, so would everything work just fine if I added a no-op dhcp_release command? I'll give this a shot later tonight. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
can you try with the flag: force_dhcp_release=false It turns out I already had force_dhcp_release set to False. I've opened https://bugs.launchpad.net/nova/+bug/1017013 on this issue. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Nova doesn't release ips when terminating instances
When an instance terminates, the allocated field in the fixed_ips is set to 0, but the instance_id field remains set. Once all the addresses are in this state, new instances fail to start, and the following error is logged by nova-network: 2012-06-22 23:09:34 ERROR nova.rpc.amqp [req-1fea207d-cd65-4375-9a04-17ba1ab92e3e 22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] Returning exception Zero fixed ips available. to caller Which shows up in compute.log as: 2012-06-22 23:08:35 TRACE nova.rpc.amqp RemoteError: Remote error: NoMoreFixedIps Zero fixed ips available. Manually set instance_id=NULL in the fixed_ips table allows things to work again. We're running the 2012.1.1 release and we're using the FlatDHCP model. Is this a known bug? Thanks, -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
