Re: [Openstack-operators] Security group rules not working on instances kilo

Thanks Kris, issue resolved after adding below lines to sysctl.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 appreciate your help, thanks a lot again. On Thu, Apr 21, 2016 at 8:25 PM, Kris G. Lindgren wrote: > Ma

Re: [Openstack-operators] Security group rules not working on instances kilo

Make sure that the bridges are being created (1 bridge per vm) they should be named close to the vm tap device name. Then make sure that you have bridge nf-call-* files enabled: http://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf Under hybrid mode what happens is a linux bri

[Openstack-operators] Security group rules not working on instances kilo

Hi, I am running into a issue where security group rules are not applying to instances when I create a new security group with default rules it should reject all incoming traffic but it is allowing everything without blocking here is my config for nova : security_group_api = neutron firewall_dri

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

See something similar with heartbeat seems like reconnection attempt fails 2016-04-21 15:27:01.294 6 DEBUG nova.openstack.common.loopingcall [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Dynamic looping call > sleeping for 60.00 seconds _inner /usr/lib/python2.7/site-packages/nova/openst

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

We are seeing issues only on client side as of now. But we do have net.ipv4.tcp_retries2 = 3 set Ajay From: "Edmund Rhudy (BLOOMBERG/ 731 LEX)" mailto:erh...@bloomberg.net>> Reply-To: "Edmund Rhudy (BLOOMBERG/ 731 LEX)" mailto:erh...@bloomberg.net>> Date: Thursday, April 21, 2016 at 12:11 PM To

[Openstack-operators] [Openstack] OpenStack Mitaka for Ubuntu 14.04 LTS and Ubuntu 16.04 LTS

Hi All, The Ubuntu OpenStack Engineering team is pleased to announce the general availability of OpenStack Mitaka in Ubuntu 16.04 LTS and for Ubuntu 14.04 LTS via the Ubuntu Cloud Archive. Ubuntu 14.04 LTS You can enable the Ubuntu Cloud Archive for OpenStack Mitaka on U

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Thanks Kris that’s good information will try out your suggestions Ajay From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>> Date: Thursday, April 21, 2016 at 12:08 PM To: Ajay Kalambur mailto:akala...@cisco.com>>, "openstack-operators@lists.openstack.org

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

We just use heartbeat. But from what I recall other people have good luck with both set. I would keep them if they are already set , maybe just dial down how aggressive they are. One thing I should mention is that if you have a large number of RPC workers, enabling heartbeats will increase cpu

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Are you seeing issues only on the client side, or anything on the broker side? We were having issues with nodes not successfully reconnecting and ended up making a number of changes on the broker side to improve resiliency (upgrading to RabbitMQ 3.5.5 or higher, reducing net.ipv4.tcp_retries2 to

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Do you recommend both or can I do away with the system timers and just keep the heartbeat? Ajay From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>> Date: Thursday, April 21, 2016 at 11:54 AM To: Ajay Kalambur mailto:akala...@cisco.com>>, "openstack-operators@lists.openstack.org

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Yea, that only fixes part of the issue. The other part is getting the openstack messaging code itself to figure out the connection its using is no longer valid. Heartbeats by itself solved 90%+ of our issues with rabbitmq and nodes being disconnected and never reconnecting. __

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Trying that now. I had aggressive system keepalive timers before net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 5 From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>> Date: Thursday, April 21, 2016 at 11:50 AM To: Ajay Kalambur mailto:akala..

Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Do you have rabbitmq/oslo messaging heartbeats enabled? If you aren't using heartbeats it will take a long time for the nova-compute agent to figure out that its actually no longer attached to anything. Heartbeat does periodic checks against rabbitmq and will catch this state and reconnect.

[Openstack-operators] [oslo]nova compute reconnection Issue Kilo

Hi I am seeing on Kilo if I bring down one contoller node sometimes some computes report down forever. I need to restart the compute service on compute node to recover. Looks like oslo is not reconnecting in nova-compute Here is the Trace from nova-compute 2016-04-19 20:25:39.090 6 TRACE nova.se

Re: [Openstack-operators] [OpenStack-Ansible] Liberty to Mitaka upgrade?

On 20 April 2016 at 04:27, Dale Baley wrote: > Is there a document or guide for upgrading from Liberty to Mitaka yet? > Hi Dale, The active work to test and implement any plays to assist with upgrades has not yet been implemented for Liberty->Mitaka. We hope to do this work by Newton Milestone-

[Openstack-operators] [Performance] Austin Performance Team working group session - please comment

Folks, we're going to have Performance Working Group session during the upcoming summit - here is the event description . Our team was kicked off during Mitaka Summit, so that was our first cycle :) Please attend the session