Public bug reported: If exceptions raise during neutron-server initiation, will result in lots of fanout queues leak in RabbitMQ. Since these fanout queues will continue to receive message, the size of it will grow infinitely until RabbitMQ refuse to accept new message.
Reproduction ============== In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py : class OVSNeutronAgent(.....): def __init__(self, integ_br, tun_br, local_ip......): .... .... self.setup_rpc() .... raise Exception("boo") ## <<< insert a fake exception here ...... self.connection.consume_in_thread() Start neutron-openvswitch-agent server, it will quit because the exception. check out the RabbitMQ queues: $ sudo rabbitmqctl list_queues|grep fanout q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90 0 q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0 q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0 0 q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399 0 q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08 0 Although we are using fake exception in this case, in real world, there are many cases that will raise exception between self.setup_rpc() and consume_in_thread(). Even neutron-openvswitch-agent recuperated, these fanout queue will continue receive messages that from neutron-server, result in certain infinitely growing queue there. ** Affects: neutron Importance: Undecided Status: New ** Description changed: If exceptions raise during neutron-server initiation, will result in lots of fanout queues leak in RabbitMQ. Since these fanout queues will continue to receive message, the size of it will grow infinitely until RabbitMQ refuse to accept new message. Reproduction ============== In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py : - class OVSNeutronAgent(.....): - def __init__(self, integ_br, tun_br, local_ip......): - .... - .... - self.setup_rpc() - .... - raise Exception("boo") ## <<< insert a fake exception here - ...... - self.connection.consume_in_thread() + class OVSNeutronAgent(.....): + def __init__(self, integ_br, tun_br, local_ip......): + .... + .... + self.setup_rpc() + .... + raise Exception("boo") ## <<< insert a fake exception here + ...... + self.connection.consume_in_thread() - - Start neutron-openvswitch-agent server, it will quite because the exception. + Start neutron-openvswitch-agent server, it will quit because the + exception. check out the RabbitMQ queues: $ sudo rabbitmqctl list_queues|grep fanout q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90 0 q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0 q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0 0 q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399 0 q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08 0 - - Although we are using fake exception in this case, in real world, there are many cases that will raise exception between self.setup_rpc() and consume_in_thread(). Even neutron-openvswitch-agent recuperated, these fanout queue will continue receive messages that from neutron-server, result in certain infinitely growing queue there. + Although we are using fake exception in this case, in real world, there + are many cases that will raise exception between self.setup_rpc() and + consume_in_thread(). Even neutron-openvswitch-agent recuperated, these + fanout queue will continue receive messages that from neutron-server, + result in certain infinitely growing queue there. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1434378 Title: openvswitch-agent leak fanout queues Status in OpenStack Neutron (virtual network service): New Bug description: If exceptions raise during neutron-server initiation, will result in lots of fanout queues leak in RabbitMQ. Since these fanout queues will continue to receive message, the size of it will grow infinitely until RabbitMQ refuse to accept new message. Reproduction ============== In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py : class OVSNeutronAgent(.....): def __init__(self, integ_br, tun_br, local_ip......): .... .... self.setup_rpc() .... raise Exception("boo") ## <<< insert a fake exception here ...... self.connection.consume_in_thread() Start neutron-openvswitch-agent server, it will quit because the exception. check out the RabbitMQ queues: $ sudo rabbitmqctl list_queues|grep fanout q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90 0 q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0 q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0 0 q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399 0 q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08 0 Although we are using fake exception in this case, in real world, there are many cases that will raise exception between self.setup_rpc() and consume_in_thread(). Even neutron-openvswitch- agent recuperated, these fanout queue will continue receive messages that from neutron-server, result in certain infinitely growing queue there. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1434378/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp