Public bug reported: when api_workers > 0 i got a TRACE in the logs, neutron server works, but only one server appears to be connected to db and working
2014-01-12 21:21:52.639 10512 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last): 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp **args) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 44, in dispatch 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp neutron_ctxt, version, method, namespace, **kwargs) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 186, in report_state 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp self.plugin.create_or_update_agent(context, agent_state) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 149, in create_or_update_agent 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context, agent['agent_type'], agent['host']) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 125, in _get_agent_by_type_and_host 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Agent.host == host).one() 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp ret = list(self) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__ 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp return self._execute_and_instances(context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp result = conn.execute(querycontext.statement, self._params) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp params) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp compiled_sql, distilled_params 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp cursor.execute(statement, parameters) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp OperationalError: (OperationalError) SSL connection has been closed unexpectedly 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp 'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents."binary" AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations \nFROM agents \nWHERE agents.agent_type = %(agent_type_1)s AND agents.host = %(host_1)s' {'host_1': u'kvm03d', 'agent_type_1': u'Open vSwitch agent'} i can reproduce the issue running on devstack with postgresl, if i switch to mysql it works fine ** Affects: neutron Importance: Undecided Status: New ** Tags: havana ** Tags added: havana -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1268423 Title: multiple api_wokers cannot connect to pgsql Status in OpenStack Neutron (virtual network service): New Bug description: when api_workers > 0 i got a TRACE in the logs, neutron server works, but only one server appears to be connected to db and working 2014-01-12 21:21:52.639 10512 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last): 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp **args) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 44, in dispatch 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp neutron_ctxt, version, method, namespace, **kwargs) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 186, in report_state 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp self.plugin.create_or_update_agent(context, agent_state) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 149, in create_or_update_agent 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context, agent['agent_type'], agent['host']) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 125, in _get_agent_by_type_and_host 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Agent.host == host).one() 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp ret = list(self) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__ 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp return self._execute_and_instances(context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp result = conn.execute(querycontext.statement, self._params) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp params) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp compiled_sql, distilled_params 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp context) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp cursor.execute(statement, parameters) 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp OperationalError: (OperationalError) SSL connection has been closed unexpectedly 2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp 'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents."binary" AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations \nFROM agents \nWHERE agents.agent_type = %(agent_type_1)s AND agents.host = %(host_1)s' {'host_1': u'kvm03d', 'agent_type_1': u'Open vSwitch agent'} i can reproduce the issue running on devstack with postgresl, if i switch to mysql it works fine To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1268423/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp