Public bug reported:
In router rescheduling looping task db call to get down bindings is done
outside of try/except block which may cause task to fail (see traceback
below). Need to put db operation inside try/except.
2016-02-15T10:44:44.259995+00:00 err: 2016-02-15 10:44:44.250 15419 ERROR
oslo.service.loopingcall [req-79bce4c3-2e81-446c-8b37-6d30e3a964e2 - - - - -]
Fixed interval looping call
'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin.reschedule_routers_from_down_agents'
failed
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall Traceback (most
recent call last):
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in
_run_loop
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall result =
func(*self.args, **self.kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line
101, in reschedule_routers_from_down_agents
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall down_bindings =
self._get_down_bindings(context, cutoff)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_dvrscheduler_db.py", line 460,
in _get_down_bindings
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall context, cutoff)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line
149, in _get_down_bindings
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return query.all()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2399, in all
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return list(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in
__iter__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
self._execute_and_instances(context)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in
_execute_and_instances
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
close_with_result=True)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in
_connection_from_session
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall **kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in
connection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
execution_options=execution_options)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in
_connection_for_bind
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall conn =
engine.contextual_connect(**kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in
contextual_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
self._wrap_pool_connect(self.pool.connect, None),
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in
_wrap_pool_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall e, dialect, self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in
_handle_dbapi_exception_noconnection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
util.raise_from_cause(newraise, exc_info)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
reraise(type(exception), exception, tb=exc_tb)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in
_wrap_pool_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return fn()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
_ConnectionFairy._checkout(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall fairy =
_ConnectionRecord.checkout(pool)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall rec.checkin()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in
__exit__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
compat.reraise(exc_type, exc_value, exc_tb)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall dbapi_connection =
rec.get_connection()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 594, in
get_connection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall self.connection =
self.__connect()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall connection =
self.__pool._invoke_creator(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in
connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
dialect.connect(*cargs, **cparams)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in
connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
self.dbapi.connect(*cargs, **cparams)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
Connection(*args, **kwargs)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 206, in __init__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall super(Connection,
self).__init__(*args, **kwargs2)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall DBConnectionError:
(_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at
'reading initial communication packet', system error: 0")
** Affects: neutron
Importance: Medium
Assignee: Oleg Bondarev (obondarev)
Status: New
** Tags: l3-ipam-dhcp liberty-backport-potential
** Tags added: liberty-backport-potential
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1546110
Title:
DB error causes router rescheduling loop to fail
Status in neutron:
New
Bug description:
In router rescheduling looping task db call to get down bindings is
done outside of try/except block which may cause task to fail (see
traceback below). Need to put db operation inside try/except.
2016-02-15T10:44:44.259995+00:00 err: 2016-02-15 10:44:44.250 15419 ERROR
oslo.service.loopingcall [req-79bce4c3-2e81-446c-8b37-6d30e3a964e2 - - - - -]
Fixed interval looping call
'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin.reschedule_routers_from_down_agents'
failed
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall Traceback (most
recent call last):
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in
_run_loop
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall result =
func(*self.args, **self.kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line
101, in reschedule_routers_from_down_agents
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall down_bindings =
self._get_down_bindings(context, cutoff)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_dvrscheduler_db.py", line 460,
in _get_down_bindings
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall context, cutoff)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line
149, in _get_down_bindings
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
query.all()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2399, in all
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return list(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in
__iter__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
self._execute_and_instances(context)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in
_execute_and_instances
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
close_with_result=True)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in
_connection_from_session
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall **kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in
connection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
execution_options=execution_options)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in
_connection_for_bind
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall conn =
engine.contextual_connect(**kw)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in
contextual_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
self._wrap_pool_connect(self.pool.connect, None),
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in
_wrap_pool_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall e, dialect, self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in
_handle_dbapi_exception_noconnection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
util.raise_from_cause(newraise, exc_info)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
reraise(type(exception), exception, tb=exc_tb)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in
_wrap_pool_connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return fn()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
_ConnectionFairy._checkout(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall fairy =
_ConnectionRecord.checkout(pool)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall rec.checkin()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in
__exit__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
compat.reraise(exc_type, exc_value, exc_tb)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall dbapi_connection
= rec.get_connection()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 594, in
get_connection
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall self.connection
= self.__connect()
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall connection =
self.__pool._invoke_creator(self)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in
connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
dialect.connect(*cargs, **cparams)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in
connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
self.dbapi.connect(*cargs, **cparams)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return
Connection(*args, **kwargs)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File
"/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 206, in __init__
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
super(Connection, self).__init__(*args, **kwargs2)
2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall
DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection
to MySQL server at 'reading initial communication packet', system error: 0")
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1546110/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
