[Yahoo-eng-team] [Bug 1376211] Re: Retry mechanism does not work on startup when used with MySQL
The fix was ported to Juno here - https://review.openstack.org/#/c/126732/ ** Changed in: oslo.db/juno Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1376211 Title: Retry mechanism does not work on startup when used with MySQL Status in neutron: Invalid Status in oslo.db: Fix Released Status in oslo.db juno series: Fix Released Bug description: This is initially revealed as Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144181 The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1. This is because during engine initialization, "engine.execute("SHOW VARIABLES LIKE 'sql_mode'")" is called, which opens the connection, *before* _test_connection() succeeds. So the server just bail out to sys.exit() at the top of the stack. This behaviour was checked for both oslo.db 0.4.0 and 1.0.1. I suspect this is a regression from the original db code from oslo- incubator though I haven't checked it specifically. The easiest way to reproduce the traceback is: 1. stop MariaDB. 2. execute the following Python script: ''' import oslo.db.sqlalchemy.session url = 'mysql://neutron:123456@10.35.161.235/neutron' engine = oslo.db.sqlalchemy.session.EngineFacade(url) ''' The following traceback can be seen in service log: 2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last): 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/bin/neutron-server", line 10, in 2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main()) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 47, in main 2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 105, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__ 2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron service.start() 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start 2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 168, in _run_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron app = config.load_paste_app(app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 182, in load_paste_app 2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.loadapp("config:%s" % config_path, name=app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2014-10-01 13:46:10.588 5812 TRACE neutron return context.create() 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_type.invoke(self) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2014-10-01 13:46:10.588 5812 TRACE neutron **context.local_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory 2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.get_app(app_name, global_conf=global_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2014-10-01 13:46:10.588 5812 TRACE neutron name=name,
[Yahoo-eng-team] [Bug 1376211] Re: Retry mechanism does not work on startup when used with MySQL
Per discussion above, marking this bug as Invalid for Neutron. It looks to be fixed in oslo.db and the bug was in fact in oslo.db. ** Changed in: neutron Status: Confirmed = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1376211 Title: Retry mechanism does not work on startup when used with MySQL Status in OpenStack Neutron (virtual network service): Invalid Status in Oslo Database library: Fix Released Status in oslo.db juno series: Confirmed Bug description: This is initially revealed as Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144181 The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1. This is because during engine initialization, engine.execute(SHOW VARIABLES LIKE 'sql_mode') is called, which opens the connection, *before* _test_connection() succeeds. So the server just bail out to sys.exit() at the top of the stack. This behaviour was checked for both oslo.db 0.4.0 and 1.0.1. I suspect this is a regression from the original db code from oslo- incubator though I haven't checked it specifically. The easiest way to reproduce the traceback is: 1. stop MariaDB. 2. execute the following Python script: ''' import oslo.db.sqlalchemy.session url = 'mysql://neutron:123456@10.35.161.235/neutron' engine = oslo.db.sqlalchemy.session.EngineFacade(url) ''' The following traceback can be seen in service log: 2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last): 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/bin/neutron-server, line 10, in module 2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main()) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/server/__init__.py, line 47, in main 2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 105, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py, line 82, in __exit__ 2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 102, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron service.start() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 73, in start 2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 168, in _run_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron app = config.load_paste_app(app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/common/config.py, line 182, in load_paste_app 2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.loadapp(config:%s % config_path, name=app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 247, in loadapp 2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 272, in loadobj 2014-10-01 13:46:10.588 5812 TRACE neutron return context.create() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 710, in create 2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_type.invoke(self) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 144, in invoke 2014-10-01 13:46:10.588 5812 TRACE neutron **context.local_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/util.py, line 56, in fix_call 2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/urlmap.py, line 25, in urlmap_factory 2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.get_app(app_name, global_conf=global_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 350, in get_app 2014-10-01 13:46:10.588 5812
[Yahoo-eng-team] [Bug 1376211] Re: Retry mechanism does not work on startup when used with MySQL
** Changed in: oslo.db Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1376211 Title: Retry mechanism does not work on startup when used with MySQL Status in OpenStack Neutron (virtual network service): Confirmed Status in Oslo Database library: Fix Released Status in oslo.db juno series: Confirmed Bug description: This is initially revealed as Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144181 The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1. This is because during engine initialization, engine.execute(SHOW VARIABLES LIKE 'sql_mode') is called, which opens the connection, *before* _test_connection() succeeds. So the server just bail out to sys.exit() at the top of the stack. This behaviour was checked for both oslo.db 0.4.0 and 1.0.1. I suspect this is a regression from the original db code from oslo- incubator though I haven't checked it specifically. The easiest way to reproduce the traceback is: 1. stop MariaDB. 2. execute the following Python script: ''' import oslo.db.sqlalchemy.session url = 'mysql://neutron:123456@10.35.161.235/neutron' engine = oslo.db.sqlalchemy.session.EngineFacade(url) ''' The following traceback can be seen in service log: 2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last): 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/bin/neutron-server, line 10, in module 2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main()) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/server/__init__.py, line 47, in main 2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 105, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py, line 82, in __exit__ 2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 102, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron service.start() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 73, in start 2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 168, in _run_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron app = config.load_paste_app(app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/common/config.py, line 182, in load_paste_app 2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.loadapp(config:%s % config_path, name=app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 247, in loadapp 2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 272, in loadobj 2014-10-01 13:46:10.588 5812 TRACE neutron return context.create() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 710, in create 2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_type.invoke(self) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 144, in invoke 2014-10-01 13:46:10.588 5812 TRACE neutron **context.local_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/util.py, line 56, in fix_call 2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/urlmap.py, line 25, in urlmap_factory 2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.get_app(app_name, global_conf=global_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 350, in get_app 2014-10-01 13:46:10.588 5812 TRACE neutron name=name, global_conf=global_conf).create() 2014-10-01 13:46:10.588 5812 TRACE neutron File
[Yahoo-eng-team] [Bug 1376211] Re: Retry mechanism does not work on startup when used with MySQL
** Changed in: oslo.db Milestone: None = next-kilo ** Also affects: oslo.db/juno Importance: Undecided Status: New ** Changed in: oslo.db/juno Importance: Undecided = High ** Changed in: oslo.db/juno Status: New = Confirmed ** Changed in: oslo.db/juno Assignee: (unassigned) = Roman Podoliaka (rpodolyaka) ** Changed in: oslo.db/juno Milestone: None = next-juno -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1376211 Title: Retry mechanism does not work on startup when used with MySQL Status in OpenStack Neutron (virtual network service): Confirmed Status in Oslo Database library: In Progress Status in oslo.db juno series: Confirmed Bug description: This is initially revealed as Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144181 The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1. This is because during engine initialization, engine.execute(SHOW VARIABLES LIKE 'sql_mode') is called, which opens the connection, *before* _test_connection() succeeds. So the server just bail out to sys.exit() at the top of the stack. This behaviour was checked for both oslo.db 0.4.0 and 1.0.1. I suspect this is a regression from the original db code from oslo- incubator though I haven't checked it specifically. The easiest way to reproduce the traceback is: 1. stop MariaDB. 2. execute the following Python script: ''' import oslo.db.sqlalchemy.session url = 'mysql://neutron:123456@10.35.161.235/neutron' engine = oslo.db.sqlalchemy.session.EngineFacade(url) ''' The following traceback can be seen in service log: 2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last): 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/bin/neutron-server, line 10, in module 2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main()) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/server/__init__.py, line 47, in main 2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 105, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py, line 82, in __exit__ 2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 102, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron service.start() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 73, in start 2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 168, in _run_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron app = config.load_paste_app(app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/common/config.py, line 182, in load_paste_app 2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.loadapp(config:%s % config_path, name=app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 247, in loadapp 2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 272, in loadobj 2014-10-01 13:46:10.588 5812 TRACE neutron return context.create() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 710, in create 2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_type.invoke(self) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 144, in invoke 2014-10-01 13:46:10.588 5812 TRACE neutron **context.local_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/util.py, line 56, in fix_call 2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/urlmap.py, line 25, in urlmap_factory 2014-10-01 13:46:10.588 5812 TRACE neutron app =
[Yahoo-eng-team] [Bug 1376211] Re: Retry mechanism does not work on startup when used with MySQL
This bug affects neutron since after a reboot it might be possible that the service will fail at startup if it's started before the mysql service. ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1376211 Title: Retry mechanism does not work on startup when used with MySQL Status in OpenStack Neutron (virtual network service): New Status in Oslo Database library: New Bug description: This is initially revealed as Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144181 The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1. This is because during engine initialization, engine.execute(SHOW VARIABLES LIKE 'sql_mode') is called, which opens the connection, *before* _test_connection() succeeds. So the server just bail out to sys.exit() at the top of the stack. This behaviour was checked for both oslo.db 0.4.0 and 1.0.1. I suspect this is a regression from the original db code from oslo- incubator though I haven't checked it specifically. The easiest way to reproduce the traceback is: 1. stop MariaDB. 2. execute the following Python script: ''' import oslo.db.sqlalchemy.session url = 'mysql://neutron:123456@10.35.161.235/neutron' engine = oslo.db.sqlalchemy.session.EngineFacade(url) ''' The following traceback can be seen in service log: 2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last): 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/bin/neutron-server, line 10, in module 2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main()) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/server/__init__.py, line 47, in main 2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 105, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(_('Unrecoverable error: please check log ' 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py, line 82, in __exit__ 2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(self.type_, self.value, self.tb) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 102, in serve_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron service.start() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 73, in start 2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(self.app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/service.py, line 168, in _run_wsgi 2014-10-01 13:46:10.588 5812 TRACE neutron app = config.load_paste_app(app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/neutron/common/config.py, line 182, in load_paste_app 2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.loadapp(config:%s % config_path, name=app_name) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 247, in loadapp 2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 272, in loadobj 2014-10-01 13:46:10.588 5812 TRACE neutron return context.create() 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 710, in create 2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_type.invoke(self) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 144, in invoke 2014-10-01 13:46:10.588 5812 TRACE neutron **context.local_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/util.py, line 56, in fix_call 2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/urlmap.py, line 25, in urlmap_factory 2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.get_app(app_name, global_conf=global_conf) 2014-10-01 13:46:10.588 5812 TRACE neutron File /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py, line 350, in get_app 2014-10-01 13:46:10.588 5812 TRACE neutron name=name,