glanceclient has always been that way.... it's terrible. :( the nova message timeout looks like it's becauce nova-cert is not running, and it keeps trying to ping it. What actions were you doing that triggered that path?
On 04/16/2015 07:23 PM, Danny Choi (dannchoi) wrote: > When use devstack with Kilo, after stack.sh, the following tracebacks are > logged: > > localadmin@qa4:/opt/stack/logs$ grep -r Traceback * > g-api.log:2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client > Traceback (most recent call last): > g-api.log:2015-04-16 14:16:20.237 TRACE glance.registry.client.v1.client > Traceback (most recent call last): > g-api.log:2015-04-16 14:16:21.973 TRACE glance.registry.client.v1.client > Traceback (most recent call last): > g-api.log:2015-04-16 14:16:24.538 TRACE glance.registry.client.v1.client > Traceback (most recent call last): > g-api.log.2015-04-16-141127:2015-04-16 14:16:17.996 TRACE > glance.registry.client.v1.client Traceback (most recent call last): > g-api.log.2015-04-16-141127:2015-04-16 14:16:20.237 TRACE > glance.registry.client.v1.client Traceback (most recent call last): > g-api.log.2015-04-16-141127:2015-04-16 14:16:21.973 TRACE > glance.registry.client.v1.client Traceback (most recent call last): > g-api.log.2015-04-16-141127:2015-04-16 14:16:24.538 TRACE > glance.registry.client.v1.client Traceback (most recent call last): > n-api.log:2015-04-16 14:18:28.163 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log:2015-04-16 14:19:34.470 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log:2015-04-16 14:20:39.624 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log:2015-04-16 14:21:44.879 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log:2015-04-16 14:22:49.676 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log:2015-04-16 14:23:55.475 TRACE nova.api.openstack Traceback (most > recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:18:28.163 TRACE nova.api.openstack > Traceback (most recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:19:34.470 TRACE nova.api.openstack > Traceback (most recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:20:39.624 TRACE nova.api.openstack > Traceback (most recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:21:44.879 TRACE nova.api.openstack > Traceback (most recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:22:49.676 TRACE nova.api.openstack > Traceback (most recent call last): > n-api.log.2015-04-16-141127:2015-04-16 14:23:55.475 TRACE nova.api.openstack > Traceback (most recent call last): > > Traceback #1: > ============ > > 2015-04-16 14:16:17.996 ERROR glance.registry.client.v1.client > [req-dc54f751-eb4b-4a03-9146-52623ed733a9 d0feb86a66d54df8b4aff1848c35729e > 8afd7feb78ee445b8d642a66c96d49d5] > Registry client request GET /images/cirros-0.3.3-x86_64-uec-kernel raised > NotFound > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client Traceback > (most recent call last): > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client File > "/opt/stack/glance/glance/registry/client/v1/client.py", line 117, in > do_request > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client **kwargs) > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client File > "/opt/stack/glance/glance/common/client.py", line 71, in wrapped > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client return > func(self, *args, **kwargs) > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client File > "/opt/stack/glance/glance/common/client.py", line 376, in do_request > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client > headers=copy.deepcopy(headers)) > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client File > "/opt/stack/glance/glance/common/client.py", line 88, in wrapped > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client return > func(self, method, url, body, headers) > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client File > "/opt/stack/glance/glance/common/client.py", line 523, in _do_request > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client raise > exception.NotFound(res.read()) > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client NotFound: 404 > Not Found > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client > 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client The resource > could not be found. > > Traceback #2: > ============== > > 2015-04-16 14:17:28.159 INFO oslo_messaging._drivers.impl_rabbit > [req-78c0070a-2a89-452f-8f3b-7aa549b9a3ed admin admin] Connected to AMQP > server on 172.29.172.1 > 61:5672 > 2015-04-16 14:18:28.163 ERROR nova.api.openstack > [req-78c0070a-2a89-452f-8f3b-7aa549b9a3ed admin admin] Caught error: Timed > out waiting for a reply to message I > D 79c4306de3fc46aa986664735573100b > 2015-04-16 14:18:28.163 TRACE nova.api.openstack Traceback (most recent call > last): > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return > req.get_response(self.application) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send > 2015-04-16 14:18:28.163 TRACE nova.api.openstack application, > catch_exc_info=False) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in > call_application > 2015-04-16 14:18:28.163 TRACE nova.api.openstack app_iter = > application(self.environ, start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return resp(environ, > start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", > line 634, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return > self._call_app(env, start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", > line 554, in _call_app > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return self._app(env, > _fake_start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return resp(environ, > start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return resp(environ, > start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in > __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack response = > self.app(environ, start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return resp(environ, > start_response) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack resp = > self.call_func(req, *args, **self.kwargs) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return self.func(req, > *args, **kwargs) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in __call__ > 2015-04-16 14:18:28.163 TRACE nova.api.openstack content_type, body, > accept) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack > 2015-04-16 14:18:28.163 TRACE nova.api.openstack action_result = > self.dispatch(meth, request, action_args) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/api/openstack/wsgi.py", line 911, in dispatch > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return > method(req=request, **action_args) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/api/openstack/compute/contrib/certificates.py", line > 48, in show > 2015-04-16 14:18:28.163 TRACE nova.api.openstack > project_id=context.project_id) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/opt/stack/nova/nova/cert/rpcapi.py", line 93, in fetch_ca > 2015-04-16 14:18:28.163 TRACE nova.api.openstack return cctxt.call(ctxt, > 'fetch_ca', project_id=project_id) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line > 156, in call > 2015-04-16 14:18:28.163 TRACE nova.api.openstack retry=self.retry) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line > 90, in _send > 2015-04-16 14:18:28.163 TRACE nova.api.openstack timeout=timeout, > retry=retry) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 350, in send > 2015-04-16 14:18:28.163 TRACE nova.api.openstack retry=retry) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 339, in _send > 2015-04-16 14:18:28.163 TRACE nova.api.openstack result = > self._waiter.wait(msg_id, timeout) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 243, in wait > 2015-04-16 14:18:28.163 TRACE nova.api.openstack message = > self.waiters.get(msg_id, timeout=timeout) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack File > "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 149, in get > 2015-04-16 14:18:28.163 TRACE nova.api.openstack 'to message ID %s' % > msg_id) > 2015-04-16 14:18:28.163 TRACE nova.api.openstack MessagingTimeout: Timed out > waiting for a reply to message ID 79c4306de3fc46aa986664735573100b > 2015-04-16 14:18:28.163 TRACE nova.api.openstack > > ##################### > > stack.sh does completed. > > Are these tracebacks harmful? > > Also, I noticed "nova-cert" and "nova-consoleauth" are missing from show > output: > > localadmin@qa4:~/devstack-kilo$ nova service-list > +----+----------------+------+----------+---------+-------+----------------------------+-----------------+ > | Id | Binary | Host | Zone | Status | State | Updated_at > | Disabled Reason | > +----+----------------+------+----------+---------+-------+----------------------------+-----------------+ > | 1 | nova-conductor | qa4 | internal | enabled | up | > 2015-04-16T18:51:40.000000 | - | > | 2 | nova-scheduler | qa4 | internal | enabled | up | > 2015-04-16T18:51:44.000000 | - | > | 3 | nova-compute | qa5 | nova | enabled | up | > 2015-04-16T18:51:39.000000 | - | > | 4 | nova-compute | qa6 | nova | enabled | up | > 2015-04-16T18:51:46.000000 | - | > +----+----------------+------+----------+---------+-------+----------------------------+-----------------+ > > And I cannot launch the VM's console, both from Horizon and CLI. > > Note that I’m using the same local.conf settings that are working fine in > Juno. > > > Thanks, > > Danny > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Sean Dague http://dague.net __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
