Public bug reported: With the same configuration that is confirmed to work with mitaka b1, I'm seeing the following error with mitaka b2 - neutron is unable to send external event notifications back to nova:
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'8cbc4861-97cb-484e-bd10-fd6a226c332a', 'name': 'network-vif-plugged', 'server_uuid': u'2831a5ed-26f9-42ed-bb9d-363b1117307f'}] 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova Traceback (most recent call last): 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 209, in send_events 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova batched_events) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return_raw=True) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 176, in post 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 345, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova base_url = self.get_endpoint(auth, **endpoint_filter) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 615, in get_endpoint 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return auth.get_endpoint(self, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 213, in get_endpoint 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova service_name=service_name) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova raise exceptions.EmptyCatalog('The service catalog is empty.') 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova EmptyCatalog: The service catalog is empty. 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova neutron.conf: [DEFAULT] verbose = False debug = False use_syslog = False state_path = /var/lib/neutron bind_host = 0.0.0.0 auth_strategy = keystone notification_driver = neutron.openstack.common.notifier.rpc_notifier api_workers = 2 rpc_workers = 2 router_distributed = False l3_ha = False bind_port = 9686 core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin service_plugins = router,firewall,lbaas,vpnaas,metering allow_overlapping_ips = True neutron_firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True [nova] auth_plugin = password region_name = RegionOne auth_url = http://10.5.3.86:35357/v2.0 username = neutron password = XXX tenant_id = 8679f8352c4b4be0a7edda3b41099731 [quotas] quota_driver = neutron.db.quota_db.DbQuotaDriver quota_items = network,subnet,port,security_group,security_group_rule quota_security_group = 10 quota_security_group_rule = 100 quota_network = 10 quota_subnet = 10 quota_port = 50 quota_vip = 10 quota_pool = 10 quota_member = -1 quota_health_monitors = -1 quota_router = 10 quota_floatingip = 50 [agent] root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf [keystone_authtoken] identity_uri = http://10.5.3.86:35357/ auth_uri = http://10.5.3.86:5000/ admin_tenant_name = services admin_user = neutron admin_password = XXXX signing_dir = /var/cache/neutron [database] connection = mysql://neutron:XXXX@10.5.3.88/neutron max_pool_size = 8 [oslo_messaging_rabbit] rabbit_userid = neutron rabbit_virtual_host = openstack rabbit_password = XXX rabbit_host = 10.5.3.96 [oslo_concurrency] lock_path = $state_path/lock ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1549227 Title: mitaka b2: neutron->nova external event notification "Service catalog is empty" Status in neutron: New Bug description: With the same configuration that is confirmed to work with mitaka b1, I'm seeing the following error with mitaka b2 - neutron is unable to send external event notifications back to nova: 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'8cbc4861-97cb-484e-bd10-fd6a226c332a', 'name': 'network-vif-plugged', 'server_uuid': u'2831a5ed-26f9-42ed-bb9d-363b1117307f'}] 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova Traceback (most recent call last): 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 209, in send_events 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova batched_events) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return_raw=True) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 176, in post 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 345, in request 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova base_url = self.get_endpoint(auth, **endpoint_filter) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 615, in get_endpoint 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return auth.get_endpoint(self, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 213, in get_endpoint 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova service_name=service_name) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs) 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova raise exceptions.EmptyCatalog('The service catalog is empty.') 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova EmptyCatalog: The service catalog is empty. 2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova neutron.conf: [DEFAULT] verbose = False debug = False use_syslog = False state_path = /var/lib/neutron bind_host = 0.0.0.0 auth_strategy = keystone notification_driver = neutron.openstack.common.notifier.rpc_notifier api_workers = 2 rpc_workers = 2 router_distributed = False l3_ha = False bind_port = 9686 core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin service_plugins = router,firewall,lbaas,vpnaas,metering allow_overlapping_ips = True neutron_firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True [nova] auth_plugin = password region_name = RegionOne auth_url = http://10.5.3.86:35357/v2.0 username = neutron password = XXX tenant_id = 8679f8352c4b4be0a7edda3b41099731 [quotas] quota_driver = neutron.db.quota_db.DbQuotaDriver quota_items = network,subnet,port,security_group,security_group_rule quota_security_group = 10 quota_security_group_rule = 100 quota_network = 10 quota_subnet = 10 quota_port = 50 quota_vip = 10 quota_pool = 10 quota_member = -1 quota_health_monitors = -1 quota_router = 10 quota_floatingip = 50 [agent] root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf [keystone_authtoken] identity_uri = http://10.5.3.86:35357/ auth_uri = http://10.5.3.86:5000/ admin_tenant_name = services admin_user = neutron admin_password = XXXX signing_dir = /var/cache/neutron [database] connection = mysql://neutron:XXXX@10.5.3.88/neutron max_pool_size = 8 [oslo_messaging_rabbit] rabbit_userid = neutron rabbit_virtual_host = openstack rabbit_password = XXX rabbit_host = 10.5.3.96 [oslo_concurrency] lock_path = $state_path/lock To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1549227/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp