Confirmed this also affects Ironic. Searched and found this open bug in oslo.messaging, addressing the underlying issue: https://bugs.launchpad.net/oslo.messaging/+bug/1586840
** Also affects: ironic Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1586979 Title: AMQP 2.0 prevents services from starting Status in Ironic: New Status in OpenStack Identity (keystone): New Status in neutron: New Status in OpenStack Compute (nova): New Bug description: We are running nova from the stable/mitaka branch (not using packages). Our build process has flagged this. Previously, when we've built, we've used the requirements.txt file to pull in dependencies, and one of the dependencies pulls in AMQP. Initial investigation leads us to believe that oslo.messaging pulls in AMQP as a dependency. It looks to be pulling in the latest version (2.0). Our previous successful builds show AMQP of 1.4.9, which is fully functional. 2.0 completely breaks, for example, the nova-compute service with the following trace: 2016-05-30 01:04:58.192 11335 CRITICAL nova [req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 'Connection' object has no attribute '_frame_writer' 2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last): 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/bin/nova-compute", line 10, in <module> 2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main()) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py", line 73, in main 2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=CONF.conductor.use_local) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 218, in create 2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 101, in init 2016-05-30 01:04:58.192 11335 ERROR nova self.conductor_api.wait_until_ready(context.get_admin_context()) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py", line 157, in wait_until_ready 2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", line 58, in ping 2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 'ping', arg=arg_p) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 157, in call 2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 91, in _send 2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 466, in send 2016-05-30 01:04:58.192 11335 ERROR nova retry=retry) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 410, in _send 2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': self._get_reply_q()}) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 382, in _get_reply_q 2016-05-30 01:04:58.192 11335 ERROR nova conn = self._get_connection(rpc_common.PURPOSE_LISTEN) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 373, in _get_connection 2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 396, in init 2016-05-30 01:04:58.192 11335 ERROR nova self.connection = connection_pool.create(purpose) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 110, in create 2016-05-30 01:04:58.192 11335 ERROR nova return self.connection_cls(self.conf, self.url, purpose) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 556, in init 2016-05-30 01:04:58.192 11335 ERROR nova self.ensure_connection() 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 654, in ensure_connection 2016-05-30 01:04:58.192 11335 ERROR nova self.ensure(method=lambda: self.connection.connection) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 759, in ensure 2016-05-30 01:04:58.192 11335 ERROR nova ret, channel = autoretry_method() 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured 2016-05-30 01:04:58.192 11335 ERROR nova return fun(*args, **kwargs) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 507, in call 2016-05-30 01:04:58.192 11335 ERROR nova self.revive(create_channel()) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel 2016-05-30 01:04:58.192 11335 ERROR nova chan = self.transport.create_channel(self.connection) 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 92, in create_channel 2016-05-30 01:04:58.192 11335 ERROR nova return connection.channel() 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/connection.py", line 380, in channel 2016-05-30 01:04:58.192 11335 ERROR nova channel.open() 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/channel.py", line 455, in open 2016-05-30 01:04:58.192 11335 ERROR nova spec.Channel.Open, 's', ('',), wait=spec.Channel.OpenOk, 2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 62, in send_method 2016-05-30 01:04:58.192 11335 ERROR nova conn._frame_writer.send((1, self.channel_id, sig, args, content)) 2016-05-30 01:04:58.192 11335 ERROR nova AttributeError: 'Connection' object has no attribute '_frame_writer' 2016-05-30 01:04:58.192 11335 ERROR nova Investigation also indicates this impacts other services. I've isolated builds into virtualenvs to prevent external environment from being a concern. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1586979/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp