[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting

2016-05-31 Thread Jay Jahns
Would like to hear from folks - this is impacting a lot of projects, and
people can't build from git branches...

IMO this is extremely critical and we need to get a response.

** Also affects: glance
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in Barbican:
  New
Status in Glance:
  New
Status in Ironic:
  New
Status in OpenStack Identity (keystone):
  New
Status in neutron:
  New
Status in OpenStack Compute (nova):
  New
Status in oslo.messaging:
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova 
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 
'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in 
  2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
 line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova 
db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova 
self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
 line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", 
line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 
'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
 line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': 
self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova conn = 
self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
 line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.connection = 
connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",
 line 110, in create
  2016-05-30 01:04:58.192 11335 ERROR nova return 
self.connection_cls(self.conf, self.url, purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 

[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting

2016-05-31 Thread Fernando Diaz
** Also affects: barbican
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in Barbican:
  New
Status in Ironic:
  New
Status in OpenStack Identity (keystone):
  New
Status in neutron:
  New
Status in OpenStack Compute (nova):
  New
Status in oslo.messaging:
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova 
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 
'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in 
  2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
 line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova 
db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova 
self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
 line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", 
line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 
'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
 line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': 
self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova conn = 
self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
 line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.connection = 
connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",
 line 110, in create
  2016-05-30 01:04:58.192 11335 ERROR nova return 
self.connection_cls(self.conf, self.url, purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
 line 556, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.ensure_connection()
  2016-05-30 01:04:58.192 11335 ERROR nova   

[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting

2016-05-31 Thread Jay Jahns
I added oslo.messaging to this.  If it needs to be marked duplicate then
that is fine, but considering nobody can build Openstack in this state,
there needs to be some urgency on this.  I tested a couple of
workarounds, but the best one so far for us has been to uninstall amqp
and reinstall the 1.4.9 version at the tail end of our automation.

The upper-constraints has 1.4.9 listed.

** Also affects: oslo.messaging
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in Ironic:
  New
Status in OpenStack Identity (keystone):
  New
Status in neutron:
  New
Status in OpenStack Compute (nova):
  New
Status in oslo.messaging:
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova 
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 
'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in 
  2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
 line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova 
db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova 
self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
 line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", 
line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 
'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
 line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': 
self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova conn = 
self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
 line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.connection = 
connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",
 line 110, in create
  2016-05-30 01:04:58.192 

[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting

2016-05-31 Thread Devananda van der Veen
Confirmed this also affects Ironic.

Searched and found this open bug in oslo.messaging, addressing the underlying 
issue:
https://bugs.launchpad.net/oslo.messaging/+bug/1586840

** Also affects: ironic
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in Ironic:
  New
Status in OpenStack Identity (keystone):
  New
Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova 
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 
'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in 
  2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
 line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova 
db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova 
self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
 line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", 
line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 
'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
 line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': 
self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova conn = 
self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
 line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.connection = 
connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",
 line 110, in create
  2016-05-30 01:04:58.192 11335 ERROR nova return 
self.connection_cls(self.conf, self.url, purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
 line 556, in 

[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting

2016-05-31 Thread Jay Jahns
Added affected projects that we've tested so far.  I have not
specifically tested Glance in my sandbox.  Overall, AMQP 2.0 is breaking
everything.  Our automation is working around the problem by
uninstalling 2.0 and installing 1.4.9 manually.

Would like to hear feedback on this.  Some folks are saying to use the
upper-constraints file to install dependencies.  This doesn't appear to
be a solution to the common man.

** Also affects: neutron
   Importance: Undecided
   Status: New

** Also affects: keystone
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in OpenStack Identity (keystone):
  New
Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova 
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 
'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in 
  2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
 line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova 
db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", 
line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova 
self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
 line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", 
line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 
'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
 line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
 line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': 
self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova conn = 
self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
 line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
 line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova self.connection = 
connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File 
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",