Public bug reported: 1. Exact version of Nova ii nova-api 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - API frontend ii nova-cert 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - certificate management ii nova-common 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - common files ii nova-conductor 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - conductor service ii nova-console 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - Console ii nova-consoleauth 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - Console Authenticatorii nova-novncproxy 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - NoVNC proxy ii nova-scheduler 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - virtual machine scheduler ii python-nova 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute Python libraries ii python-novaclient 1:2.17.0.74.g2598714+git201404220131~trusty-0ubuntu1 all client library for OpenStack Compute API
rabbit configuration in nova.conf: rabbit_hosts = m610-2:5672, m610-1:5672 rabbit_ha_queues = true 2. Relevant log files: /var/log/nova/nova-conductor.log exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.486 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672 2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.489 16425 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_bea18a6133c548f099b85b168fddf83c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.704 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-2:5672 2015-06-01 08:23:56.707 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672 2015-06-01 08:23:56.709 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-1:5672 2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.713 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... /var/log/rabbitmq/rab...@m610-1.log =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.300.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.294.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.297.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.303.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'", 'exchange.declare'} 3. Reproduce steps: 2 controller nodes with rabbitMQ HA cluster and nova services (m610-2, m610-1) rabbit configuration in nova.conf: It happens after crush one of controllers. Restarting of nova-conductor service does not solving this problem. ** Affects: nova Importance: Undecided Status: New ** Tags: messaging -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1460652 Title: nova-conductor infinitely reconnets to rabbit Status in OpenStack Compute (Nova): New Bug description: 1. Exact version of Nova ii nova-api 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - API frontend ii nova-cert 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - certificate management ii nova-common 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - common files ii nova-conductor 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - conductor service ii nova-console 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - Console ii nova-consoleauth 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - Console Authenticatorii nova-novncproxy 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - NoVNC proxy ii nova-scheduler 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute - virtual machine scheduler ii python-nova 1:2014.1.100+git201410062002~trusty-0ubuntu1 all OpenStack Compute Python libraries ii python-novaclient 1:2.17.0.74.g2598714+git201404220131~trusty-0ubuntu1 all client library for OpenStack Compute API rabbit configuration in nova.conf: rabbit_hosts = m610-2:5672, m610-1:5672 rabbit_ha_queues = true 2. Relevant log files: /var/log/nova/nova-conductor.log exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.486 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672 2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.489 16425 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_bea18a6133c548f099b85b168fddf83c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/' 2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.704 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-2:5672 2015-06-01 08:23:56.707 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672 2015-06-01 08:23:56.709 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-1:5672 2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-06-01 08:23:56.713 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__ 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit reply_code, reply_text, (class_id, method_id), ChannelError, 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/' 2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit 2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672 2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... /var/log/rabbitmq/rab...@m610-1.log =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.300.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.294.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.297.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'", 'exchange.declare'} =ERROR REPORT==== 1-Jun-2015::08:56:38 === connection <0.303.15>, channel 1 - soft error: {amqp_error,not_found, "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'", 'exchange.declare'} 3. Reproduce steps: 2 controller nodes with rabbitMQ HA cluster and nova services (m610-2, m610-1) rabbit configuration in nova.conf: It happens after crush one of controllers. Restarting of nova-conductor service does not solving this problem. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1460652/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp