Public bug reported:
[Alchemy][LB_namespaceV1]after 817 tenants was create for namespace 1v1 maping, the new LB pool with pending_creat status test step 1 create a tenant 2 create a pool network with subnet 10.1.1.0 create a vip network with subnet 20.1.1.0 3 create a router and add 2 interface on it for subnet 10 and subnet20 4 create a LB pool with subnet 10 5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and port 80 6(option ) add one or two member 7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce) for scale test add to following tenant 80 tenants with 2 backend and 1 client (LB monitor enable )with simple traffic 100 tenants with 1 backend and 1 client (LB monitor enable) with simple traffic 1067 tenants without member and monitor total of 1247 pools , total of 817 pools with active , total of 376 pools with error or pending-create status ) log in attachment the issue :after add 817 namespace , created new lb pool with pending-status or error status after check log, AMQP can't connect to server , and then reset rabbitmq service on controller node " service rabbitmq-server restart" the setup can't recover , the new lb pool still with pending-create status for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping) after delete tenant to 328(remain), the setup recover , new lb pool can be create with active status , so it was a scale issue , and file this bug for trace this issue log [root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log 2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure': None, u'result': None} 2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>} 2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure': None, u'result': None, u'ending': True} 2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None _waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>} 2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit **options) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51)) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:36:57.747 2581 ERROR neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance for pool: 92fb078d-f203-4972-a555-e2ba033d50d8 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call last): 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py", line 169, in _reload_pool 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager logical_config = self.plugin_rpc.get_logical_device(pool_id) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 57, in get_logical_device 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager host=self.host 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args']) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 78db36a255ae4b20b090554216923a20 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager 2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit **options) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51)) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672 [root@nsj17 ~]# Notes Log: IBM Licensed Material - Property of IBM ? Cop ==== State: Open by: liujgwx on 18 August 2015 11:25:02 ==== correct the number 180 with traffic 690 without traffic Total Active LB Namespace = 870 Total LB Namespace = 1247 Inactive = 377 ==== State: Open by: liujgwx on 13 August 2015 01:02:45 ==== [root@nsj17 ~]# sar -P ALL 60 1 Linux 3.10.0-229.7.2.el7.x86_64 (nsj17) 08/12/2015 _x86_64_ (80 CPU) 07:28:50 PM CPU %user %nice %system %iowait %steal %idle 07:29:50 PM all 2.45 0.00 72.87 0.00 0.00 24.68 07:29:50 PM 0 3.09 0.00 86.10 0.00 0.00 10.81 07:29:50 PM 1 3.24 0.00 83.92 0.00 0.00 12.84 07:29:50 PM 2 2.88 0.00 77.57 0.00 0.00 19.54 07:29:50 PM 3 2.16 0.00 75.61 0.00 0.00 22.23 07:29:50 PM 4 2.37 0.00 76.48 0.00 0.00 21.15 07:29:50 PM 5 2.44 0.00 75.88 0.00 0.00 21.69 07:29:50 PM 6 2.23 0.00 78.28 0.00 0.00 19.48 07:29:50 PM 7 2.49 0.00 72.97 0.00 0.00 24.54 07:29:50 PM 8 2.63 0.00 75.52 0.00 0.00 21.84 07:29:50 PM 9 2.95 0.00 71.80 0.00 0.00 25.25 07:29:50 PM 10 2.23 0.00 76.04 0.00 0.00 21.73 07:29:50 PM 11 2.26 0.00 75.26 0.00 0.00 22.48 07:29:50 PM 12 2.43 0.00 74.74 0.00 0.00 22.83 07:29:50 PM 13 1.95 0.00 73.81 0.00 0.00 24.23 07:29:50 PM 14 2.33 0.00 72.21 0.00 0.00 25.46 07:29:50 PM 15 2.22 0.00 73.05 0.00 0.00 24.74 07:29:50 PM 16 2.08 0.00 72.20 0.00 0.00 25.72 07:29:50 PM 17 2.13 0.00 71.66 0.00 0.00 26.21 07:29:50 PM 18 2.11 0.00 71.55 0.00 0.00 26.33 07:29:50 PM 19 2.18 0.00 72.35 0.00 0.00 25.47 07:29:50 PM 20 1.81 0.00 73.13 0.00 0.00 25.06 07:29:50 PM 21 1.57 0.00 73.98 0.00 0.00 24.45 07:29:50 PM 22 1.94 0.00 73.67 0.00 0.00 24.39 07:29:50 PM 23 1.79 0.00 75.57 0.00 0.00 22.64 07:29:50 PM 24 2.03 0.00 75.06 0.00 0.00 22.91 07:29:50 PM 25 1.85 0.00 73.46 0.00 0.00 24.69 07:29:50 PM 26 1.62 0.00 70.14 0.00 0.00 28.24 07:29:50 PM 27 1.85 0.00 73.08 0.00 0.00 25.07 07:29:50 PM 28 1.77 0.00 73.98 0.00 0.00 24.25 07:29:50 PM 29 1.58 0.00 72.81 0.00 0.00 25.60 07:29:50 PM 30 2.82 0.00 70.37 0.00 0.00 26.81 07:29:50 PM 31 2.74 0.00 73.64 0.00 0.00 23.62 07:29:50 PM 32 2.77 0.00 70.33 0.00 0.00 26.91 07:29:50 PM 33 2.57 0.00 72.26 0.00 0.00 25.17 07:29:50 PM 34 3.16 0.00 71.70 0.00 0.00 25.14 07:29:50 PM 35 2.61 0.00 70.21 0.00 0.00 27.18 07:29:50 PM 36 2.42 0.00 72.79 0.00 0.00 24.79 07:29:50 PM 37 2.31 0.00 74.05 0.00 0.00 23.64 07:29:50 PM 38 3.01 0.00 77.82 0.00 0.00 19.17 07:29:50 PM 39 2.60 0.00 74.25 0.00 0.00 23.15 07:29:50 PM 40 3.72 0.00 70.53 0.00 0.00 25.74 07:29:50 PM 41 2.77 0.00 68.57 0.00 0.00 28.66 07:29:50 PM 42 2.79 0.00 71.16 0.00 0.00 26.06 07:29:50 PM 43 2.94 0.00 71.23 0.00 0.00 25.83 07:29:50 PM 44 2.64 0.00 68.09 0.00 0.00 29.27 07:29:50 PM 45 2.63 0.00 68.67 0.00 0.00 28.70 07:29:50 PM 46 2.16 0.00 63.88 0.00 0.00 33.96 07:29:50 PM 47 3.06 0.00 66.90 0.00 0.00 30.04 07:29:50 PM 48 2.64 0.00 66.79 0.00 0.00 30.57 07:29:50 PM 49 2.98 0.00 65.90 0.00 0.00 31.12 07:29:50 PM 50 2.56 0.00 73.78 0.00 0.00 23.65 07:29:50 PM 51 2.16 0.00 76.16 0.00 0.00 21.67 07:29:50 PM 52 2.60 0.00 74.06 0.00 0.00 23.33 07:29:50 PM 53 2.46 0.00 73.68 0.00 0.00 23.87 07:29:50 PM 54 2.52 0.00 72.06 0.00 0.00 25.42 07:29:50 PM 55 2.24 0.00 73.17 0.00 0.00 24.60 07:29:50 PM 56 2.42 0.00 72.65 0.00 0.00 24.93 07:29:50 PM 57 2.49 0.00 72.79 0.00 0.00 24.72 07:29:50 PM 58 2.32 0.00 70.32 0.00 0.00 27.35 07:29:50 PM 59 1.96 0.00 68.68 0.00 0.00 29.37 07:29:50 PM 60 1.91 0.00 72.05 0.00 0.00 26.04 07:29:50 PM 61 2.35 0.00 74.71 0.00 0.00 22.93 07:29:50 PM 62 1.85 0.00 74.49 0.00 0.00 23.66 07:29:50 PM 63 2.49 0.00 73.61 0.00 0.00 23.90 07:29:50 PM 64 2.40 0.00 76.26 0.00 0.00 21.34 07:29:50 PM 65 1.97 0.00 73.83 0.00 0.00 24.20 07:29:50 PM 66 1.93 0.00 74.30 0.00 0.00 23.77 07:29:50 PM 67 2.24 0.00 70.79 0.00 0.00 26.98 07:29:50 PM 68 2.01 0.00 67.80 0.00 0.00 30.19 07:29:50 PM 69 1.85 0.00 71.32 0.00 0.00 26.83 07:29:50 PM 70 3.24 0.00 75.25 0.00 0.00 21.51 07:29:50 PM 71 3.20 0.00 77.97 0.00 0.00 18.83 07:29:50 PM 72 3.20 0.00 71.82 0.00 0.00 24.98 07:29:50 PM 73 2.74 0.00 73.68 0.00 0.00 23.59 07:29:50 PM 74 2.89 0.00 73.68 0.00 0.00 23.42 07:29:50 PM 75 3.24 0.00 74.40 0.00 0.00 22.35 07:29:50 PM 76 2.81 0.00 70.84 0.00 0.00 26.35 07:29:50 PM 77 2.83 0.00 73.13 0.00 0.00 24.04 07:29:50 PM 78 2.64 0.00 69.46 0.00 0.00 27.90 07:29:50 PM 79 2.88 0.00 64.28 0.00 0.00 32.84 Average: CPU %user %nice %system %iowait %steal %idle Average: all 2.45 0.00 72.87 0.00 0.00 24.68 Average: 0 3.09 0.00 86.10 0.00 0.00 10.81 Average: 1 3.24 0.00 83.92 0.00 0.00 12.84 Average: 2 2.88 0.00 77.57 0.00 0.00 19.54 Average: 3 2.16 0.00 75.61 0.00 0.00 22.23 Average: 4 2.37 0.00 76.48 0.00 0.00 21.15 Average: 5 2.44 0.00 75.88 0.00 0.00 21.69 Average: 6 2.23 0.00 78.28 0.00 0.00 19.48 Average: 7 2.49 0.00 72.97 0.00 0.00 24.54 Average: 8 2.63 0.00 75.52 0.00 0.00 21.84 Average: 9 2.95 0.00 71.80 0.00 0.00 25.25 Average: 10 2.23 0.00 76.04 0.00 0.00 21.73 Average: 11 2.26 0.00 75.26 0.00 0.00 22.48 Average: 12 2.43 0.00 74.74 0.00 0.00 22.83 Average: 13 1.95 0.00 73.81 0.00 0.00 24.23 Average: 14 2.33 0.00 72.21 0.00 0.00 25.46 Average: 15 2.22 0.00 73.05 0.00 0.00 24.74 Average: 16 2.08 0.00 72.20 0.00 0.00 25.72 Average: 17 2.13 0.00 71.66 0.00 0.00 26.21 Average: 18 2.11 0.00 71.55 0.00 0.00 26.33 Average: 19 2.18 0.00 72.35 0.00 0.00 25.47 Average: 20 1.81 0.00 73.13 0.00 0.00 25.06 Average: 21 1.57 0.00 73.98 0.00 0.00 24.45 Average: 22 1.94 0.00 73.67 0.00 0.00 24.39 Average: 23 1.79 0.00 75.57 0.00 0.00 22.64 Average: 24 2.03 0.00 75.06 0.00 0.00 22.91 Average: 25 1.85 0.00 73.46 0.00 0.00 24.69 Average: 26 1.62 0.00 70.14 0.00 0.00 28.24 Average: 27 1.85 0.00 73.08 0.00 0.00 25.07 Average: 28 1.77 0.00 73.98 0.00 0.00 24.25 Average: 29 1.58 0.00 72.81 0.00 0.00 25.60 Average: 30 2.82 0.00 70.37 0.00 0.00 26.81 Average: 31 2.74 0.00 73.64 0.00 0.00 23.62 Average: 32 2.77 0.00 70.33 0.00 0.00 26.91 Average: 33 2.57 0.00 72.26 0.00 0.00 25.17 Average: 34 3.16 0.00 71.70 0.00 0.00 25.14 Average: 35 2.61 0.00 70.21 0.00 0.00 27.18 Average: 36 2.42 0.00 72.79 0.00 0.00 24.79 Average: 37 2.31 0.00 74.05 0.00 0.00 23.64 Average: 38 3.01 0.00 77.82 0.00 0.00 19.17 Average: 39 2.60 0.00 74.25 0.00 0.00 23.15 Average: 40 3.72 0.00 70.53 0.00 0.00 25.74 Average: 41 2.77 0.00 68.57 0.00 0.00 28.66 Average: 42 2.79 0.00 71.16 0.00 0.00 26.06 Average: 43 2.94 0.00 71.23 0.00 0.00 25.83 Average: 44 2.64 0.00 68.09 0.00 0.00 29.27 Average: 45 2.63 0.00 68.67 0.00 0.00 28.70 Average: 46 2.16 0.00 63.88 0.00 0.00 33.96 Average: 47 3.06 0.00 66.90 0.00 0.00 30.04 Average: 48 2.64 0.00 66.79 0.00 0.00 30.57 Average: 49 2.98 0.00 65.90 0.00 0.00 31.12 Average: 50 2.56 0.00 73.78 0.00 0.00 23.65 Average: 51 2.16 0.00 76.16 0.00 0.00 21.67 Average: 52 2.60 0.00 74.06 0.00 0.00 23.33 Average: 53 2.46 0.00 73.68 0.00 0.00 23.87 Average: 54 2.52 0.00 72.06 0.00 0.00 25.42 Average: 55 2.24 0.00 73.17 0.00 0.00 24.60 Average: 56 2.42 0.00 72.65 0.00 0.00 24.93 Average: 57 2.49 0.00 72.79 0.00 0.00 24.72 Average: 58 2.32 0.00 70.32 0.00 0.00 27.35 Average: 59 1.96 0.00 68.68 0.00 0.00 29.37 Average: 60 1.91 0.00 72.05 0.00 0.00 26.04 Average: 61 2.35 0.00 74.71 0.00 0.00 22.93 Average: 62 1.85 0.00 74.49 0.00 0.00 23.66 Average: 63 2.49 0.00 73.61 0.00 0.00 23.90 Average: 64 2.40 0.00 76.26 0.00 0.00 21.34 Average: 65 1.97 0.00 73.83 0.00 0.00 24.20 Average: 66 1.93 0.00 74.30 0.00 0.00 23.77 Average: 67 2.24 0.00 70.79 0.00 0.00 26.98 Average: 68 2.01 0.00 67.80 0.00 0.00 30.19 Average: 69 1.85 0.00 71.32 0.00 0.00 26.83 Average: 70 3.24 0.00 75.25 0.00 0.00 21.51 Average: 71 3.20 0.00 77.97 0.00 0.00 18.83 Average: 72 3.20 0.00 71.82 0.00 0.00 24.98 Average: 73 2.74 0.00 73.68 0.00 0.00 23.59 Average: 74 2.89 0.00 73.68 0.00 0.00 23.42 Average: 75 3.24 0.00 74.40 0.00 0.00 22.35 Average: 76 2.81 0.00 70.84 0.00 0.00 26.35 Average: 77 2.83 0.00 73.13 0.00 0.00 24.04 Average: 78 2.64 0.00 69.46 0.00 0.00 27.90 Average: 79 2.88 0.00 64.28 0.00 0.00 32.84 [root@nsj17 ~]# free total used free shared buff/cache available Mem: 263961496 19596608 220503484 4236472 23861404 231497040 Swap: 1048572 0 1048572 [root@nsj17 ~]# ==== State: Open by: liujgwx on 12 August 2015 02:52:28 ==== liujgwx (liuj...@cn.ibm.com) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/console output .txt on 2015-08-12 02:52:28 liujgwx (liuj...@cn.ibm.com) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/total pools .txt on 2015-08-12 02:52:28 ==== State: Open by: liujgwx on 12 August 2015 02:45:28 ==== Notice: This Note entry was modified. 1 non-ascii character(s) were replaced with question marks. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1498472 Title: after 871 namespace was created for 1v1 maping, the new LB pool with pending_creat status Status in neutron: New Bug description: [Alchemy][LB_namespaceV1]after 817 tenants was create for namespace 1v1 maping, the new LB pool with pending_creat status test step 1 create a tenant 2 create a pool network with subnet 10.1.1.0 create a vip network with subnet 20.1.1.0 3 create a router and add 2 interface on it for subnet 10 and subnet20 4 create a LB pool with subnet 10 5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and port 80 6(option ) add one or two member 7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce) for scale test add to following tenant 80 tenants with 2 backend and 1 client (LB monitor enable )with simple traffic 100 tenants with 1 backend and 1 client (LB monitor enable) with simple traffic 1067 tenants without member and monitor total of 1247 pools , total of 817 pools with active , total of 376 pools with error or pending-create status ) log in attachment the issue :after add 817 namespace , created new lb pool with pending-status or error status after check log, AMQP can't connect to server , and then reset rabbitmq service on controller node " service rabbitmq-server restart" the setup can't recover , the new lb pool still with pending-create status for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping) after delete tenant to 328(remain), the setup recover , new lb pool can be create with active status , so it was a scale issue , and file this bug for trace this issue log [root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log 2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure': None, u'result': None} 2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>} 2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure': None, u'result': None, u'ending': True} 2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None _waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>} 2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit **options) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__ 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok() 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51)) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe 2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:36:57.747 2581 ERROR neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance for pool: 92fb078d-f203-4972-a555-e2ba033d50d8 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call last): 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py", line 169, in _reload_pool 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager logical_config = self.plugin_rpc.get_logical_device(pool_id) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 57, in get_logical_device 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager host=self.host 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args']) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id) 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 78db36a255ae4b20b090554216923a20 2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager 2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit **options) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__ 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok() 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51)) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe 2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return method() 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError) 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit 2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds... 2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672 2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672 [root@nsj17 ~]# Notes Log: IBM Licensed Material - Property of IBM ? Cop ==== State: Open by: liujgwx on 18 August 2015 11:25:02 ==== correct the number 180 with traffic 690 without traffic Total Active LB Namespace = 870 Total LB Namespace = 1247 Inactive = 377 ==== State: Open by: liujgwx on 13 August 2015 01:02:45 ==== [root@nsj17 ~]# sar -P ALL 60 1 Linux 3.10.0-229.7.2.el7.x86_64 (nsj17) 08/12/2015 _x86_64_ (80 CPU) 07:28:50 PM CPU %user %nice %system %iowait %steal %idle 07:29:50 PM all 2.45 0.00 72.87 0.00 0.00 24.68 07:29:50 PM 0 3.09 0.00 86.10 0.00 0.00 10.81 07:29:50 PM 1 3.24 0.00 83.92 0.00 0.00 12.84 07:29:50 PM 2 2.88 0.00 77.57 0.00 0.00 19.54 07:29:50 PM 3 2.16 0.00 75.61 0.00 0.00 22.23 07:29:50 PM 4 2.37 0.00 76.48 0.00 0.00 21.15 07:29:50 PM 5 2.44 0.00 75.88 0.00 0.00 21.69 07:29:50 PM 6 2.23 0.00 78.28 0.00 0.00 19.48 07:29:50 PM 7 2.49 0.00 72.97 0.00 0.00 24.54 07:29:50 PM 8 2.63 0.00 75.52 0.00 0.00 21.84 07:29:50 PM 9 2.95 0.00 71.80 0.00 0.00 25.25 07:29:50 PM 10 2.23 0.00 76.04 0.00 0.00 21.73 07:29:50 PM 11 2.26 0.00 75.26 0.00 0.00 22.48 07:29:50 PM 12 2.43 0.00 74.74 0.00 0.00 22.83 07:29:50 PM 13 1.95 0.00 73.81 0.00 0.00 24.23 07:29:50 PM 14 2.33 0.00 72.21 0.00 0.00 25.46 07:29:50 PM 15 2.22 0.00 73.05 0.00 0.00 24.74 07:29:50 PM 16 2.08 0.00 72.20 0.00 0.00 25.72 07:29:50 PM 17 2.13 0.00 71.66 0.00 0.00 26.21 07:29:50 PM 18 2.11 0.00 71.55 0.00 0.00 26.33 07:29:50 PM 19 2.18 0.00 72.35 0.00 0.00 25.47 07:29:50 PM 20 1.81 0.00 73.13 0.00 0.00 25.06 07:29:50 PM 21 1.57 0.00 73.98 0.00 0.00 24.45 07:29:50 PM 22 1.94 0.00 73.67 0.00 0.00 24.39 07:29:50 PM 23 1.79 0.00 75.57 0.00 0.00 22.64 07:29:50 PM 24 2.03 0.00 75.06 0.00 0.00 22.91 07:29:50 PM 25 1.85 0.00 73.46 0.00 0.00 24.69 07:29:50 PM 26 1.62 0.00 70.14 0.00 0.00 28.24 07:29:50 PM 27 1.85 0.00 73.08 0.00 0.00 25.07 07:29:50 PM 28 1.77 0.00 73.98 0.00 0.00 24.25 07:29:50 PM 29 1.58 0.00 72.81 0.00 0.00 25.60 07:29:50 PM 30 2.82 0.00 70.37 0.00 0.00 26.81 07:29:50 PM 31 2.74 0.00 73.64 0.00 0.00 23.62 07:29:50 PM 32 2.77 0.00 70.33 0.00 0.00 26.91 07:29:50 PM 33 2.57 0.00 72.26 0.00 0.00 25.17 07:29:50 PM 34 3.16 0.00 71.70 0.00 0.00 25.14 07:29:50 PM 35 2.61 0.00 70.21 0.00 0.00 27.18 07:29:50 PM 36 2.42 0.00 72.79 0.00 0.00 24.79 07:29:50 PM 37 2.31 0.00 74.05 0.00 0.00 23.64 07:29:50 PM 38 3.01 0.00 77.82 0.00 0.00 19.17 07:29:50 PM 39 2.60 0.00 74.25 0.00 0.00 23.15 07:29:50 PM 40 3.72 0.00 70.53 0.00 0.00 25.74 07:29:50 PM 41 2.77 0.00 68.57 0.00 0.00 28.66 07:29:50 PM 42 2.79 0.00 71.16 0.00 0.00 26.06 07:29:50 PM 43 2.94 0.00 71.23 0.00 0.00 25.83 07:29:50 PM 44 2.64 0.00 68.09 0.00 0.00 29.27 07:29:50 PM 45 2.63 0.00 68.67 0.00 0.00 28.70 07:29:50 PM 46 2.16 0.00 63.88 0.00 0.00 33.96 07:29:50 PM 47 3.06 0.00 66.90 0.00 0.00 30.04 07:29:50 PM 48 2.64 0.00 66.79 0.00 0.00 30.57 07:29:50 PM 49 2.98 0.00 65.90 0.00 0.00 31.12 07:29:50 PM 50 2.56 0.00 73.78 0.00 0.00 23.65 07:29:50 PM 51 2.16 0.00 76.16 0.00 0.00 21.67 07:29:50 PM 52 2.60 0.00 74.06 0.00 0.00 23.33 07:29:50 PM 53 2.46 0.00 73.68 0.00 0.00 23.87 07:29:50 PM 54 2.52 0.00 72.06 0.00 0.00 25.42 07:29:50 PM 55 2.24 0.00 73.17 0.00 0.00 24.60 07:29:50 PM 56 2.42 0.00 72.65 0.00 0.00 24.93 07:29:50 PM 57 2.49 0.00 72.79 0.00 0.00 24.72 07:29:50 PM 58 2.32 0.00 70.32 0.00 0.00 27.35 07:29:50 PM 59 1.96 0.00 68.68 0.00 0.00 29.37 07:29:50 PM 60 1.91 0.00 72.05 0.00 0.00 26.04 07:29:50 PM 61 2.35 0.00 74.71 0.00 0.00 22.93 07:29:50 PM 62 1.85 0.00 74.49 0.00 0.00 23.66 07:29:50 PM 63 2.49 0.00 73.61 0.00 0.00 23.90 07:29:50 PM 64 2.40 0.00 76.26 0.00 0.00 21.34 07:29:50 PM 65 1.97 0.00 73.83 0.00 0.00 24.20 07:29:50 PM 66 1.93 0.00 74.30 0.00 0.00 23.77 07:29:50 PM 67 2.24 0.00 70.79 0.00 0.00 26.98 07:29:50 PM 68 2.01 0.00 67.80 0.00 0.00 30.19 07:29:50 PM 69 1.85 0.00 71.32 0.00 0.00 26.83 07:29:50 PM 70 3.24 0.00 75.25 0.00 0.00 21.51 07:29:50 PM 71 3.20 0.00 77.97 0.00 0.00 18.83 07:29:50 PM 72 3.20 0.00 71.82 0.00 0.00 24.98 07:29:50 PM 73 2.74 0.00 73.68 0.00 0.00 23.59 07:29:50 PM 74 2.89 0.00 73.68 0.00 0.00 23.42 07:29:50 PM 75 3.24 0.00 74.40 0.00 0.00 22.35 07:29:50 PM 76 2.81 0.00 70.84 0.00 0.00 26.35 07:29:50 PM 77 2.83 0.00 73.13 0.00 0.00 24.04 07:29:50 PM 78 2.64 0.00 69.46 0.00 0.00 27.90 07:29:50 PM 79 2.88 0.00 64.28 0.00 0.00 32.84 Average: CPU %user %nice %system %iowait %steal %idle Average: all 2.45 0.00 72.87 0.00 0.00 24.68 Average: 0 3.09 0.00 86.10 0.00 0.00 10.81 Average: 1 3.24 0.00 83.92 0.00 0.00 12.84 Average: 2 2.88 0.00 77.57 0.00 0.00 19.54 Average: 3 2.16 0.00 75.61 0.00 0.00 22.23 Average: 4 2.37 0.00 76.48 0.00 0.00 21.15 Average: 5 2.44 0.00 75.88 0.00 0.00 21.69 Average: 6 2.23 0.00 78.28 0.00 0.00 19.48 Average: 7 2.49 0.00 72.97 0.00 0.00 24.54 Average: 8 2.63 0.00 75.52 0.00 0.00 21.84 Average: 9 2.95 0.00 71.80 0.00 0.00 25.25 Average: 10 2.23 0.00 76.04 0.00 0.00 21.73 Average: 11 2.26 0.00 75.26 0.00 0.00 22.48 Average: 12 2.43 0.00 74.74 0.00 0.00 22.83 Average: 13 1.95 0.00 73.81 0.00 0.00 24.23 Average: 14 2.33 0.00 72.21 0.00 0.00 25.46 Average: 15 2.22 0.00 73.05 0.00 0.00 24.74 Average: 16 2.08 0.00 72.20 0.00 0.00 25.72 Average: 17 2.13 0.00 71.66 0.00 0.00 26.21 Average: 18 2.11 0.00 71.55 0.00 0.00 26.33 Average: 19 2.18 0.00 72.35 0.00 0.00 25.47 Average: 20 1.81 0.00 73.13 0.00 0.00 25.06 Average: 21 1.57 0.00 73.98 0.00 0.00 24.45 Average: 22 1.94 0.00 73.67 0.00 0.00 24.39 Average: 23 1.79 0.00 75.57 0.00 0.00 22.64 Average: 24 2.03 0.00 75.06 0.00 0.00 22.91 Average: 25 1.85 0.00 73.46 0.00 0.00 24.69 Average: 26 1.62 0.00 70.14 0.00 0.00 28.24 Average: 27 1.85 0.00 73.08 0.00 0.00 25.07 Average: 28 1.77 0.00 73.98 0.00 0.00 24.25 Average: 29 1.58 0.00 72.81 0.00 0.00 25.60 Average: 30 2.82 0.00 70.37 0.00 0.00 26.81 Average: 31 2.74 0.00 73.64 0.00 0.00 23.62 Average: 32 2.77 0.00 70.33 0.00 0.00 26.91 Average: 33 2.57 0.00 72.26 0.00 0.00 25.17 Average: 34 3.16 0.00 71.70 0.00 0.00 25.14 Average: 35 2.61 0.00 70.21 0.00 0.00 27.18 Average: 36 2.42 0.00 72.79 0.00 0.00 24.79 Average: 37 2.31 0.00 74.05 0.00 0.00 23.64 Average: 38 3.01 0.00 77.82 0.00 0.00 19.17 Average: 39 2.60 0.00 74.25 0.00 0.00 23.15 Average: 40 3.72 0.00 70.53 0.00 0.00 25.74 Average: 41 2.77 0.00 68.57 0.00 0.00 28.66 Average: 42 2.79 0.00 71.16 0.00 0.00 26.06 Average: 43 2.94 0.00 71.23 0.00 0.00 25.83 Average: 44 2.64 0.00 68.09 0.00 0.00 29.27 Average: 45 2.63 0.00 68.67 0.00 0.00 28.70 Average: 46 2.16 0.00 63.88 0.00 0.00 33.96 Average: 47 3.06 0.00 66.90 0.00 0.00 30.04 Average: 48 2.64 0.00 66.79 0.00 0.00 30.57 Average: 49 2.98 0.00 65.90 0.00 0.00 31.12 Average: 50 2.56 0.00 73.78 0.00 0.00 23.65 Average: 51 2.16 0.00 76.16 0.00 0.00 21.67 Average: 52 2.60 0.00 74.06 0.00 0.00 23.33 Average: 53 2.46 0.00 73.68 0.00 0.00 23.87 Average: 54 2.52 0.00 72.06 0.00 0.00 25.42 Average: 55 2.24 0.00 73.17 0.00 0.00 24.60 Average: 56 2.42 0.00 72.65 0.00 0.00 24.93 Average: 57 2.49 0.00 72.79 0.00 0.00 24.72 Average: 58 2.32 0.00 70.32 0.00 0.00 27.35 Average: 59 1.96 0.00 68.68 0.00 0.00 29.37 Average: 60 1.91 0.00 72.05 0.00 0.00 26.04 Average: 61 2.35 0.00 74.71 0.00 0.00 22.93 Average: 62 1.85 0.00 74.49 0.00 0.00 23.66 Average: 63 2.49 0.00 73.61 0.00 0.00 23.90 Average: 64 2.40 0.00 76.26 0.00 0.00 21.34 Average: 65 1.97 0.00 73.83 0.00 0.00 24.20 Average: 66 1.93 0.00 74.30 0.00 0.00 23.77 Average: 67 2.24 0.00 70.79 0.00 0.00 26.98 Average: 68 2.01 0.00 67.80 0.00 0.00 30.19 Average: 69 1.85 0.00 71.32 0.00 0.00 26.83 Average: 70 3.24 0.00 75.25 0.00 0.00 21.51 Average: 71 3.20 0.00 77.97 0.00 0.00 18.83 Average: 72 3.20 0.00 71.82 0.00 0.00 24.98 Average: 73 2.74 0.00 73.68 0.00 0.00 23.59 Average: 74 2.89 0.00 73.68 0.00 0.00 23.42 Average: 75 3.24 0.00 74.40 0.00 0.00 22.35 Average: 76 2.81 0.00 70.84 0.00 0.00 26.35 Average: 77 2.83 0.00 73.13 0.00 0.00 24.04 Average: 78 2.64 0.00 69.46 0.00 0.00 27.90 Average: 79 2.88 0.00 64.28 0.00 0.00 32.84 [root@nsj17 ~]# free total used free shared buff/cache available Mem: 263961496 19596608 220503484 4236472 23861404 231497040 Swap: 1048572 0 1048572 [root@nsj17 ~]# ==== State: Open by: liujgwx on 12 August 2015 02:52:28 ==== liujgwx (liuj...@cn.ibm.com) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/console output .txt on 2015-08-12 02:52:28 liujgwx (liuj...@cn.ibm.com) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/total pools .txt on 2015-08-12 02:52:28 ==== State: Open by: liujgwx on 12 August 2015 02:45:28 ==== Notice: This Note entry was modified. 1 non-ascii character(s) were replaced with question marks. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1498472/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp