Public bug reported:
Defect:XB300349 New Note: ==== State: Open by: liujgwx on 28 August 2015 07:51:41 ==== add setup infor my setup is openstack RDO juno with docker Controller 1) nsj13 PW: Lab4man1 Public IP: 158.85.164.68 Private IP: 10.53.87.252 Network Node 1) NSJ17 PW: XZ9cXWEr Public IP: 158.85.164.81 Private IP: 10.53.87.139 7) nsj4 PW: Ahzylr2X Public IP: 158.85.164.79 Private IP: 10.53.7.167 compute node 1) nsj19 PW: WPrr5pdm Public IP: 158.85.164.87 Private IP: 10.53.87.153 2) nsj3 PW: Rq3rqj6T Public IP: 158.85.164.78 Private IP: 10.53.87.192 3) nsj14 PW: XdZq6Qjc Public IP: 158.85.164.84 Private IP: 10.53.87.203 4) nsj1 PW; FmzCqH36 Public IP: 158.85.175.87 Private IP: 10.90.7.108 5) nsj30 PW: FGHk6x3y Public IP: 158.85.175.74 Private IP: 10.91.181.76 6) nsj18 PW : F6lWequn Public IP: 158.85.164.88 Private IP: 10.53.7.172 network 10.53.87.139,10.53.7.167 compute 10.53.87.153,10.53.87.192,10.53.87.203,10.90.7.108,10.91.181.76,10.53.7.172 New Note: ==== State: Open by: liujgwx on 28 August 2015 04:55:50 ==== add setup infor my setup is openstack RDO juno with docker Controller 1) nsj13 PW: Lab4man1 Public IP: 158.85.164.68 Private IP: 10.53.87.252 Network Node 1) NSJ17 PW: XZ9cXWEr Public IP: 158.85.164.81 Private IP: 10.53.87.139 7) nsj4 PW: Ahzylr2X Public IP: 158.85.164.79 Private IP: 10.53.7.167 compute node 1) nsj19 PW: WPrr5pdm Public IP: 158.85.164.87 Private IP: 10.53.87.153 2) nsj3 PW: Rq3rqj6T Public IP: 158.85.164.78 Private IP: 10.53.87.192 3) nsj14 PW: XdZq6Qjc Public IP: 158.85.164.84 Private IP: 10.53.87.203 4) nsj1 PW; FmzCqH36 Public IP: 158.85.175.87 Private IP: 10.90.7.108 5) nsj30 PW: FGHk6x3y Public IP: 158.85.175.74 Private IP: 10.91.181.76 6) nsj18 PW : F6lWequn Public IP: 158.85.164.88 Private IP: 10.53.7.172 network 10.53.87.139,10.53.7.167 compute 10.53.87.153,10.53.87.192,10.53.87.203,10.90.7.108,10.91.181.76,10.53.7.172 ==== State: Open by: liujgwx on 28 August 2015 07:51:06 ==== 1 create two-arm LB with 1client and 1 backend server on a tenant 2 repeat step1 to create 375 tenants 3 after step 2 , the LB network unstable [root@nsj13 ~(keystone_admin)]# neutron lb-vip-list |grep ERROR | 054bc376-ff50-40cb-b003-831569b41f0b | Scale1_vip_420 | 20.1.1.100 | HTTP | True | ERROR | | 06fcb386-4563-48d1-b6de-003967a97a54 | Scale1_vip_418 | 20.1.1.100 | HTTP | True | ERROR | | 160f9c87-5cb1-4a5e-9829-cc5de30114c6 | Scale1_vip_444 | 20.1.1.100 | HTTP | True | ERROR | | 1f49c1ca-0dc9-4f6a-9efd-2c741fcc6ff3 | Scale1_vip_380 | 20.1.1.100 | HTTP | True | ERROR | | 25040c46-446d-4e3f-841c-da9166bb4ad0 | Scale1_vip_384 | 20.1.1.100 | HTTP | True | ERROR | | 2690bbdb-30e3-4c6c-b346-f96be83ea745 | Scale1_vip_419 | 20.1.1.100 | HTTP | True | ERROR | | 27e9fbb4-a3c8-4084-af11-c3e71af1a762 | Scale1_vip_431 | 20.1.1.100 | HTTP | True | ERROR | | 283aea00-ba23-403d-a4cd-0d275c36c53f | Scale1_vip_375 | 20.1.1.100 | HTTP | True | ERROR | | 28dbb59d-28a9-4af2-b02a-9f953d4c9dd3 | Scale1_vip_428 | 20.1.1.100 | HTTP | True | ERROR | | 30e99e31-2739-4d1b-a05e-f67d41f26c14 | Scale1_vip_430 | 20.1.1.100 | HTTP | True | ERROR | | 37ee787f-d684-4968-95ea-15666b0cd0e7 | Scale1_vip_410 | 20.1.1.100 | HTTP | True | ERROR | | 38fe0379-2c5a-4bcc-9aab-c0f75466b5bd | Scale1_vip_408 | 20.1.1.100 | HTTP | True | ERROR | | 4190852c-0327-4b05-b1fd-20ef62cf06c0 | Scale1_vip_450 | 20.1.1.100 | HTTP | True | ERROR | | 461a6da3-07e3-47d3-9d47-18ba2add0692 | Scale1_vip_445 | 20.1.1.100 | HTTP | True | ERROR | | 49fb200a-9a73-4d93-a2ef-a5d033377ee2 | Scale1_vip_378 | 20.1.1.100 | HTTP | True | ERROR | | 556a7e3a-d922-43c8-b584-416a8689421d | Scale1_vip_383 | 20.1.1.100 | HTTP | True | ERROR | | 55a500b0-682d-4fc6-93e3-588dfc590b35 | Scale1_vip_421 | 20.1.1.100 | HTTP | True | ERROR | | 5cb8347d-fbcc-497e-ba47-244d5b3f394b | Scale1_vip_429 | 20.1.1.100 | HTTP | True | ERROR | | 606380c4-fd1e-43a6-b67d-01a01c7ad327 | Scale1_vip_412 | 20.1.1.100 | HTTP | True | ERROR | | 60ff74a8-c464-49d0-9f8c-143c7629201e | Scale1_vip_407 | 20.1.1.100 | HTTP | True | ERROR | | 62ce8e28-0c8f-41e1-831d-5ef7500dbb1f | Scale1_vip_453 | 20.1.1.100 | HTTP | True | ERROR | | 691ab6a0-c06e-4e6a-9c7d-b70fc72146e0 | Scale1_vip_436 | 20.1.1.100 | HTTP | True | ERROR | | 6f5c12d8-3640-45b5-a600-19f522227d2c | Scale1_vip_427 | 20.1.1.100 | HTTP | True | ERROR | | 746ab955-20da-41cc-9830-9c33e41abc9c | Scale1_vip_415 | 20.1.1.100 | HTTP | True | ERROR | | 75342fd8-99bd-4527-92f5-7f3d91193adb | Scale1_vip_414 | 20.1.1.100 | HTTP | True | ERROR | | 851311f3-d84d-43e4-9bc2-6fa08c5d4e7e | Scale1_vip_416 | 20.1.1.100 | HTTP | True | ERROR | | 8624e33d-bc23-4b78-bde2-ceb2f12e6b65 | Scale1_vip_417 | 20.1.1.100 | HTTP | True | ERROR | | 895e8ee6-1e78-44e5-a542-3713e07f49cb | Scale1_vip_377 | 20.1.1.100 | HTTP | True | ERROR | | 94855fcf-03fc-431a-b6bb-58abc0f2a084 | Scale1_vip_437 | 20.1.1.100 | HTTP | True | ERROR | | 9af109fa-f301-4942-a559-f6ad75459f3d | Scale1_vip_448 | 20.1.1.100 | HTTP | True | ERROR | | 9cf4bfa3-0073-4a46-a437-2f960ac41d34 | Scale1_vip_422 | 20.1.1.100 | HTTP | True | ERROR | | a138b439-c8b8-459f-85b6-3193d9df5e6d | Scale1_vip_433 | 20.1.1.100 | HTTP | True | ERROR | | a49b0dd1-55ce-4b52-8024-ce8f3219feab | Scale1_vip_435 | 20.1.1.100 | HTTP | True | ERROR | | ae57eadd-3a2d-457c-9149-014d43b2a91b | Scale1_vip_374 | 20.1.1.100 | HTTP | True | ERROR | | b70e7123-3e92-4ad1-bdf3-da58d5ed1a9f | Scale1_vip_426 | 20.1.1.100 | HTTP | True | ERROR | | b79eef81-8631-4012-97ed-b68cf7f5706b | Scale1_vip_434 | 20.1.1.100 | HTTP | True | ERROR | | bffecb85-1992-4c54-a411-8b8064c5b6ec | Scale1_vip_405 | 20.1.1.100 | HTTP | True | ERROR | | c51a6ed5-3134-412c-aad0-320ed7e7aaa1 | Scale1_vip_373 | 20.1.1.100 | HTTP | True | ERROR | | c6b8bb0f-eeca-4f2a-b9be-6162017daa5c | Scale1_vip_424 | 20.1.1.100 | HTTP | True | ERROR | | c8153eaa-fc15-4af4-91ce-eb26c8ff3478 | Scale1_vip_432 | 20.1.1.100 | HTTP | True | ERROR | | c883db3b-675f-443f-8295-6bc4ec35364f | Scale1_vip_423 | 20.1.1.100 | HTTP | True | ERROR | | dec84569-902a-46e4-99ae-71698e9294c0 | Scale1_vip_406 | 20.1.1.100 | HTTP | True | ERROR | | ec0e21a5-4a3a-4ba2-9682-d71234f0aaa4 | Scale1_vip_381 | 20.1.1.100 | HTTP | True | ERROR | | ecbc2bc0-2c43-46a0-86af-e1dc13264131 | Scale1_vip_404 | 20.1.1.100 | HTTP | True | ERROR | | f1149949-8ca3-4f13-8f7b-d7709743e832 | Scale1_vip_413 | 20.1.1.100 | HTTP | True | ERROR | | f2b4a764-ca53-4d15-9f27-db3bb4852fce | Scale1_vip_382 | 20.1.1.100 | HTTP | True | ERROR | | f2d20652-88ef-463f-859e-401987c40916 | Scale1_vip_379 | 20.1.1.100 | HTTP | True | ERROR | | fc62d8bd-82ec-43c6-9bf6-351f5442bea0 | Scale1_vip_446 | 20.1.1.100 | HTTP | True | ERROR | | ff435ed0-20f7-4331-bf4c-0b46f9596f56 | Scale1_vip_425 | 20.1.1.100 | HTTP | True | ERROR | [root@nsj13 ~(keystone_admin)]# neutron lb-pool-list |grep ERROR | 36253c92-8069-4543-bf56-c9a381ba7c1a | pool_462 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 4c252c93-30a0-4495-a1fe-f2e218b2c71e | pool_452 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 5a593359-ff14-45e1-a71e-153010847b08 | pool_461 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 63e982cd-05a1-4f39-a3d6-06130cb5cd91 | pool_455 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 8268f1db-b318-4a93-ab32-8a6dc872792f | pool_460 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 87f48c5e-962b-434f-ba68-f0520d22b3d1 | pool_454 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 8b328094-53e0-4cf4-bb69-90078c61c0fa | pool_457 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 970f73db-abe1-4aec-8189-d1326801fdd7 | pool_456 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | a8d4051d-795b-4c95-b330-fe2cfcffc6bc | pool_449 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | bc3429b4-0b57-4b2e-a737-774ca066ae72 | pool_463 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | bc8c023a-64e3-441a-aed9-fca4296fc876 | pool_459 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | e7339e01-093f-45ba-a1a2-8da33289a494 | pool_451 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | e8a88420-68f5-46ef-a95d-920e6269342e | pool_458 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | fc65aef1-b684-4a17-b1ff-792d25e1145e | pool_447 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | [root@nsj13 ~(keystone_admin)]# openstack-status log [root@nsj4 ~]# tail -n 100 /var/log/neutron/lbaas-agent.log 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 71, in plug_vip_port 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager self.make_msg('plug_vip_port', port_id=port_id, host=self.host) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args']) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID d66dca467e69496595d4464815bbf6e9 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager 2015-08-28 05:44:37.762 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : f02c140b867e4b1c998e7447c2e3d380, message : {u'_unique_id': u'c023475cefc6477db675ebb7d64727a4', u'failure': None, u'result': None} 2015-08-28 05:44:37.765 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.834 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : f02c140b867e4b1c998e7447c2e3d380, message : {u'_unique_id': u'af5e3db8efe04c43b41c819e11bf1880', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:37.835 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.860 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 006fe7ee3eb24367aa8100dda1814910, message : {u'_unique_id': u'035e3f0ce8f24bd6a53671e4116a0b40', u'failure': None, u'result': None} 2015-08-28 05:44:37.862 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.890 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 006fe7ee3eb24367aa8100dda1814910, message : {u'_unique_id': u'8707a91bbc2144e1955c96ab75c0468b', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:37.891 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.044 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : ea968c06105340949c2b130a0786c092, message : {u'_unique_id': u'01ff1c98d63145978fbb734535d20589', u'failure': None, u'result': None} 2015-08-28 05:44:38.046 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.100 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : ea968c06105340949c2b130a0786c092, message : {u'_unique_id': u'c92290f3b4f847f998e5408f88a30d97', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:38.101 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.105 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 88f054e542f742acac0ee271768dbf57, message : {u'_unique_id': u'3740812ef08a4967b487c5ac18dc4671', u'failure': None, u'result': None} 2015-08-28 05:44:38.106 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.196 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 88f054e542f742acac0ee271768dbf57, message : {u'_unique_id': u'adf1bb3cbdc1449e99366a3612a4e219', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:38.197 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:42.359 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : e03acfd5c68445b380988c88b29edcd7, message : {u'_unique_id': u'fcb662f50bf442beb89f5a377350e5a2', u'failure': None, u'result': None} 2015-08-28 05:44:42.360 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:42.462 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : e03acfd5c68445b380988c88b29edcd7, message : {u'_unique_id': u'8a4379403bf54ff89e8edaf112af93c3', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:42.463 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:43.065 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 67786396677c46b7ad7d95ceb1d8c825, message : {u'_unique_id': u'db431d6005964ebda5e8183a1a6a17b3', u'failure': None, u'result': None} 2015-08-28 05:44:43.067 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:43.114 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 67786396677c46b7ad7d95ceb1d8c825, message : {u'_unique_id': u'9cccefdd6e0a49568726b7d591eacf3a', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:43.114 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:45.327 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 596ad3b35e5744f8b7e0675e9cf72987, message : {u'_unique_id': u'f6525182203d4e65ab318a13367fe5aa', u'failure': None, u'result': None} 2015-08-28 05:44:45.327 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:45.383 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 596ad3b35e5744f8b7e0675e9cf72987, message : {u'_unique_id': u'01d81f2e74894d3b965caf162bcec693', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:45.383 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': < [root@nsj4 ~]# sar -P ALL 60 1 Linux 3.10.0-229.11.1.el7.x86_64 (nsj4) 08/28/2015 _x86_64_ (80 CPU) 06:43:31 AM CPU %user %nice %system %iowait %steal %idle 06:44:31 AM all 3.75 0.00 77.35 0.00 0.00 18.90 06:44:31 AM 0 2.93 0.00 87.15 0.00 0.00 9.92 06:44:31 AM 1 2.95 0.00 85.64 0.00 0.00 11.41 06:44:31 AM 2 3.06 0.00 85.34 0.00 0.00 11.60 06:44:31 AM 3 2.93 0.00 84.70 0.00 0.00 12.37 06:44:31 AM 4 2.90 0.00 82.70 0.00 0.00 14.40 06:44:31 AM 5 2.72 0.00 83.16 0.00 0.00 14.12 06:44:31 AM 6 2.48 0.00 77.42 0.00 0.00 20.11 06:44:31 AM 7 3.16 0.00 81.64 0.00 0.00 15.20 06:44:31 AM 8 2.79 0.00 81.67 0.00 0.00 15.54 06:44:31 AM 9 2.85 0.00 82.88 0.00 0.00 14.27 06:44:31 AM 10 2.84 0.00 78.69 0.00 0.00 18.47 06:44:31 AM 11 2.61 0.00 77.22 0.00 0.00 20.17 06:44:31 AM 12 10.38 0.00 72.14 0.00 0.00 17.48 06:44:31 AM 13 10.11 0.00 73.92 0.00 0.00 15.97 06:44:31 AM 14 12.90 0.00 71.64 0.00 0.00 15.45 06:44:31 AM 15 3.15 0.00 77.23 0.00 0.00 19.62 06:44:31 AM 16 9.18 0.00 69.87 0.00 0.00 20.95 06:44:31 AM 17 2.66 0.00 76.96 0.02 0.00 20.37 06:44:31 AM 18 2.37 0.00 79.16 0.00 0.00 18.47 06:44:31 AM 19 8.62 0.00 74.03 0.00 0.00 17.35 06:44:31 AM 20 2.50 0.00 78.26 0.00 0.00 19.24 06:44:31 AM 21 1.85 0.00 82.39 0.00 0.00 15.76 06:44:31 AM 22 1.84 0.00 78.68 0.00 0.00 19.48 06:44:31 AM 23 2.08 0.00 80.77 0.00 0.00 17.15 06:44:31 AM 24 1.65 0.00 77.46 0.00 0.00 20.89 06:44:31 AM 25 1.86 0.00 81.13 0.00 0.00 17.01 06:44:31 AM 26 1.94 0.00 82.37 0.00 0.00 15.69 06:44:31 AM 27 1.58 0.00 81.70 0.00 0.00 16.72 06:44:31 AM 28 1.93 0.00 80.35 0.00 0.00 17.72 06:44:31 AM 29 2.03 0.00 81.64 0.00 0.00 16.33 06:44:31 AM 30 3.25 0.00 80.76 0.00 0.00 15.98 06:44:31 AM 31 3.39 0.00 76.86 0.00 0.00 19.75 06:44:31 AM 32 3.16 0.00 76.93 0.00 0.00 19.91 06:44:31 AM 33 3.33 0.00 76.24 0.00 0.00 20.44 06:44:31 AM 34 3.31 0.00 77.69 0.00 0.00 19.00 06:44:31 AM 35 2.71 0.00 76.26 0.00 0.00 21.04 06:44:31 AM 36 3.12 0.00 77.72 0.00 0.00 19.16 06:44:31 AM 37 2.84 0.00 76.71 0.00 0.00 20.45 06:44:31 AM 38 2.94 0.00 76.91 0.00 0.00 20.15 06:44:31 AM 39 3.45 0.00 75.24 0.00 0.00 21.31 06:44:31 AM 40 3.20 0.00 77.62 0.00 0.00 19.19 06:44:31 AM 41 3.17 0.00 74.88 0.00 0.00 21.95 06:44:31 AM 42 3.12 0.00 75.38 0.00 0.00 21.51 06:44:31 AM 43 3.60 0.00 76.31 0.00 0.00 20.09 06:44:31 AM 44 3.21 0.00 74.74 0.00 0.00 22.05 06:44:31 AM 45 3.23 0.00 77.64 0.00 0.00 19.12 06:44:31 AM 46 3.43 0.00 76.91 0.00 0.00 19.66 06:44:31 AM 47 3.21 0.00 72.91 0.00 0.00 23.88 06:44:31 AM 48 3.36 0.00 76.26 0.00 0.00 20.38 06:44:31 AM 49 3.51 0.00 71.24 0.00 0.00 25.25 06:44:31 AM 50 2.93 0.00 77.88 0.00 0.00 19.19 06:44:31 AM 51 7.24 0.00 76.29 0.00 0.00 16.47 06:44:31 AM 52 16.88 0.00 66.45 0.00 0.00 16.66 06:44:31 AM 53 5.08 0.00 73.77 0.00 0.00 21.15 06:44:31 AM 54 5.10 0.00 73.80 0.00 0.00 21.10 06:44:31 AM 55 2.80 0.00 75.70 0.00 0.00 21.50 06:44:31 AM 56 2.73 0.00 79.63 0.00 0.00 17.64 06:44:31 AM 57 7.22 0.00 74.76 0.00 0.00 18.02 06:44:31 AM 58 11.35 0.00 70.01 0.00 0.00 18.64 06:44:31 AM 59 4.72 0.00 73.01 0.00 0.00 22.27 06:44:31 AM 60 2.38 0.00 77.60 0.00 0.00 20.02 06:44:31 AM 61 1.81 0.00 76.30 0.00 0.00 21.89 06:44:31 AM 62 2.10 0.00 77.76 0.00 0.00 20.15 06:44:31 AM 63 2.04 0.00 76.05 0.00 0.00 21.90 06:44:31 AM 64 2.43 0.00 78.39 0.00 0.00 19.18 06:44:31 AM 65 2.26 0.00 76.61 0.00 0.00 21.12 06:44:31 AM 66 1.86 0.00 76.79 0.00 0.00 21.35 06:44:31 AM 67 1.81 0.00 76.18 0.00 0.00 22.01 06:44:31 AM 68 2.08 0.00 75.54 0.00 0.00 22.38 06:44:31 AM 69 2.38 0.00 73.08 0.00 0.00 24.54 06:44:31 AM 70 3.89 0.00 76.35 0.00 0.00 19.77 06:44:31 AM 71 2.97 0.00 76.53 0.00 0.00 20.50 06:44:31 AM 72 4.56 0.00 75.57 0.00 0.00 19.88 06:44:31 AM 73 3.44 0.00 80.77 0.00 0.00 15.79 06:44:31 AM 74 3.45 0.00 79.40 0.00 0.00 17.14 06:44:31 AM 75 3.13 0.00 77.52 0.00 0.00 19.35 06:44:31 AM 76 3.34 0.00 77.10 0.00 0.00 19.57 06:44:31 AM 77 3.67 0.00 77.45 0.00 0.00 18.88 06:44:31 AM 78 3.21 0.00 76.19 0.00 0.00 20.60 06:44:31 AM 79 3.60 0.00 73.31 0.00 0.00 23.10 Average: CPU %user %nice %system %iowait %steal %idle Average: all 3.75 0.00 77.35 0.00 0.00 18.90 Average: 0 2.93 0.00 87.15 0.00 0.00 9.92 Average: 1 2.95 0.00 85.64 0.00 0.00 11.41 Average: 2 3.06 0.00 85.34 0.00 0.00 11.60 Average: 3 2.93 0.00 84.70 0.00 0.00 12.37 Average: 4 2.90 0.00 82.70 0.00 0.00 14.40 Average: 5 2.72 0.00 83.16 0.00 0.00 14.12 Average: 6 2.48 0.00 77.42 0.00 0.00 20.11 Average: 7 3.16 0.00 81.64 0.00 0.00 15.20 Average: 8 2.79 0.00 81.67 0.00 0.00 15.54 Average: 9 2.85 0.00 82.88 0.00 0.00 14.27 Average: 10 2.84 0.00 78.69 0.00 0.00 18.47 Average: 11 2.61 0.00 77.22 0.00 0.00 20.17 Average: 12 10.38 0.00 72.14 0.00 0.00 17.48 Average: 13 10.11 0.00 73.92 0.00 0.00 15.97 Average: 14 12.90 0.00 71.64 0.00 0.00 15.45 Average: 15 3.15 0.00 77.23 0.00 0.00 19.62 Average: 16 9.18 0.00 69.87 0.00 0.00 20.95 Average: 17 2.66 0.00 76.96 0.02 0.00 20.37 Average: 18 2.37 0.00 79.16 0.00 0.00 18.47 Average: 19 8.62 0.00 74.03 0.00 0.00 17.35 Average: 20 2.50 0.00 78.26 0.00 0.00 19.24 Average: 21 1.85 0.00 82.39 0.00 0.00 15.76 Average: 22 1.84 0.00 78.68 0.00 0.00 19.48 Average: 23 2.08 0.00 80.77 0.00 0.00 17.15 Average: 24 1.65 0.00 77.46 0.00 0.00 20.89 Average: 25 1.86 0.00 81.13 0.00 0.00 17.01 Average: 26 1.94 0.00 82.37 0.00 0.00 15.69 Average: 27 1.58 0.00 81.70 0.00 0.00 16.72 Average: 28 1.93 0.00 80.35 0.00 0.00 17.72 Average: 29 2.03 0.00 81.64 0.00 0.00 16.33 Average: 30 3.25 0.00 80.76 0.00 0.00 15.98 Average: 31 3.39 0.00 76.86 0.00 0.00 19.75 Average: 32 3.16 0.00 76.93 0.00 0.00 19.91 Average: 33 3.33 0.00 76.24 0.00 0.00 20.44 Average: 34 3.31 0.00 77.69 0.00 0.00 19.00 Average: 35 2.71 0.00 76.26 0.00 0.00 21.04 Average: 36 3.12 0.00 77.72 0.00 0.00 19.16 Average: 37 2.84 0.00 76.71 0.00 0.00 20.45 Average: 38 2.94 0.00 76.91 0.00 0.00 20.15 Average: 39 3.45 0.00 75.24 0.00 0.00 21.31 Average: 40 3.20 0.00 77.62 0.00 0.00 19.19 Average: 41 3.17 0.00 74.88 0.00 0.00 21.95 Average: 42 3.12 0.00 75.38 0.00 0.00 21.51 Average: 43 3.60 0.00 76.31 0.00 0.00 20.09 Average: 44 3.21 0.00 74.74 0.00 0.00 22.05 Average: 45 3.23 0.00 77.64 0.00 0.00 19.12 Average: 46 3.43 0.00 76.91 0.00 0.00 19.66 Average: 47 3.21 0.00 72.91 0.00 0.00 23.88 Average: 48 3.36 0.00 76.26 0.00 0.00 20.38 Average: 49 3.51 0.00 71.24 0.00 0.00 25.25 Average: 50 2.93 0.00 77.88 0.00 0.00 19.19 Average: 51 7.24 0.00 76.29 0.00 0.00 16.47 Average: 52 16.88 0.00 66.45 0.00 0.00 16.66 Average: 53 5.08 0.00 73.77 0.00 0.00 21.15 Average: 54 5.10 0.00 73.80 0.00 0.00 21.10 Average: 55 2.80 0.00 75.70 0.00 0.00 21.50 Average: 56 2.73 0.00 79.63 0.00 0.00 17.64 Average: 57 7.22 0.00 74.76 0.00 0.00 18.02 Average: 58 11.35 0.00 70.01 0.00 0.00 18.64 Average: 59 4.72 0.00 73.01 0.00 0.00 22.27 Average: 60 2.38 0.00 77.60 0.00 0.00 20.02 Average: 61 1.81 0.00 76.30 0.00 0.00 21.89 Average: 62 2.10 0.00 77.76 0.00 0.00 20.15 Average: 63 2.04 0.00 76.05 0.00 0.00 21.90 Average: 64 2.43 0.00 78.39 0.00 0.00 19.18 Average: 65 2.26 0.00 76.61 0.00 0.00 21.12 Average: 66 1.86 0.00 76.79 0.00 0.00 21.35 Average: 67 1.81 0.00 76.18 0.00 0.00 22.01 Average: 68 2.08 0.00 75.54 0.00 0.00 22.38 Average: 69 2.38 0.00 73.08 0.00 0.00 24.54 Average: 70 3.89 0.00 76.35 0.00 0.00 19.77 Average: 71 2.97 0.00 76.53 0.00 0.00 20.50 Average: 72 4.56 0.00 75.57 0.00 0.00 19.88 Average: 73 3.44 0.00 80.77 0.00 0.00 15.79 Average: 74 3.45 0.00 79.40 0.00 0.00 17.14 Average: 75 3.13 0.00 77.52 0.00 0.00 19.35 Average: 76 3.34 0.00 77.10 0.00 0.00 19.57 Average: 77 3.67 0.00 77.45 0.00 0.00 18.88 Average: 78 3.21 0.00 76.19 0.00 0.00 20.60 Average: 79 3.60 0.00 73.31 0.00 0.00 23.10 [root@nsj4 ~]# free total used free shared buff/cache available Mem: 263961496 13581348 243980080 1533280 6400068 245304560 Swap: 1048572 0 1048572 [root@nsj4 ~]# tail -n 100 /var/log/neutron/lbaas-agent.log ** Affects: neutron Importance: Undecided Status: New ** Tags: lbaas -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1498359 Title: after create 375 LB pool , the new lb -pool and vip get in error status Status in neutron: New Bug description: Defect:XB300349 New Note: ==== State: Open by: liujgwx on 28 August 2015 07:51:41 ==== add setup infor my setup is openstack RDO juno with docker Controller 1) nsj13 PW: Lab4man1 Public IP: 158.85.164.68 Private IP: 10.53.87.252 Network Node 1) NSJ17 PW: XZ9cXWEr Public IP: 158.85.164.81 Private IP: 10.53.87.139 7) nsj4 PW: Ahzylr2X Public IP: 158.85.164.79 Private IP: 10.53.7.167 compute node 1) nsj19 PW: WPrr5pdm Public IP: 158.85.164.87 Private IP: 10.53.87.153 2) nsj3 PW: Rq3rqj6T Public IP: 158.85.164.78 Private IP: 10.53.87.192 3) nsj14 PW: XdZq6Qjc Public IP: 158.85.164.84 Private IP: 10.53.87.203 4) nsj1 PW; FmzCqH36 Public IP: 158.85.175.87 Private IP: 10.90.7.108 5) nsj30 PW: FGHk6x3y Public IP: 158.85.175.74 Private IP: 10.91.181.76 6) nsj18 PW : F6lWequn Public IP: 158.85.164.88 Private IP: 10.53.7.172 network 10.53.87.139,10.53.7.167 compute 10.53.87.153,10.53.87.192,10.53.87.203,10.90.7.108,10.91.181.76,10.53.7.172 New Note: ==== State: Open by: liujgwx on 28 August 2015 04:55:50 ==== add setup infor my setup is openstack RDO juno with docker Controller 1) nsj13 PW: Lab4man1 Public IP: 158.85.164.68 Private IP: 10.53.87.252 Network Node 1) NSJ17 PW: XZ9cXWEr Public IP: 158.85.164.81 Private IP: 10.53.87.139 7) nsj4 PW: Ahzylr2X Public IP: 158.85.164.79 Private IP: 10.53.7.167 compute node 1) nsj19 PW: WPrr5pdm Public IP: 158.85.164.87 Private IP: 10.53.87.153 2) nsj3 PW: Rq3rqj6T Public IP: 158.85.164.78 Private IP: 10.53.87.192 3) nsj14 PW: XdZq6Qjc Public IP: 158.85.164.84 Private IP: 10.53.87.203 4) nsj1 PW; FmzCqH36 Public IP: 158.85.175.87 Private IP: 10.90.7.108 5) nsj30 PW: FGHk6x3y Public IP: 158.85.175.74 Private IP: 10.91.181.76 6) nsj18 PW : F6lWequn Public IP: 158.85.164.88 Private IP: 10.53.7.172 network 10.53.87.139,10.53.7.167 compute 10.53.87.153,10.53.87.192,10.53.87.203,10.90.7.108,10.91.181.76,10.53.7.172 ==== State: Open by: liujgwx on 28 August 2015 07:51:06 ==== 1 create two-arm LB with 1client and 1 backend server on a tenant 2 repeat step1 to create 375 tenants 3 after step 2 , the LB network unstable [root@nsj13 ~(keystone_admin)]# neutron lb-vip-list |grep ERROR | 054bc376-ff50-40cb-b003-831569b41f0b | Scale1_vip_420 | 20.1.1.100 | HTTP | True | ERROR | | 06fcb386-4563-48d1-b6de-003967a97a54 | Scale1_vip_418 | 20.1.1.100 | HTTP | True | ERROR | | 160f9c87-5cb1-4a5e-9829-cc5de30114c6 | Scale1_vip_444 | 20.1.1.100 | HTTP | True | ERROR | | 1f49c1ca-0dc9-4f6a-9efd-2c741fcc6ff3 | Scale1_vip_380 | 20.1.1.100 | HTTP | True | ERROR | | 25040c46-446d-4e3f-841c-da9166bb4ad0 | Scale1_vip_384 | 20.1.1.100 | HTTP | True | ERROR | | 2690bbdb-30e3-4c6c-b346-f96be83ea745 | Scale1_vip_419 | 20.1.1.100 | HTTP | True | ERROR | | 27e9fbb4-a3c8-4084-af11-c3e71af1a762 | Scale1_vip_431 | 20.1.1.100 | HTTP | True | ERROR | | 283aea00-ba23-403d-a4cd-0d275c36c53f | Scale1_vip_375 | 20.1.1.100 | HTTP | True | ERROR | | 28dbb59d-28a9-4af2-b02a-9f953d4c9dd3 | Scale1_vip_428 | 20.1.1.100 | HTTP | True | ERROR | | 30e99e31-2739-4d1b-a05e-f67d41f26c14 | Scale1_vip_430 | 20.1.1.100 | HTTP | True | ERROR | | 37ee787f-d684-4968-95ea-15666b0cd0e7 | Scale1_vip_410 | 20.1.1.100 | HTTP | True | ERROR | | 38fe0379-2c5a-4bcc-9aab-c0f75466b5bd | Scale1_vip_408 | 20.1.1.100 | HTTP | True | ERROR | | 4190852c-0327-4b05-b1fd-20ef62cf06c0 | Scale1_vip_450 | 20.1.1.100 | HTTP | True | ERROR | | 461a6da3-07e3-47d3-9d47-18ba2add0692 | Scale1_vip_445 | 20.1.1.100 | HTTP | True | ERROR | | 49fb200a-9a73-4d93-a2ef-a5d033377ee2 | Scale1_vip_378 | 20.1.1.100 | HTTP | True | ERROR | | 556a7e3a-d922-43c8-b584-416a8689421d | Scale1_vip_383 | 20.1.1.100 | HTTP | True | ERROR | | 55a500b0-682d-4fc6-93e3-588dfc590b35 | Scale1_vip_421 | 20.1.1.100 | HTTP | True | ERROR | | 5cb8347d-fbcc-497e-ba47-244d5b3f394b | Scale1_vip_429 | 20.1.1.100 | HTTP | True | ERROR | | 606380c4-fd1e-43a6-b67d-01a01c7ad327 | Scale1_vip_412 | 20.1.1.100 | HTTP | True | ERROR | | 60ff74a8-c464-49d0-9f8c-143c7629201e | Scale1_vip_407 | 20.1.1.100 | HTTP | True | ERROR | | 62ce8e28-0c8f-41e1-831d-5ef7500dbb1f | Scale1_vip_453 | 20.1.1.100 | HTTP | True | ERROR | | 691ab6a0-c06e-4e6a-9c7d-b70fc72146e0 | Scale1_vip_436 | 20.1.1.100 | HTTP | True | ERROR | | 6f5c12d8-3640-45b5-a600-19f522227d2c | Scale1_vip_427 | 20.1.1.100 | HTTP | True | ERROR | | 746ab955-20da-41cc-9830-9c33e41abc9c | Scale1_vip_415 | 20.1.1.100 | HTTP | True | ERROR | | 75342fd8-99bd-4527-92f5-7f3d91193adb | Scale1_vip_414 | 20.1.1.100 | HTTP | True | ERROR | | 851311f3-d84d-43e4-9bc2-6fa08c5d4e7e | Scale1_vip_416 | 20.1.1.100 | HTTP | True | ERROR | | 8624e33d-bc23-4b78-bde2-ceb2f12e6b65 | Scale1_vip_417 | 20.1.1.100 | HTTP | True | ERROR | | 895e8ee6-1e78-44e5-a542-3713e07f49cb | Scale1_vip_377 | 20.1.1.100 | HTTP | True | ERROR | | 94855fcf-03fc-431a-b6bb-58abc0f2a084 | Scale1_vip_437 | 20.1.1.100 | HTTP | True | ERROR | | 9af109fa-f301-4942-a559-f6ad75459f3d | Scale1_vip_448 | 20.1.1.100 | HTTP | True | ERROR | | 9cf4bfa3-0073-4a46-a437-2f960ac41d34 | Scale1_vip_422 | 20.1.1.100 | HTTP | True | ERROR | | a138b439-c8b8-459f-85b6-3193d9df5e6d | Scale1_vip_433 | 20.1.1.100 | HTTP | True | ERROR | | a49b0dd1-55ce-4b52-8024-ce8f3219feab | Scale1_vip_435 | 20.1.1.100 | HTTP | True | ERROR | | ae57eadd-3a2d-457c-9149-014d43b2a91b | Scale1_vip_374 | 20.1.1.100 | HTTP | True | ERROR | | b70e7123-3e92-4ad1-bdf3-da58d5ed1a9f | Scale1_vip_426 | 20.1.1.100 | HTTP | True | ERROR | | b79eef81-8631-4012-97ed-b68cf7f5706b | Scale1_vip_434 | 20.1.1.100 | HTTP | True | ERROR | | bffecb85-1992-4c54-a411-8b8064c5b6ec | Scale1_vip_405 | 20.1.1.100 | HTTP | True | ERROR | | c51a6ed5-3134-412c-aad0-320ed7e7aaa1 | Scale1_vip_373 | 20.1.1.100 | HTTP | True | ERROR | | c6b8bb0f-eeca-4f2a-b9be-6162017daa5c | Scale1_vip_424 | 20.1.1.100 | HTTP | True | ERROR | | c8153eaa-fc15-4af4-91ce-eb26c8ff3478 | Scale1_vip_432 | 20.1.1.100 | HTTP | True | ERROR | | c883db3b-675f-443f-8295-6bc4ec35364f | Scale1_vip_423 | 20.1.1.100 | HTTP | True | ERROR | | dec84569-902a-46e4-99ae-71698e9294c0 | Scale1_vip_406 | 20.1.1.100 | HTTP | True | ERROR | | ec0e21a5-4a3a-4ba2-9682-d71234f0aaa4 | Scale1_vip_381 | 20.1.1.100 | HTTP | True | ERROR | | ecbc2bc0-2c43-46a0-86af-e1dc13264131 | Scale1_vip_404 | 20.1.1.100 | HTTP | True | ERROR | | f1149949-8ca3-4f13-8f7b-d7709743e832 | Scale1_vip_413 | 20.1.1.100 | HTTP | True | ERROR | | f2b4a764-ca53-4d15-9f27-db3bb4852fce | Scale1_vip_382 | 20.1.1.100 | HTTP | True | ERROR | | f2d20652-88ef-463f-859e-401987c40916 | Scale1_vip_379 | 20.1.1.100 | HTTP | True | ERROR | | fc62d8bd-82ec-43c6-9bf6-351f5442bea0 | Scale1_vip_446 | 20.1.1.100 | HTTP | True | ERROR | | ff435ed0-20f7-4331-bf4c-0b46f9596f56 | Scale1_vip_425 | 20.1.1.100 | HTTP | True | ERROR | [root@nsj13 ~(keystone_admin)]# neutron lb-pool-list |grep ERROR | 36253c92-8069-4543-bf56-c9a381ba7c1a | pool_462 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 4c252c93-30a0-4495-a1fe-f2e218b2c71e | pool_452 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 5a593359-ff14-45e1-a71e-153010847b08 | pool_461 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 63e982cd-05a1-4f39-a3d6-06130cb5cd91 | pool_455 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 8268f1db-b318-4a93-ab32-8a6dc872792f | pool_460 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 87f48c5e-962b-434f-ba68-f0520d22b3d1 | pool_454 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 8b328094-53e0-4cf4-bb69-90078c61c0fa | pool_457 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | 970f73db-abe1-4aec-8189-d1326801fdd7 | pool_456 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | a8d4051d-795b-4c95-b330-fe2cfcffc6bc | pool_449 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | bc3429b4-0b57-4b2e-a737-774ca066ae72 | pool_463 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | bc8c023a-64e3-441a-aed9-fca4296fc876 | pool_459 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | e7339e01-093f-45ba-a1a2-8da33289a494 | pool_451 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | e8a88420-68f5-46ef-a95d-920e6269342e | pool_458 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | | fc65aef1-b684-4a17-b1ff-792d25e1145e | pool_447 | haproxy | ROUND_ROBIN | HTTP | True | ERROR | [root@nsj13 ~(keystone_admin)]# openstack-status log [root@nsj4 ~]# tail -n 100 /var/log/neutron/lbaas-agent.log 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 71, in plug_vip_port 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager self.make_msg('plug_vip_port', port_id=port_id, host=self.host) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args']) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id) 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID d66dca467e69496595d4464815bbf6e9 2015-08-28 05:43:35.765 40095 TRACE neutron.services.loadbalancer.agent.agent_manager 2015-08-28 05:44:37.762 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : f02c140b867e4b1c998e7447c2e3d380, message : {u'_unique_id': u'c023475cefc6477db675ebb7d64727a4', u'failure': None, u'result': None} 2015-08-28 05:44:37.765 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.834 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : f02c140b867e4b1c998e7447c2e3d380, message : {u'_unique_id': u'af5e3db8efe04c43b41c819e11bf1880', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:37.835 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.860 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 006fe7ee3eb24367aa8100dda1814910, message : {u'_unique_id': u'035e3f0ce8f24bd6a53671e4116a0b40', u'failure': None, u'result': None} 2015-08-28 05:44:37.862 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:37.890 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 006fe7ee3eb24367aa8100dda1814910, message : {u'_unique_id': u'8707a91bbc2144e1955c96ab75c0468b', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:37.891 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.044 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : ea968c06105340949c2b130a0786c092, message : {u'_unique_id': u'01ff1c98d63145978fbb734535d20589', u'failure': None, u'result': None} 2015-08-28 05:44:38.046 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.100 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : ea968c06105340949c2b130a0786c092, message : {u'_unique_id': u'c92290f3b4f847f998e5408f88a30d97', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:38.101 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.105 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 88f054e542f742acac0ee271768dbf57, message : {u'_unique_id': u'3740812ef08a4967b487c5ac18dc4671', u'failure': None, u'result': None} 2015-08-28 05:44:38.106 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:38.196 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 88f054e542f742acac0ee271768dbf57, message : {u'_unique_id': u'adf1bb3cbdc1449e99366a3612a4e219', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:38.197 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:42.359 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : e03acfd5c68445b380988c88b29edcd7, message : {u'_unique_id': u'fcb662f50bf442beb89f5a377350e5a2', u'failure': None, u'result': None} 2015-08-28 05:44:42.360 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:42.462 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : e03acfd5c68445b380988c88b29edcd7, message : {u'_unique_id': u'8a4379403bf54ff89e8edaf112af93c3', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:42.463 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:43.065 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 67786396677c46b7ad7d95ceb1d8c825, message : {u'_unique_id': u'db431d6005964ebda5e8183a1a6a17b3', u'failure': None, u'result': None} 2015-08-28 05:44:43.067 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:43.114 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 67786396677c46b7ad7d95ceb1d8c825, message : {u'_unique_id': u'9cccefdd6e0a49568726b7d591eacf3a', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:43.114 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:45.327 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 596ad3b35e5744f8b7e0675e9cf72987, message : {u'_unique_id': u'f6525182203d4e65ab318a13367fe5aa', u'failure': None, u'result': None} 2015-08-28 05:44:45.327 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': <Queue at 0x369db10 maxsize=None tasks=1 _cond=<Event at 0x372ab90 result=NOT_USED _exc=None _waiters[0]>>} 2015-08-28 05:44:45.383 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] No calling threads waiting for msg_id : 596ad3b35e5744f8b7e0675e9cf72987, message : {u'_unique_id': u'01d81f2e74894d3b965caf162bcec693', u'failure': None, u'result': None, u'ending': True} 2015-08-28 05:44:45.383 40095 WARNING oslo.messaging._drivers.amqpdriver [req-a2a28e4a-9e09-426b-b176-c3c5f16be16d ] _queues: {'5afbf8f442dc495d8290298e9c06bc9e': < [root@nsj4 ~]# sar -P ALL 60 1 Linux 3.10.0-229.11.1.el7.x86_64 (nsj4) 08/28/2015 _x86_64_ (80 CPU) 06:43:31 AM CPU %user %nice %system %iowait %steal %idle 06:44:31 AM all 3.75 0.00 77.35 0.00 0.00 18.90 06:44:31 AM 0 2.93 0.00 87.15 0.00 0.00 9.92 06:44:31 AM 1 2.95 0.00 85.64 0.00 0.00 11.41 06:44:31 AM 2 3.06 0.00 85.34 0.00 0.00 11.60 06:44:31 AM 3 2.93 0.00 84.70 0.00 0.00 12.37 06:44:31 AM 4 2.90 0.00 82.70 0.00 0.00 14.40 06:44:31 AM 5 2.72 0.00 83.16 0.00 0.00 14.12 06:44:31 AM 6 2.48 0.00 77.42 0.00 0.00 20.11 06:44:31 AM 7 3.16 0.00 81.64 0.00 0.00 15.20 06:44:31 AM 8 2.79 0.00 81.67 0.00 0.00 15.54 06:44:31 AM 9 2.85 0.00 82.88 0.00 0.00 14.27 06:44:31 AM 10 2.84 0.00 78.69 0.00 0.00 18.47 06:44:31 AM 11 2.61 0.00 77.22 0.00 0.00 20.17 06:44:31 AM 12 10.38 0.00 72.14 0.00 0.00 17.48 06:44:31 AM 13 10.11 0.00 73.92 0.00 0.00 15.97 06:44:31 AM 14 12.90 0.00 71.64 0.00 0.00 15.45 06:44:31 AM 15 3.15 0.00 77.23 0.00 0.00 19.62 06:44:31 AM 16 9.18 0.00 69.87 0.00 0.00 20.95 06:44:31 AM 17 2.66 0.00 76.96 0.02 0.00 20.37 06:44:31 AM 18 2.37 0.00 79.16 0.00 0.00 18.47 06:44:31 AM 19 8.62 0.00 74.03 0.00 0.00 17.35 06:44:31 AM 20 2.50 0.00 78.26 0.00 0.00 19.24 06:44:31 AM 21 1.85 0.00 82.39 0.00 0.00 15.76 06:44:31 AM 22 1.84 0.00 78.68 0.00 0.00 19.48 06:44:31 AM 23 2.08 0.00 80.77 0.00 0.00 17.15 06:44:31 AM 24 1.65 0.00 77.46 0.00 0.00 20.89 06:44:31 AM 25 1.86 0.00 81.13 0.00 0.00 17.01 06:44:31 AM 26 1.94 0.00 82.37 0.00 0.00 15.69 06:44:31 AM 27 1.58 0.00 81.70 0.00 0.00 16.72 06:44:31 AM 28 1.93 0.00 80.35 0.00 0.00 17.72 06:44:31 AM 29 2.03 0.00 81.64 0.00 0.00 16.33 06:44:31 AM 30 3.25 0.00 80.76 0.00 0.00 15.98 06:44:31 AM 31 3.39 0.00 76.86 0.00 0.00 19.75 06:44:31 AM 32 3.16 0.00 76.93 0.00 0.00 19.91 06:44:31 AM 33 3.33 0.00 76.24 0.00 0.00 20.44 06:44:31 AM 34 3.31 0.00 77.69 0.00 0.00 19.00 06:44:31 AM 35 2.71 0.00 76.26 0.00 0.00 21.04 06:44:31 AM 36 3.12 0.00 77.72 0.00 0.00 19.16 06:44:31 AM 37 2.84 0.00 76.71 0.00 0.00 20.45 06:44:31 AM 38 2.94 0.00 76.91 0.00 0.00 20.15 06:44:31 AM 39 3.45 0.00 75.24 0.00 0.00 21.31 06:44:31 AM 40 3.20 0.00 77.62 0.00 0.00 19.19 06:44:31 AM 41 3.17 0.00 74.88 0.00 0.00 21.95 06:44:31 AM 42 3.12 0.00 75.38 0.00 0.00 21.51 06:44:31 AM 43 3.60 0.00 76.31 0.00 0.00 20.09 06:44:31 AM 44 3.21 0.00 74.74 0.00 0.00 22.05 06:44:31 AM 45 3.23 0.00 77.64 0.00 0.00 19.12 06:44:31 AM 46 3.43 0.00 76.91 0.00 0.00 19.66 06:44:31 AM 47 3.21 0.00 72.91 0.00 0.00 23.88 06:44:31 AM 48 3.36 0.00 76.26 0.00 0.00 20.38 06:44:31 AM 49 3.51 0.00 71.24 0.00 0.00 25.25 06:44:31 AM 50 2.93 0.00 77.88 0.00 0.00 19.19 06:44:31 AM 51 7.24 0.00 76.29 0.00 0.00 16.47 06:44:31 AM 52 16.88 0.00 66.45 0.00 0.00 16.66 06:44:31 AM 53 5.08 0.00 73.77 0.00 0.00 21.15 06:44:31 AM 54 5.10 0.00 73.80 0.00 0.00 21.10 06:44:31 AM 55 2.80 0.00 75.70 0.00 0.00 21.50 06:44:31 AM 56 2.73 0.00 79.63 0.00 0.00 17.64 06:44:31 AM 57 7.22 0.00 74.76 0.00 0.00 18.02 06:44:31 AM 58 11.35 0.00 70.01 0.00 0.00 18.64 06:44:31 AM 59 4.72 0.00 73.01 0.00 0.00 22.27 06:44:31 AM 60 2.38 0.00 77.60 0.00 0.00 20.02 06:44:31 AM 61 1.81 0.00 76.30 0.00 0.00 21.89 06:44:31 AM 62 2.10 0.00 77.76 0.00 0.00 20.15 06:44:31 AM 63 2.04 0.00 76.05 0.00 0.00 21.90 06:44:31 AM 64 2.43 0.00 78.39 0.00 0.00 19.18 06:44:31 AM 65 2.26 0.00 76.61 0.00 0.00 21.12 06:44:31 AM 66 1.86 0.00 76.79 0.00 0.00 21.35 06:44:31 AM 67 1.81 0.00 76.18 0.00 0.00 22.01 06:44:31 AM 68 2.08 0.00 75.54 0.00 0.00 22.38 06:44:31 AM 69 2.38 0.00 73.08 0.00 0.00 24.54 06:44:31 AM 70 3.89 0.00 76.35 0.00 0.00 19.77 06:44:31 AM 71 2.97 0.00 76.53 0.00 0.00 20.50 06:44:31 AM 72 4.56 0.00 75.57 0.00 0.00 19.88 06:44:31 AM 73 3.44 0.00 80.77 0.00 0.00 15.79 06:44:31 AM 74 3.45 0.00 79.40 0.00 0.00 17.14 06:44:31 AM 75 3.13 0.00 77.52 0.00 0.00 19.35 06:44:31 AM 76 3.34 0.00 77.10 0.00 0.00 19.57 06:44:31 AM 77 3.67 0.00 77.45 0.00 0.00 18.88 06:44:31 AM 78 3.21 0.00 76.19 0.00 0.00 20.60 06:44:31 AM 79 3.60 0.00 73.31 0.00 0.00 23.10 Average: CPU %user %nice %system %iowait %steal %idle Average: all 3.75 0.00 77.35 0.00 0.00 18.90 Average: 0 2.93 0.00 87.15 0.00 0.00 9.92 Average: 1 2.95 0.00 85.64 0.00 0.00 11.41 Average: 2 3.06 0.00 85.34 0.00 0.00 11.60 Average: 3 2.93 0.00 84.70 0.00 0.00 12.37 Average: 4 2.90 0.00 82.70 0.00 0.00 14.40 Average: 5 2.72 0.00 83.16 0.00 0.00 14.12 Average: 6 2.48 0.00 77.42 0.00 0.00 20.11 Average: 7 3.16 0.00 81.64 0.00 0.00 15.20 Average: 8 2.79 0.00 81.67 0.00 0.00 15.54 Average: 9 2.85 0.00 82.88 0.00 0.00 14.27 Average: 10 2.84 0.00 78.69 0.00 0.00 18.47 Average: 11 2.61 0.00 77.22 0.00 0.00 20.17 Average: 12 10.38 0.00 72.14 0.00 0.00 17.48 Average: 13 10.11 0.00 73.92 0.00 0.00 15.97 Average: 14 12.90 0.00 71.64 0.00 0.00 15.45 Average: 15 3.15 0.00 77.23 0.00 0.00 19.62 Average: 16 9.18 0.00 69.87 0.00 0.00 20.95 Average: 17 2.66 0.00 76.96 0.02 0.00 20.37 Average: 18 2.37 0.00 79.16 0.00 0.00 18.47 Average: 19 8.62 0.00 74.03 0.00 0.00 17.35 Average: 20 2.50 0.00 78.26 0.00 0.00 19.24 Average: 21 1.85 0.00 82.39 0.00 0.00 15.76 Average: 22 1.84 0.00 78.68 0.00 0.00 19.48 Average: 23 2.08 0.00 80.77 0.00 0.00 17.15 Average: 24 1.65 0.00 77.46 0.00 0.00 20.89 Average: 25 1.86 0.00 81.13 0.00 0.00 17.01 Average: 26 1.94 0.00 82.37 0.00 0.00 15.69 Average: 27 1.58 0.00 81.70 0.00 0.00 16.72 Average: 28 1.93 0.00 80.35 0.00 0.00 17.72 Average: 29 2.03 0.00 81.64 0.00 0.00 16.33 Average: 30 3.25 0.00 80.76 0.00 0.00 15.98 Average: 31 3.39 0.00 76.86 0.00 0.00 19.75 Average: 32 3.16 0.00 76.93 0.00 0.00 19.91 Average: 33 3.33 0.00 76.24 0.00 0.00 20.44 Average: 34 3.31 0.00 77.69 0.00 0.00 19.00 Average: 35 2.71 0.00 76.26 0.00 0.00 21.04 Average: 36 3.12 0.00 77.72 0.00 0.00 19.16 Average: 37 2.84 0.00 76.71 0.00 0.00 20.45 Average: 38 2.94 0.00 76.91 0.00 0.00 20.15 Average: 39 3.45 0.00 75.24 0.00 0.00 21.31 Average: 40 3.20 0.00 77.62 0.00 0.00 19.19 Average: 41 3.17 0.00 74.88 0.00 0.00 21.95 Average: 42 3.12 0.00 75.38 0.00 0.00 21.51 Average: 43 3.60 0.00 76.31 0.00 0.00 20.09 Average: 44 3.21 0.00 74.74 0.00 0.00 22.05 Average: 45 3.23 0.00 77.64 0.00 0.00 19.12 Average: 46 3.43 0.00 76.91 0.00 0.00 19.66 Average: 47 3.21 0.00 72.91 0.00 0.00 23.88 Average: 48 3.36 0.00 76.26 0.00 0.00 20.38 Average: 49 3.51 0.00 71.24 0.00 0.00 25.25 Average: 50 2.93 0.00 77.88 0.00 0.00 19.19 Average: 51 7.24 0.00 76.29 0.00 0.00 16.47 Average: 52 16.88 0.00 66.45 0.00 0.00 16.66 Average: 53 5.08 0.00 73.77 0.00 0.00 21.15 Average: 54 5.10 0.00 73.80 0.00 0.00 21.10 Average: 55 2.80 0.00 75.70 0.00 0.00 21.50 Average: 56 2.73 0.00 79.63 0.00 0.00 17.64 Average: 57 7.22 0.00 74.76 0.00 0.00 18.02 Average: 58 11.35 0.00 70.01 0.00 0.00 18.64 Average: 59 4.72 0.00 73.01 0.00 0.00 22.27 Average: 60 2.38 0.00 77.60 0.00 0.00 20.02 Average: 61 1.81 0.00 76.30 0.00 0.00 21.89 Average: 62 2.10 0.00 77.76 0.00 0.00 20.15 Average: 63 2.04 0.00 76.05 0.00 0.00 21.90 Average: 64 2.43 0.00 78.39 0.00 0.00 19.18 Average: 65 2.26 0.00 76.61 0.00 0.00 21.12 Average: 66 1.86 0.00 76.79 0.00 0.00 21.35 Average: 67 1.81 0.00 76.18 0.00 0.00 22.01 Average: 68 2.08 0.00 75.54 0.00 0.00 22.38 Average: 69 2.38 0.00 73.08 0.00 0.00 24.54 Average: 70 3.89 0.00 76.35 0.00 0.00 19.77 Average: 71 2.97 0.00 76.53 0.00 0.00 20.50 Average: 72 4.56 0.00 75.57 0.00 0.00 19.88 Average: 73 3.44 0.00 80.77 0.00 0.00 15.79 Average: 74 3.45 0.00 79.40 0.00 0.00 17.14 Average: 75 3.13 0.00 77.52 0.00 0.00 19.35 Average: 76 3.34 0.00 77.10 0.00 0.00 19.57 Average: 77 3.67 0.00 77.45 0.00 0.00 18.88 Average: 78 3.21 0.00 76.19 0.00 0.00 20.60 Average: 79 3.60 0.00 73.31 0.00 0.00 23.10 [root@nsj4 ~]# free total used free shared buff/cache available Mem: 263961496 13581348 243980080 1533280 6400068 245304560 Swap: 1048572 0 1048572 [root@nsj4 ~]# tail -n 100 /var/log/neutron/lbaas-agent.log To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1498359/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp