Public bug reported:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on sinalge tenant
with 30M traffic on each LB
CentOS Linux release 7.1.1503 (Core)
reproduce step
1 create 1 client and 1 backend server by docker container
2 create a lb and add backend as member
3 send 30M traffic with 100 session by siege
setsid siege -t1000H {vip_ip_address}/test_150KB.iso -c 100
4 repeat step 1 to 3 for create 200 LB with traffic
after step 4 , you will look at a prompt on LB agent host node
and have some traffic will can not pass on LB from client and can't ping to vip
address from client container
log
[root@scalenetwork1 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
Message from syslogd@scalenetwork1 at Sep 13 23:24:05 ...
kernel:BUG: soft lockup - CPU#5 stuck for 23s! [rcuos/6:24]
<<<<<<<<<<<<<<<<<<<<
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]# sar -P ALL 60 1
Linux 3.10.0-229.11.1.el7.x86_64 (scalenetwork1) 09/13/2015
_x86_64_ (8 CPU)
11:24:22 PM CPU %user %nice %system %iowait %steal %idle
11:25:22 PM all 11.71 0.00 43.13 0.00 0.00 45.17
11:25:22 PM 0 13.81 0.00 34.69 0.00 0.00 51.49
11:25:22 PM 1 6.11 0.00 19.80 0.00 0.00 74.10
11:25:22 PM 2 14.09 0.00 34.71 0.00 0.00 51.21
11:25:22 PM 3 13.66 0.00 35.00 0.00 0.00 51.34
11:25:22 PM 4 15.34 0.00 38.34 0.00 0.00 46.32
11:25:22 PM 5 0.05 0.00 99.78 0.00 0.00 0.17
11:25:22 PM 6 14.95 0.00 41.34 0.00 0.00 43.71
11:25:22 PM 7 15.80 0.00 39.70 0.00 0.00 44.50
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 11.71 0.00 43.13 0.00 0.00 45.17
Average: 0 13.81 0.00 34.69 0.00 0.00 51.49
Average: 1 6.11 0.00 19.80 0.00 0.00 74.10
Average: 2 14.09 0.00 34.71 0.00 0.00 51.21
Average: 3 13.66 0.00 35.00 0.00 0.00 51.34
Average: 4 15.34 0.00 38.34 0.00 0.00 46.32
Average: 5 0.05 0.00 99.78 0.00 0.00 0.17
Average: 6 14.95 0.00 41.34 0.00 0.00 43.71
Average: 7 15.80 0.00 39.70 0.00 0.00 44.50
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1498506
Title:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on
sinalge tenant with 30M traffic on each LB
Status in neutron:
New
Bug description:
LB_NFV[Alchemy] linux kernel soft lockup after create 200 LB on sinalge
tenant with 30M traffic on each LB
CentOS Linux release 7.1.1503 (Core)
reproduce step
1 create 1 client and 1 backend server by docker container
2 create a lb and add backend as member
3 send 30M traffic with 100 session by siege
setsid siege -t1000H {vip_ip_address}/test_150KB.iso -c 100
4 repeat step 1 to 3 for create 200 LB with traffic
after step 4 , you will look at a prompt on LB agent host node
and have some traffic will can not pass on LB from client and can't ping to
vip address from client container
log
[root@scalenetwork1 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
Message from syslogd@scalenetwork1 at Sep 13 23:24:05 ...
kernel:BUG: soft lockup - CPU#5 stuck for 23s! [rcuos/6:24]
<<<<<<<<<<<<<<<<<<<<
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]#
[root@scalenetwork1 ~]# sar -P ALL 60 1
Linux 3.10.0-229.11.1.el7.x86_64 (scalenetwork1) 09/13/2015
_x86_64_ (8 CPU)
11:24:22 PM CPU %user %nice %system %iowait %steal
%idle
11:25:22 PM all 11.71 0.00 43.13 0.00 0.00
45.17
11:25:22 PM 0 13.81 0.00 34.69 0.00 0.00
51.49
11:25:22 PM 1 6.11 0.00 19.80 0.00 0.00
74.10
11:25:22 PM 2 14.09 0.00 34.71 0.00 0.00
51.21
11:25:22 PM 3 13.66 0.00 35.00 0.00 0.00
51.34
11:25:22 PM 4 15.34 0.00 38.34 0.00 0.00
46.32
11:25:22 PM 5 0.05 0.00 99.78 0.00 0.00
0.17
11:25:22 PM 6 14.95 0.00 41.34 0.00 0.00
43.71
11:25:22 PM 7 15.80 0.00 39.70 0.00 0.00
44.50
Average: CPU %user %nice %system %iowait %steal
%idle
Average: all 11.71 0.00 43.13 0.00 0.00
45.17
Average: 0 13.81 0.00 34.69 0.00 0.00
51.49
Average: 1 6.11 0.00 19.80 0.00 0.00
74.10
Average: 2 14.09 0.00 34.71 0.00 0.00
51.21
Average: 3 13.66 0.00 35.00 0.00 0.00
51.34
Average: 4 15.34 0.00 38.34 0.00 0.00
46.32
Average: 5 0.05 0.00 99.78 0.00 0.00
0.17
Average: 6 14.95 0.00 41.34 0.00 0.00
43.71
Average: 7 15.80 0.00 39.70 0.00 0.00
44.50
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1498506/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
