> On June 1, 2015 at 11:48 AM Martin Willi <mar...@strongswan.org> wrote: > > > > > Even at these rates, the CPU did not appear to be very busy. We had one at > > 85% > > occupied but that was the one running nuttcp. > > On the outgoing path, the Linux kernel usually accounts ESP encryption > under the process that sends traffic using a socket send() call. So > these 85% probably include AES-GCM. > > On the receiving or forwarding path, you'll have to look at the software > interrupt usage (si in top). > > > We have seen these boxes pass almost 20 Gbps with single digit > > utilization so they have plenty of horsepower. > > That does not have to mean much. Its all about encryption, and that is > rather expensive. If you have specialized hardware, this most likely > means it is good at shuffling data over the network, but might be > underpowered when it has to do encryption in software. > > > We are also running haveged on them to prevent entropy starvation for the > > encryption. > > Only the key exchange needs entropy, raw AES-GCM does not. > > Regards > Martin > Hello, all. Still battling this problem. The system is a SuperMicro server and not a specialized device. It looks like the problem may be software interrupts but on the sending side so I am very curious about the recommendation to check the receiving side. In sending only 100 Mbps or so, I see a single CPU pegged at 100% for si. I wondered if it might be the number of ACK packets being returned from the other side. We have a large window size so we get a flood of ACK packets in reply send but that really doesn't seem to make sense. But it would explain why we see this more in TCP than UDP . . . . or so we thought. I then sent 200 Mbps of UDP traffic so virtually no reply traffic and the sender was still at 100% si. What might be generating such a huge number of software interrupts and how can we reduce them or spread them across multiple processors?
Here is an interesting screen scrape showing the high si utilization, the multiple kworker threads from pcrypt and that the ethernet drivers are utilizing multiple cores (at least I think that is what ethtool -L is showing). Thanks - John top - 22:19:45 up 2 days, 18:49, 1 user, load average: 0.88, 0.73, 0.47 Tasks: 161 total, 4 running, 157 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 1.8 sy, 0.0 ni, 36.6 id, 0.0 wa, 0.0 hi, 61.6 si, 0.0 st %Cpu1 : 0.0 us, 4.7 sy, 0.0 ni, 95.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 0.0 us, 1.0 sy, 0.0 ni, 98.7 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu8 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu9 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu10 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32987140 total, 1931224 used, 31055916 free, 141196 buffers KiB Swap: 7843824 total, 0 used, 7843824 free, 1110780 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21654 root 20 0 0 0 0 R 23 0.0 1:15.43 kworker/0:0 21695 root 20 0 0 0 0 S 19 0.0 0:01.68 kworker/0:2 20784 root 20 0 0 0 0 S 16 0.0 3:20.98 kworker/0:1 21668 root 20 0 0 0 0 R 3 0.0 0:52.09 kworker/0:3 21669 root 20 0 0 0 0 S 3 0.0 0:09.51 kworker/1:2 21560 root 20 0 0 0 0 S 2 0.0 0:22.76 kworker/1:1 6605 root 0 -20 18508 6508 484 S 2 0.0 50:19.27 conntrackd 21613 root 20 0 0 0 0 S 1 0.0 0:07.16 kworker/8:2 21616 root 20 0 0 0 0 S 1 0.0 0:02.07 kworker/2:1 21621 root 20 0 0 0 0 S 1 0.0 0:01.18 kworker/9:0 21646 root 20 0 0 0 0 S 1 0.0 0:05.64 kworker/5:2 21653 root 20 0 0 0 0 R 1 0.0 0:05.48 kworker/3:0 21465 root 20 0 0 0 0 S 1 0.0 0:05.54 kworker/10:0 21611 root 20 0 0 0 0 S 1 0.0 0:05.18 kworker/6:0 21617 root 20 0 0 0 0 S 1 0.0 0:06.43 kworker/4:0 21618 root 20 0 0 0 0 S 1 0.0 0:01.76 kworker/11:1 21634 root 20 0 0 0 0 S 1 0.0 0:03.82 kworker/7:0 362 root 20 0 0 0 0 S 0 0.0 0:06.10 jbd2/md0-8 4460 quagga 20 0 26296 1396 608 S 0 0.0 0:05.52 zebra 5114 root 20 0 24372 4224 2088 S 0 0.0 37:38.64 openvpn 21498 root 20 0 0 0 0 S 0 0.0 0:07.12 kworker/1:0 1 root 20 0 10652 820 680 S 0 0.0 0:04.25 init 2 root 20 0 0 0 0 S 0 0.0 0:00.06 kthreadd 3 root 20 0 0 0 0 S 0 0.0 6:56.21 ksoftirqd/0 6 root rt 0 0 0 0 S 0 0.0 0:05.71 migration/0 7 root rt 0 0 0 0 S 0 0.0 0:22.36 watchdog/0 8 root rt 0 0 0 0 S 0 0.0 0:00.45 migration/1 10 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/1 root@gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth^C root@gwhq-1:/etc/ipsec.d/remotenets# ip link ls | grep bond1 7: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000 10: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000 15: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 32: vrrp.1@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UNKNOWN mode DEFAULT group default root@gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth3 no channel parameters changed, aborting current values: tx 0 rx 0 other 1combined 8 root@gwhq-1:/etc/ipsec.d/remotenets# ethtool -L eth5 no channel parameters changed, aborting current values: tx 0 rx 0 other 1combined 8 _______________________________________________ Users mailing list Users@lists.strongswan.org https://lists.strongswan.org/mailman/listinfo/users