On 01.09.2016 13:02, Jesper Dangaard Brouer wrote: > On Wed, 31 Aug 2016 23:51:16 +0200 > Jesper Dangaard Brouer <jbro...@redhat.com> wrote: > >> On Wed, 31 Aug 2016 13:42:30 -0700 >> Eric Dumazet <eric.duma...@gmail.com> wrote: >> >>> On Wed, 2016-08-31 at 21:40 +0200, Jesper Dangaard Brouer wrote: >>> >>>> I can confirm the improvement of approx 900Kpps (no wonder people have >>>> been complaining about DoS against UDP/DNS servers). >>>> >>>> BUT during my extensive testing, of this patch, I also think that we >>>> have not gotten to the bottom of this. I was expecting to see a higher >>>> (collective) PPS number as I add more UDP servers, but I don't. >>>> >>>> Running many UDP netperf's with command: >>>> super_netperf 4 -H 198.18.50.3 -l 120 -t UDP_STREAM -T 0,0 -- -m 1472 -n >>>> -N >>> >>> Are you sure sender can send fast enough ? >> >> Yes, as I can see drops (overrun UDP limit UdpRcvbufErrors). Switching >> to pktgen and udp_sink to be sure. >> >>>> >>>> With 'top' I can see ksoftirq are still getting a higher %CPU time: >>>> >>>> PID %CPU TIME+ COMMAND >>>> 3 36.5 2:28.98 ksoftirqd/0 >>>> 10724 9.6 0:01.05 netserver >>>> 10722 9.3 0:01.05 netserver >>>> 10723 9.3 0:01.05 netserver >>>> 10725 9.3 0:01.05 netserver >>> >>> Looks much better on my machine, with "udprcv -n 4" (using 4 threads, >>> and 4 sockets using SO_REUSEPORT) >>> >>> 10755 root 20 0 34948 4 0 S 79.7 0.0 0:33.66 udprcv >>> 3 root 20 0 0 0 0 R 19.9 0.0 0:25.49 >>> ksoftirqd/0 >>> >>> Pressing 'H' in top gives : >>> >>> 3 root 20 0 0 0 0 R 19.9 0.0 0:47.84 >>> ksoftirqd/0 >>> 10756 root 20 0 34948 4 0 R 19.9 0.0 0:30.76 udprcv >>> 10757 root 20 0 34948 4 0 R 19.9 0.0 0:30.76 udprcv >>> 10758 root 20 0 34948 4 0 S 19.9 0.0 0:30.76 udprcv >>> 10759 root 20 0 34948 4 0 S 19.9 0.0 0:30.76 udprcv >> >> Yes, I'm seeing the same when unning 5 instances my own udp_sink[1]: >> sudo taskset -c 0 ./udp_sink --port 10003 --recvmsg --reuse-port --count >> $((10**10)) >> >> PID S %CPU TIME+ COMMAND >> 3 R 21.6 2:21.33 ksoftirqd/0 >> 3838 R 15.9 0:02.18 udp_sink >> 3856 R 15.6 0:02.16 udp_sink >> 3862 R 15.6 0:02.16 udp_sink >> 3844 R 15.3 0:02.15 udp_sink >> 3850 S 15.3 0:02.15 udp_sink >> >> This is the expected result, that adding more userspace receivers >> scales up. I needed 5 udp_sink's before I don't see any drops, either >> this says the job performed by ksoftirqd is 5 times faster or the >> collective queue size of the programs was fast enough to absorb the >> scheduling jitter. > > I need some help from scheduler people explaining this! > > In above run of udp_sink (which had expected behavior), I ran udp_sink > in 5 different xterm/shells. Below, I'm running all 5 udp_sink > programs from the same bash shell (just backgrounding them). > > PID S %CPU TIME+ COMMAND > 3 R 50.0 29:02.23 ksoftirqd/0 > 10881 R 10.7 1:01.61 udp_sink > 10837 R 10.0 1:05.20 udp_sink > 10852 S 10.0 1:01.78 udp_sink > 10862 R 10.0 1:05.19 udp_sink > 10844 S 9.7 1:01.91 udp_sink
Could you enable schedstats (sysctl schedstats) and show /proc/ksoftirq*/sched? Thanks, Hannes