On Tue, 28 Mar 2017 16:34:36 +0200
Frederic Weisbecker <[email protected]> wrote:
> On Tue, Mar 28, 2017 at 10:14:03AM +0200, Jesper Dangaard Brouer wrote:
> >
> > (While evaluating some changes to the page allocator) I ran into an
> > issue with ksoftirqd getting too much CPU sched time.
> >
> > I bisected the problem to
> > a499a5a14dbd ("sched/cputime: Increment kcpustat directly on irqtime
> > account")
> >
> > a499a5a14dbd1d0315a96fc62a8798059325e9e6 is the first bad commit
> > commit a499a5a14dbd1d0315a96fc62a8798059325e9e6
> > Author: Frederic Weisbecker <[email protected]>
> > Date: Tue Jan 31 04:09:32 2017 +0100
> >
> > sched/cputime: Increment kcpustat directly on irqtime account
> >
> > The irqtime is accounted is nsecs and stored in
> > cpu_irq_time.hardirq_time and cpu_irq_time.softirq_time. Once the
> > accumulated amount reaches a new jiffy, this one gets accounted to the
> > kcpustat.
> >
> > This was necessary when kcpustat was stored in cputime_t, which could at
> > worst have jiffies granularity. But now kcpustat is stored in nsecs
> > so this whole discretization game with temporary irqtime storage has
> > become unnecessary.
> >
> > We can now directly account the irqtime to the kcpustat.
> >
> > Signed-off-by: Frederic Weisbecker <[email protected]>
> > Cc: Benjamin Herrenschmidt <[email protected]>
> > Cc: Fenghua Yu <[email protected]>
> > Cc: Heiko Carstens <[email protected]>
> > Cc: Linus Torvalds <[email protected]>
> > Cc: Martin Schwidefsky <[email protected]>
> > Cc: Michael Ellerman <[email protected]>
> > Cc: Paul Mackerras <[email protected]>
> > Cc: Peter Zijlstra <[email protected]>
> > Cc: Rik van Riel <[email protected]>
> > Cc: Stanislaw Gruszka <[email protected]>
> > Cc: Thomas Gleixner <[email protected]>
> > Cc: Tony Luck <[email protected]>
> > Cc: Wanpeng Li <[email protected]>
> > Link:
> > http://lkml.kernel.org/r/[email protected]
> > Signed-off-by: Ingo Molnar <[email protected]>
> >
> > The reproducer is running a userspace udp_sink[1] program, and taskset
> > pinning the process to the same CPU as softirq RX is running on, and
> > starting a UDP flood with pktgen (tool part of kernel tree:
> > samples/pktgen/pktgen_sample03_burst_single_flow.sh).
>
> So that means I need to run udp_sink on the same CPU than pktgen?
No, you misunderstood. I run pktgen on another physical machine, which
is sending UDP packets towards my Device-Under-Test (DUT) target. The
DUT-target is receiving packets and I observe which CPU the NIC is
delivering these packets to.
E.g determine RX-CPU via mpstat command:
mpstat -P ALL -u -I SCPU -I SUM 2
I then start udp_sink, pinned to the RX-CPU, like:
sudo taskset -c 2 ./udp_sink --port 9 --count $((10**6)) --recvmsg --repeat
1000
> > [1] udp_sink
> > https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c
> >
> > The expected results (after commit 4cd13c21b207 ("softirq: Let
> > ksoftirqd do its job")) is that the scheduler split the CPU time 50/50
> > between udp_sink and ksoftirqd.
>
> I guess you mean that this is what happened before this commit?
Yes. (I just pointed out the kernel had another softirq bug, that I was
involved in fixing)
> >
> > After this commit, the udp_sink program does not get any sched CPU
> > time, and no packets are delivered to userspace. (All packets are
> > dropped by softirq due to a full socket queue, nstat
> > UdpRcvbufErrors).
> >
> > A related symptom is that ksoftirqd no longer get accounted in
> > top.
>
> That's indeed what I observe. udp_sink has almost no CPU time,
> neither has ksoftirqd but kpktgend_0 has everything.
>
> Finally a bug I can reproduce!
Good to hear you can reproduce it! :-)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
