On Tue, 2015-02-10 at 13:43 -0800, trevor_davenp...@selinc.com wrote:
> I've recently encountered a problem after upgrading from 3.0.57-rt82 to
> 3.10.47-rt50 where process_backlog gets interrupted and does not resume
> for a while, which results in packets not being processed in time. I see
> net_rx_action, which then calls process_backlog (as the poll method to
> process the backlog of packets queued up the netif_rx) but then after the
> interruption, it does not finish for about 5ms. In the older kernel it
> would finish based on the priority of ksoftirqd. This is no longer the
> case.
>
> I have priorities configured so that hard interrupts are highest,
> ksoftirqd next (both are SCHED_FIFO) and then my program is currently
> SCHED_OTHER but I still do not see the rx softirq finish before my program
> runs.
>
> This is all on a single core powerpc device. I do not see these problems
> with a net device which uses NAPI directly (as such i'm updating my driver
> to use NAPI) but it seems like there is a real bug here somewhere. I have
> not been able to find any mention of similar problems (perhaps few people
> are using netif_rx these days).
>
> I've attached a recording from perf which shows the problem. Specifically,
> you see net_rx_action run at time 213.079014 and then it doesn't finish
> until about 5ms later at time 213.084953 which i not the case on the older
> kernels. It seems something has changed with softirq handling or
> process_backlog needed adapted for it. My suspicion is this has something
> to do with the work mentioned in 210dc110063cf040d3209fddf766f6fcafccdc34
> but I'm not an expert with this area of the kernel.
Your suspicion is correct. Your net traffic is being handled by your
SCHED_OTHER database task, which lost the CPU for a while due to it
being a SCHED_OTHER task. It's a behavior change from previous rt
kernels, but not a bad one. At the rt mini-summit of whatever year that
was, this change was shown to be a massive win. Low priority network
traffic now won't hinder a high priority task getting to the CPU, and
should a high priority task block due to your low priority task having
been preempted while holding the sirq lock it wants, PI will kick in.
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
