Re: Why do soft interrupt coelescing?

[Kenneth D. Merry]

On Tue, Oct 09, 2001 at 12:28:02 -0700, Terry Lambert wrote:
> "Kenneth D. Merry" wrote:
> [ ... soft interrupt coelescing ... ]
> 
> > As you say above, this is actually a good thing.  I don't see how this ties
> > into the patch to introduce some sort of interrupt coalescing into the
> > ti(4) driver.   IMO, you should be able to tweak the coalescing parameters
> > on the board to do what you want.
> 
> I have tweaked all tunables on the board, and I have not gotten
> anywhere near the increased performance.
> 
> The limit on how far you can push this is based on how much
> RAM you can have on the card, and the limits to coelescing.
> 
> Here's the reason: when you receive packets to the board, they
> get DMA'ed into the ring.  No matter how large the ring, it
> won't matter, if the ring is not being emptied asynchronously
> relative to it being filled.
> 
> In the case of a full-on receiver livelock situation, the ring
> contents will be continuously overwritten.  This is actually
> what happens when you put a ti card into a machine with a
> slower processor, and hit it hard.

eh?  The card won't write past the point that has been acked by the kernel.
If the kernel hasn't acked the packets and one of the receive rings fills
up, the card will hold off on sending packets up to the kernel.

> In the case of interrupt processing, where you jam the data up
> through ether input at interrupt time, the buffer will be able
> to potentially overrun, as well.  Admittedly, you can spend a
> huge percentage of your CPU time in interrupt processing, and
> if your CPU is fast enough, unload the queue very quickly.
> 
> But if you then look at doing this for multiple gigabit cards
> at the same time, you quickly reach the limits... and you
> spend so much of your time in interrupt processing, that you
> spend none running NETISR.

I agree that you can end up spending large portions of your time doing
interrupt processing, but I haven't seen instances of "buffer overrun", at
least not in the kernel.  The case where you'll see a "buffer overrun", at
least with the ti(4) driver, is when you have a sender that's faster than
the receiver.

So the receiver can't process the data in time and the card just drops
packets.

That's a different situation from the card spamming the receive ring over
and over again, which is what you're describing.  I've never seen that
happen, and if it does actually happen, I'd be interested in seeing
evidence.

> So you have moved your livelock up one layer.
> 
> 
> In any case, doing the coelescing on the board delays the
> packet processing until that number of packets has been
> received, or a timer expires.  The timer latency must be
> increased proportionally to the maximum number of packets
> that you coelesce into a single interrupt.
> 
> In other words, you do not interleave your I/O when you
> do this, and the bursty conditions that result in your
> coelescing window ending up full or close to full are
> the conditions under which you should be attempting the
> maximum concurrency you can possibly attain.
> 
> Basically, in any case where the load is high enough to
> trigger the hardware coelescing, the ring would need to
> be the next power of two larger to ensure that the end
> does not overwrite the beginning of the ring.
> 
> In practice, the firmware on the card does not support
> this, so what you do instead is push a couple of packets
> that may have been corrupted through DMA occurring during
> the fact -- in other words, you drop packets.
> 
> This is arguably "correct", in that it permits you to shed
> load, _but_ the DMAs still occur into your rings; it would
> be much better if the load were shed by the card firmware,
> based on some knowledge of ring depth instead (RED Queueing),
> since this would leave the bus clear for other traffinc (e.g.
> communication with main memory to provide network content for
> the cards for, e.g., and NFS server, etc.).
> 
> Without hacking firmware, the best you can do is to ensure
> that you process as much of all the traffic as you possibly
> can, and that means avoiding livelock.

Uhh, the Tigon firmware *does* drop packets when there is no more room in
the proper receive ring on the host side.  It doesn't spam things.

What gives you that idea?  You've really got some strange ideas about what
goes on with that board.  Why would someone design firmware so obviously
broken?

> [ ... LRP ... ]
> 
> > That sounds cool, but I still don't see how this ties into the patch you
> > sent out.
> 
> OK.  LRP removes NETISR entirely.
> 
> This is the approach Van Jacobson stated he used in his
> mythical TCP/IP stack, which we may never see.
> 
> What this does is push the stack processing down to the
> interrupt time for the hardware interrupt.  This is a
> good idea, in that it avoids the livelock for the NETISR
> never running because you are too busy taking hardware
> interrupts to be able to do any stack processing.
> 
> The way this ties into t

Re: Why do soft interrupt coelescing?

[Terry Lambert]

"Kenneth D. Merry" wrote:
[ ... soft interrupt coelescing ... ]

> As you say above, this is actually a good thing.  I don't see how this ties
> into the patch to introduce some sort of interrupt coalescing into the
> ti(4) driver.   IMO, you should be able to tweak the coalescing parameters
> on the board to do what you want.

I have tweaked all tunables on the board, and I have not gotten
anywhere near the increased performance.

The limit on how far you can push this is based on how much
RAM you can have on the card, and the limits to coelescing.

Here's the reason: when you receive packets to the board, they
get DMA'ed into the ring.  No matter how large the ring, it
won't matter, if the ring is not being emptied asynchronously
relative to it being filled.

In the case of a full-on receiver livelock situation, the ring
contents will be continuously overwritten.  This is actually
what happens when you put a ti card into a machine with a
slower processor, and hit it hard.

In the case of interrupt processing, where you jam the data up
through ether input at interrupt time, the buffer will be able
to potentially overrun, as well.  Admittedly, you can spend a
huge percentage of your CPU time in interrupt processing, and
if your CPU is fast enough, unload the queue very quickly.

But if you then look at doing this for multiple gigabit cards
at the same time, you quickly reach the limits... and you
spend so much of your time in interrupt processing, that you
spend none running NETISR.

So you have moved your livelock up one layer.


In any case, doing the coelescing on the board delays the
packet processing until that number of packets has been
received, or a timer expires.  The timer latency must be
increased proportionally to the maximum number of packets
that you coelesce into a single interrupt.

In other words, you do not interleave your I/O when you
do this, and the bursty conditions that result in your
coelescing window ending up full or close to full are
the conditions under which you should be attempting the
maximum concurrency you can possibly attain.

Basically, in any case where the load is high enough to
trigger the hardware coelescing, the ring would need to
be the next power of two larger to ensure that the end
does not overwrite the beginning of the ring.

In practice, the firmware on the card does not support
this, so what you do instead is push a couple of packets
that may have been corrupted through DMA occurring during
the fact -- in other words, you drop packets.

This is arguably "correct", in that it permits you to shed
load, _but_ the DMAs still occur into your rings; it would
be much better if the load were shed by the card firmware,
based on some knowledge of ring depth instead (RED Queueing),
since this would leave the bus clear for other traffinc (e.g.
communication with main memory to provide network content for
the cards for, e.g., and NFS server, etc.).

Without hacking firmware, the best you can do is to ensure
that you process as much of all the traffic as you possibly
can, and that means avoiding livelock.


[ ... LRP ... ]

> That sounds cool, but I still don't see how this ties into the patch you
> sent out.

OK.  LRP removes NETISR entirely.

This is the approach Van Jacobson stated he used in his
mythical TCP/IP stack, which we may never see.

What this does is push the stack processing down to the
interrupt time for the hardware interrupt.  This is a
good idea, in that it avoids the livelock for the NETISR
never running because you are too busy taking hardware
interrupts to be able to do any stack processing.

The way this ties into the patch is that doing the stack
processing at interrupt time increases the per ether
input processing cycle overhead up.

What this means is that you get more benefit in the soft
interrupt coelescing than you otherwise would get, when
you are doing LRP.

But, you do get *some* benefit from doing it anyway, even
if your ether input processing is light: so long as it is
non-zero, you get benefit.

Note that LRP itself is not a panacea for livelock, since
it just moves the scheduling problem from the IRQ<->NETISR
scheduling into the NETISR<->process scheduling.  You end
up needing to implement weighted fair share or other code
to ensure that the user space process is permitted to run,
so you end up monitoring queue depth or something else,
and deciding not to reenable interrupts on the card until
you hit a low water mark, indicating processing has taken
place (see the papers by Druschel et. al. and Floyd et. al.).


> > > It isn't terribly clear what you're doing in the patch, since it isn't a
> > > context diff.
> >
> > It's a "cvs diff" output.  You could always check out a sys
> > tree, apply it, and then cvs diff -c (or -u or whatever your
> > favorite option is) to get a diff more to your tastes.
> 
> As Peter Wemm pointed out, we can't use non-context diffs safely without
> the exact time, date and branch of the source files.  This int

Re: uucp user shell and home directory

[Lyndon Nerenberg]

> None of those things are realproblems.  I've set up the port to be
> hosted on MASTER_SITE_LOCAL for now, but Lyndon's free to go and host
> it wherever he likes, organise whatever community support he likes (if
> theres nontrivial interest he could surely even get a freebsd.org
> mailing list set up!) and the UUCP community in FreeBSD can decide the
> future direction of that port.

I said I would maintain it if the code remained in the base system.
If UUCP is going to ports, I have a different code base (Rick Adams'
4.4 implementation) that I'm going to use (as I also mentioned to you).

BTW, could someone close out PR gnu/27715? It's not applicable now
that UUCP is unbundled.

Also, have you talked to Greg Shapiro about the disposition of
/bin/rmail? With UUCP gone, rmail should also come out of the base
system. I'm not sure if it should be built as part of the freebsd-uucp
port, or become a port unto itself.

--lyndon

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: kldxref broken, maybe?

[Bruce A. Mah]

If memory serves me right, Ruslan Ermilov wrote:
> On Thu, Sep 20, 2001 at 10:19:22PM -0700, Peter Wemm wrote:
> > Warner Losh wrote:
> > > In message  Mark Peek writes:
> > > : Install a -current kernel on a 4.X or pre-kldxref (before 9/10/01) 
> > > : 5.X system. I sent a note to Warner mentioning he might want to put a 
> > > : comment about this in UPDATING.
> > > 
> > > I'm just unsure how to describe it...
> > 
> > It is actually non-fatal.  It should probably be added to the list
> > of tools to build for making the kernel. 
> > 
> This is not enough -- it should be made a cross-tool, much like
> the gensetdefs in -STABLE is.  The binary format produced is MD.
> If we don't, we should disable it (-DNO_XREF) for cross-builds.

Was there ever any resolution to this issue?  I just tripped over it
trying to update a 4-STABLE box to -CURRENT.  (The solution I used, 
which was to manually install kldxref and the version of libc.so that 
it was linked against, probably isn't general-purpose.)

Thanks,

Bruce.





msg32413/pgp0.pgp
Description: PGP signature

Re: options NO_KLD

[Crist J. Clark]

On Mon, Oct 08, 2001 at 11:48:37AM -0700, Holtor wrote:
> Will this NO_KLD option be commited to
> -current and then hopefully -stable?
> 
> I have been checking the LINT file each morning
> after the nightly cvsup runs hoping to find this
> option in there but so far havent seen it in
> sight.
> 
> Any ideas?

I got four, count 'em, four, emails from people who thought it was the
neatest thang since sliced bread. I was surprised there were no
flames, but none of those. (Well, one came close.)

As I said, I was never planning to commit it. The illusion of security
is more dangerous than knowing the problem is there. The patch makes
it a little harder to get code into a running kernel, but does not
come close to stopping it. As lame as securelevel(8) is, you are much
better off figuring out how to raise it and still retain whatever
functionality you need.

This is what I've already said on -security,

  
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=297347+0+archive/2001/freebsd-security/20011007.freebsd-security

And the original patches,

  
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=172366+0+archive/2001/freebsd-security/20011007.freebsd-security

But hey, if people want it, I CAN JUST WRITE THE WARNINGS IN ALL CAPS
IN THE NOTES FILE and try not to be disappointed when they still don't
read it.
-- 
Crist J. Clark   [EMAIL PROTECTED]
 [EMAIL PROTECTED]
 [EMAIL PROTECTED]

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message