Scott Carter wrote:
> On 10/07/08 21:12, Garrett D'Amore wrote:
>
>> The interfaces are marked Committed. I have some concerns with this,
>> as I read the project details. One of the areas that really concerns
>> me is the fact that drivers have to deal with a situation where the
>> total number of interrupts is reduced. (*Increasing* interrupts
>> seems free from the issues I am concerned with.)
>
> You're right, we have very limited experimentation with these interfaces.
> And I should not be casting the callback semantics into stone. So I will
> change the interfaces to Uncommitted.
Thank you.
>
>> I'm interested to know how this is used with real hardware.
>>
>> For example, one potential use of this facility is to support
>> multiple receive rings. However, now the problem is that there is a
>> window of time where an extra ring may be "orphaned". What happens
>> to those packets that are received there? (What happens to the
>> interrupt that the hardware issues, for that matter?)
>
> I've been working with the Neptune driver developers, to modify it to
> use these interfaces. So let's use that real hardware example.
>
>
[ neptune details snipped ]
>
>
>
> In general, drivers should already have an ability to quiesce their
> hardware. They should have this for other reasons, like OS quiesce
> during certain DR operations.
Agreed about quiesce. But there is a new semantic here, which is that
drivers no longer have any way to enforce the QoS guarantees that they
may have previously done.
For example, with Neptune, you can imagine that you had four interrupts,
two general purpose (one send, one receive), and two that were dedicated
to providing QoS guarantees for realtime data. These guarantees are
passed up the stack to the mac clients via crossbow.
If the neptune driver gets a callback that says it only has two
interrupts available now, it might (depending on hardware limitations)
have to collapse its rings and renege on the guarantees it has made to
other clients.
Unlike with other things, the driver probably relied upon its ability to
make the guarantees in the first place by learning what interrupts and
rings were available at attach time. Drivers today are designed such
that those resources are immutable -- once interrupts are available,
they don't disappear later.
The new behavior potentially breaks these semantics in significant ways
that may limit what drivers can do.
>
> And, drivers that can utilize more interrupt vectors must already
> have logic to adapt themselves to arbitrary numbers of vectors.
> As is done by that nxge_ldgv_init() function in Neptune.
>
> The two bits of logic put together and the driver should be able to
> quiesce, re-calculate, and unquiesce as necessary during a callback.
> Things should not get orphaned. And there is no need for a driver
> to try and create a policy to lock in a specific number of interrupt
> vectors. To get and use more interrupt vectors, they must have the
> flexibility to dynamically adapt in either direction. Either to
> scale up and use more vectors, or scale down and use less.
>
>> How will this feature interact with Crossbow resource allocation? Do
>> we have any examples of the two working together yet?
>
> Our team has met with the Crossbow team. Their basic concern was that
> we must not regress the performance of certain NIC drivers. The drivers
> they rely upon have existing workaround mechanisms to get more interrupt
> vectors. My project preserves those workarounds, so they can still get
> more interrupts before they are converted to use my new interfaces.
>
My concerns go beyond just preserving the existing hack. I want to make
sure that they can implement drivers to the new framework and not
regress either. (I.e. it shouldn't be required to use a hack to get the
performance.)
See my description above of Crossbow resource guarantees to see what I'm
talking about. I think that there needs to be much closer interaction
between the resource allocation done by Crossbow (and there may be other
similar projects outside of the networking space!) and what this project
proposes.
-- Garrett
