I apologize for the delayed reply. My reservation on these machines
expired (they're currently being used for driver certification for a
different OS). I had wanted to have the machines in hand again before
discussing further, but I need to reply before this thread gets
too stale...
Nicolas Droux wrote:
Andrew Gallatin wrote:
Nicolas Droux wrote:
Drew,
Can you remind us how many RX groups and RX rings per group, as well
as TX rings are you exposing out of the box? Creating a VNIC
currently disables polling on the primary client (this is something
we are now working on to address as part of dynamic grouping), so
this issue could be related to polling, and I would suggest looking
in that area first.
One group, 8 TX and 8 RX rings. On this machine, only 2 MSI-X
interrupts were available, so that limited us to 2 TX and 2 RX
rings. I also tried exposing only a single TX / RX ring.
How does your driver currently handles the sharing of 2 MSI-X between 2
TX and 2 RX rings?
The NIC handles sharing of MSI-X interrupts among TX/RX
queues via a logical separation of the NIC into "slices"
A slice consists of a TX queue, an RX queue, and an interrupt
routine. The more MSI-X interrupt vectors the driver can get,
the more slices (eg, TX/RX queues) it can support. The driver
was recently integrated, and you can see the code for yourself
(uts/common/io/myri10ge/drv/).
Can you elaborate on looking at polling? What kind of information
would be helpful?
Since the bulk of the regression is going away once polling is disabled,
I was simply suggesting to look at that area of the code. IIRC early
versions of the driver polling code introduced reordering, and the
sharing of interrupts is another thing that could impact performance if
disabling interrupts on one ring causes traffic on other rings to stall.
No traffic is stalling. I don't remember seeing out of order
packets in netstat (I wish I had the machines so I could double check).
I do remember seeing excessive context switches, which seemed to
quiet down after enabling a vnic.
After some googling, I think I'm probably seeing the same sort of
problems as described in this thread from crossbow-discuss:
http://mail.opensolaris.org/pipermail/crossbow-discuss/2009-April/001731.html
Once I get the machines back, I'll post some numbers.
Drew
_______________________________________________
networking-discuss mailing list
[email protected]
