If you don't care about implementation details of NIC drivers, skip this
message. :-)
I've been thinking about the idea of doing "loanup", which a lot of NIC
drivers seem to do now days, but I wonder if there are latent problems
waiting with this approach. I'd like to hear from folks who believe
these concerns are not valid. (Loanup is the notion of "loaning"
receive packets up to the stack, with their DMA handles still bound, to
avoid tearing down the DMA handles later... the idea is that when the
message is freed, it can be reused without having to reestablish dma
bindings... which means you don't have to reprogram the addresses in the
IOMMU...
* when binding DMA, one specifies a few attributes, including
whether the access is read-only, write-only, read-write, and what kinds
of cache consistency is required ... (DDI_DMA_STREAMING vs
DDI_DMA_CONSISTENT)
* sometimes the stack may turn a packet around and send it back down
to another interface, without actually freeing it it until after it has
been through the transmit.
* given the above two points, the original DMA attributes may not be
appropriate for the packet being sent down!
* what happens when the sending NIC allocates a second DMA handle,
with different properties? Is it always safe to rebind a handle to the
same physical address?
Right now everything _seems_ to work. But I wonder how much of this is
just an accident of implementation, and how much of it is by design. It
seems like the whole idea of loaning up a packet with a DMA handle still
bound to it is a bad idea.
But, maybe we can guarantee that all current and future implementations
will always either permit multiple bindings to the same physical memory
addresses (perhaps with different DMA attributes requested!). And maybe
we also tacitly acknowledge that all memory on the system/IOMMU mappings
on the system are equivalent?
I guess there are two cases:
* systems with an IOMMU -- multiple mappings to the same address?
(or reference counting of a single mapping, with the understanding that
all mappings have the same attributes regardless of what was requested?
* systems without an IOMMU -- in this case the mapping is just a
simple translation from a virtual address to a physical address, maybe
even done but simple arithmetic?
There's also the whole problem of reference counting and DR.... a loaned
up packet has to be freed using a special function as part of
esballoc/desballoc, but that means you might have to reject a DDI detach
in the face of outstanding loaned up packets still held elsewhere in the
stack. (You need the dip to be valid to do ddi_dma_unbind... you also
need your driver to be kept from unloading from kernel memory so that
the function pointer associated with the free routine is still valid.)
All this is being done to reduce/eliminate the cost of
ddi_dma_addr_bind_handle/ddi_dma_unbind_handle(). I'm starting to think
that the cost of those two routines is sufficiently low on modern
systems to not be worth the risk and complication that loan up offers.
The "new" DDI DMA routines -- ddi_dma_addr_bind_handle and friends --
are apparently nearly as fast as the undocumented dvma routines (unlike
the "old" DMA routines -- ddi_dma_addr_setup and firends, which were
horribly inefficient.)
Anyway, I'd really like to hear thoughts on these issues... its likely
I'm missing one or more salient points, and if so, I'd like to
understand what I'm missing.
-- Garrett
_______________________________________________
networking-discuss mailing list
[email protected]
