On Sun, 12 Aug 2007, Gleb Natapov wrote:
> > Any objections? We can discuss what approaches we want to take
> > (there's going to be some complications because of the PML driver,
> > etc.); perhaps in the Tuesday Mellanox teleconf...?
> >
> My main objection is that the only reason you propose to do this is some
> bogus benchmark? Is there any other reason to implement header caching?
> I also hope you don't propose to break layering and somehow cache PML headers
> in BTL.
Gleb is hitting the main points I wanted to bring up. We had
examined this header caching in the context of PSM a little while
ago. 0.5us is much more than we had observed -- at 3GHz, 0.5us would
be about 1500 cycles of code that has little amounts of branches.
For us, with a much bigger header and more fields to fetch from
different structures, it was more like 350 cycles which is on the
order of 0.1us and not worth the effort (in code complexity,
readability and frankly motivation for performance). Maybe there's
more to it than just "code caching" -- like sending from pre-pinned
headers or using the RDMA with immediate, etc. But I'd be suprised
to find out that openib btl doesn't do the best thing here.
I have pretty good evidence that for CM, the latency difference comes
from the receive-side (in particular opal_progress). Doesn't the
openib btl receive-side do something similiar with opal_progress,
i.e. register a callback function? It probably does something
different like check a few RDMA mailboxes (or per-peer landing pads)
but anything that gets called before or after it as part of
opal_progress is cause for slowdown.
. . christian
--
christian.b...@qlogic.com
(QLogic Host Solutions Group, formerly Pathscale)
