On Mon, Jul 23, 2007 at 11:10:08AM -0700, Sean Hefty wrote: > >The design I was thinking to suggest for IPoIB is to almost always use > >this API since this policy makes the implementation consistent with the > >decisions made by the network stack neighbour cache > > This defeats one of the benefit of caching, which is using a single > GetTable query, versus literally hundreds or thousands of Get queries. > Consider that constant all-to-all communication using IPoIB between 1024 > ports, with a 15 minute ARP table timeout would hit the SA with close to > 600 queries per second. > > I agree with Michael that it would be better for a ULP to invalidate > cache entries.
Well, in my view, this is exactly the sort of thing you should not do. ULPs have no better idea what is going on. ARP expiry doesn't give you any special information about the cached PR. If kernel caching is used it must be viewed as authorative and kept current through some kind of external mechanism. Something like re-doing the big GetTable query prior to starting a job is a fine interm way to do this. Ideally updating the kernel sa cache would also push updated data into the neighbor AH structures as appropriate. Then there is one single source of PR data, one source of IP -> GID mapppings, etc. These problems are just an unavoidable part of trying to use caching - build the mechanism to support coherent replication and just deal with these downsides <shrug>. Sean is basically doing non-coherent replication today with his big GetTable query and that sounds like what is speeding things up, not caching indivudal PRs. > overstating the likelihood of issues occurring in practice. Even > the SA Well, any time your renumber your network you will get burned and have to restart ipoib on every node with the way things are today. Something like a SM upgrade or changing to a new vendor SM, or increasing LMC could do this to you. > caches the path data -- getting a PR from the SA doesn't provide any > additional guarantees. Erm, any SA that returns a PR that is invalid in the network outside the time the network is being updated is seriously busted, IMHO. Jason _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
