On Nov 23, 2011, at 18:28, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> I had a useful conversation with the MPICH guys today about the unstructured
> communication primitives that Mark, Matt, Barry, and I have been discussing.
> It's looking like we can do a very thin communication library that has the
> nice semantics I was looking for, namely that the local-to-global is stored
> only as indices in the local space. To implement the communication
> primitives, the owner of a global node never needs to know how many or which
> processes have that point in their local space. There is an MPI-3 feature
> that we can use to elegantly avoid two-way knowledge for pointwise
> gather/scatter. (We can always implement the same interface using vanilla
> MPI-1, but the MPI-2/3 implementation would have less overhead.)
>
> If I/we write this layer, VecScatter (with arbitrary data types, etc) could
> be implemented very easily on top of it. I'm in no rush to do this, but I
> could imagine getting there eventually. The VecScatter interface allows some
> things that might not be important. For example, any process can specify an
> edge in the communication graph, even if it does not own the source or the
> destination vertex. Also, both source and destination vertices can have
> degree higher than one. We can even put a link between the same points twice
> (effectively multiplying that contribution by 2 when using ADD_VALUES). I
> don't know if this ever makes semantic sense, but the interface allows it.
>
> I wonder if we can express all useful communication with a more restrictive
> interface similar to a local-to-global map (but mapping between any vectors)
> where one side (typically called "local", but doesn't need to actually be a
> local Vec) has at most one edge.
That would disallow accumulating a MAX "locally".
>
> What weird things do people use VecScatter for that might break with this
> more restrictive model?
