Based on my high-level understanding of the code path and according to the UCX implementation of the flush, the required level of completion is local.
George. On Tue, Sep 28, 2021 at 19:26 Zhang, Wei via devel <devel@lists.open-mpi.org> wrote: > Dear All, > > > > I have a question regarding the completion semantics of btl_flush, > > > > In opal/mca/btl/btl.h, > > > > > https://github.com/open-mpi/ompi/blob/4828663537e952e3d7cbf8fbf5359f16fdcaaade/opal/mca/btl/btl.h#L1146 > > > > the comment about btl_flush says: > > > > * This function returns when all outstanding RDMA (put, get, atomic) > operations > > * that were started prior to the flush call have completed. > > > > However, it is not clear to me what “complete” actually means? E.g. does > it mean local completion (the action on RDMA initiator side has completed), > or does it mean “remote completion”, (the action of RDMA remote side has > completed). We are interested in this because for many RDMA btls, “local > completion” does not equal to “remote completion”. > > > > From the way btl_flush is used in osc/rdma’s fence operation (which is a > call to flush followed by a MPI_Barrier), we think that btl_flush should > mean remote completion, but want to get the clarification from the > community. > > > > Sincerely, > > > > Wei Zhang > > >
