A new global structure (ompi_request_functions) holding all the pointers to the wait/test functions have been added. ompi_request_wait* and ompi_request_test* have been #defined to be replaced by ompi_request_functions.req_wait. The default implementations of the wait/test functions names have been changed from ompi_request_% to ompi_request_default_%. Those functions are static initializer of the ompi_request_functions structure.
To modify the defaults, a components 1) copy the ompi_request_functions structure (the type ompi_request_fns_t can be used to declare a suitable variable), 2) change some of the functions according to its needs. This is best done at MPI_init time when there is no threads. Should this component be unloaded it have to restore the defaults. The ompi_request_default_* functions should never be called directly anywhere in the code. If a component needs to access the previously defined implementation of wait, it should call its local copy of the function. Component implementors should keep in mind that another component might have already changed the defaults and needs to be called.
Performance impact on NetPipe -a (async recv mode) does not show measurable overhead. Here follows the "diff -y" between original and modified ompi assembly code from ompi/mpi/c/wait.c. The only significant difference is an extra movl to load the address of the ompi_request_functions structure in eax. This obviously explains why there is no measurable cost on latency.
ORIGINAL MODIFIED
L2:
L2:
movl L_ompi_request_null$non_lazy_ptr-"L00000000001$pb"(%ebx),
%eax movl L_ompi_request_null
$non_lazy_ptr-"L00000000001$pb"(%ebx), %eax
cmpl %eax, (%edi)
cmpl %eax, (%edi)
je L18
je L18
> movl L_ompi_request_functions
$non_lazy_ptr-"L00000000001$pb"(%ebx), %eax
movl %esi, 4(%esp)
movl %esi, 4(%esp)
movl %edi, (%esp)
movl %edi, (%esp)
call L_ompi_request_wait$stub
| call *16(%eax)
Here is the patch for those who want to try themselves.
custom_request_wait_and_test.patch
Description: Binary data
If I receive comments outlining the need, thread safe accessors could be added to allow components to change the functions at anytime during execution and not only during MPI_Init/Finalize. Please make noise if you find this useful. If comments does not suggest extra work, I expect this code to be committed in trunk next week.
Aurelien Le 8 oct. 07 à 06:01, Aurelien Bouteiller a écrit :
For message logging purpose, we need to interface with wait_any, wait_some, test, test_any, test_some, test_all. It is not possible to use PMPI for this purpose. During the face-to-face meeting in Paris (5-12 october 2007) we discussed this issue and came to the conclusion that the best way to achieve this is to replace direct calls to ompi_request_wait* and test* by indirect calls (same way as PML send, recv, etc). Basic idea is to declare a static structure containing the 8 pointers to all the functions. This structure is initialized at compilation time with the current basic wait/test functions. Before end of MPI_init, any component might replace the basics with specialized functions. Expected cost is less than .01us latency according to preliminary test. The method is consistent with the way we call pml send/recv. Mechanism could be used later for stripping out grequest from critical path when they are not used. -- Aurelien Bouteiller, PhD Innovative Computing Laboratory - MPI group +1 865 974 6321 1122 Volunteer Boulevard Claxton Education Building Suite 350 Knoxville, TN 37996 _______________________________________________ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
-- Dr. Aurelien Bouteiller, Sr. Research Associate Innovative Computing Laboratory - MPI group +1 865 974 6321 1122 Volunteer Boulevard Claxton Education Building Suite 350 Knoxville, TN 37996
