Hello,
I'm having problems running a simple cuda-aware mpi application; the one found
at
https://github.com/parallel-forall/code-samples/tree/master/posts/cuda-aware-mpi-example
I have modified symbol ENV_LOCAL_RANK into OMPI_COMM_WORLD_LOCAL_RANK
My cluster has 2 K20m GPUs per node, with QLogic IB stack.
The normal CUDA/MPI application works fine;
but the cuda-ware mpi app is crashing when using 2 MPI proc over the 2 GPUs of
the same node:
the error message is:
Assertion failure at ptl.c:200: nbytes == msglen
I can send the complete backtrace from cuda-gdb if needed.
The same app when running on 2 GPUs on 2 different nodes give another error:
jacobi_cuda_aware_mpi:28280 terminated with signal 11 at PC=2aae9d7c9f78
SP=7fffc06c21f8. Backtrace:
/gpfslocal/pub/local/lib64/libinfinipath.so.4(+0x8f78)[0x2aae9d7c9f78]
Can someone give me hints where to look to track this problem ?
Thank you.
Pierre Kestener.
