[OMPI users] Error When Using Open MPI SHMEM with UCX

2019-04-23 Thread Benjamin Brock via users
And, to provide more details, I'm using a fresh vanilla build of Open MPI 4.0.1 with UCX 1.5.1 (`./configure --with-ucx=$DIR/ucx-1.5.1`). Ben ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] Error When Using Open MPI SHMEM with UCX

2019-04-23 Thread Benjamin Brock via users
I get the following error when trying to run SHMEM programs using UCX. [xiii@shini dir]$ oshrun -n 1 ./target/debug/main [1556046469.890238] [shini:19769:0]sys.c:619 UCX ERROR shmget(size=2097152 flags=0xfb0) for mm_recv_desc failed: Operation not permitted, please check shared memor

[OMPI users] rmaps_base_oversubscribe Option in Open MPI 4.0

2019-01-25 Thread Benjamin Brock
I used to be able to (e.g. in Open MPI 3.1) put the line rmaps_base_oversubscribe = true in my `openmpi-mca-params.conf`, and this would enable oversubscription by default. In 4.0.0, it appears that this option doesn't work anymore, and I have to use `--oversubscribe`. Am I missing something, o

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-26 Thread Benjamin Brock
In case anyone comes across this thread in an attempt to get RDMA over Ethernet working on AWS, here's the conclusion I came to: There are two kinds of NICs exposed to VMs on AWS: - Intel 82599 VF - This NIC is old and does not support RoCE or iWARP. - It's a virtualized view of an actu

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-11 Thread Benjamin Brock
Thanks for your response. One question: why would RoCE still requiring host processing of every packet? I thought the point was that some nice server Ethernet NICs can handle RDMA requests directly? Or am I misunderstanding RoCE/how Open MPI's RoCE transport? Ben

[OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-06 Thread Benjamin Brock
I'm setting up a cluster on AWS, which will have a 10Gb/s or 25Gb/s Ethernet network. Should I expect to be able to get RoCE to work in Open MPI on AWS? More generally, what optimizations and performance tuning can I do to an Open MPI installation to get good performance on an Ethernet network?

Re: [OMPI users] Are MPI datatypes guaranteed to be compile-time constants?

2018-09-06 Thread Benjamin Brock
Thanks for the responses--from what you've said, it seems like MPI types are indeed not guaranteed to be compile-time constants. However, I worked with the people at IBM, and it seems like the difference in behavior was caused by the IBM compiler, not the Spectrum IBM implementation. Ben

[OMPI users] Are MPI datatypes guaranteed to be compile-time constants?

2018-09-04 Thread Benjamin Brock
Are MPI datatypes like MPI_INT and MPI_CHAR guaranteed to be compile-time constants? Is this defined by the MPI standard, or in the Open MPI implementation? I've written some template code where MPI datatypes are constexpr members, which requires that they be known at compile time. This works in

Re: [OMPI users] Using OpenSHMEM with Shared Memory

2018-02-07 Thread Benjamin Brock
Here's what I get with those environment variables: https://hastebin.com/ibimipuden.sql I'm running Arch Linux (but with OpenMPI/UCX installed from source as described in my earlier message). Ben ___ users mailing list users@lists.open-mpi.org https://

[OMPI users] Using OpenSHMEM with Shared Memory

2018-02-06 Thread Benjamin Brock
How can I run an OpenSHMEM program just using shared memory? I'd like to use OpenMPI to run SHMEM programs locally on my laptop. I understand that the old SHMEM component (Yoda?) was taken out, and that UCX is now required. I have a build of OpenMPI with UCX as per the directions on this random

[OMPI users] Oversubscribing When Running Locally

2018-01-24 Thread Benjamin Brock
Recently, when I try to run something locally with OpenMPI with more than two ranks (I have a dual-core machine), I get the friendly message -- There are not enough slots available in the system to satisfy the 3 slots that wer

[OMPI users] Oversubscribing

2018-01-24 Thread Benjamin Brock
Recently, when I try to run something locally with OpenMPI with more than two ranks (I have a dual-core machine), I get the friendly message -- There are not enough slots available in the system to satisfy the 3 slots that wer

Re: [OMPI users] OMPI users] Compiling Open MPI for Cross-Compilation

2017-12-17 Thread Benjamin Brock
Yeah, I just noticed that Open MPI was giving me all x86_64 binaries with the configuration flags ./configure --host=riscv64-unknown-linux --enable-static --disable-shared --disable-dlopen --enable-mca-no-build=patcher-overwrite --prefix=/home/ubuntu/src/ben-build/openmpi and was very confused.

Re: [OMPI users] Compiling Open MPI for Cross-Compilation

2017-12-16 Thread Benjamin Brock
I have the same error with ./configure --host=riscv64-unknown-linux --build=x86_64-linux-gnu --enable-static --disable-shared --prefix=/home/ubuntu/src/ben-build/openmpi Ben On Sat, Dec 16, 2017 at 4:50 PM, Benjamin Brock wrote: > > try removing the --target option. > > With t

Re: [OMPI users] Compiling Open MPI for Cross-Compilation

2017-12-16 Thread Benjamin Brock
> try removing the --target option. With the configure line ./configure --host=riscv64-unknown-linux --enable-static --disable-shared --prefix=/home/ubuntu/src/ben-build/openmpi It successfully configures, but I now get the error /home/xiii/Downloads/openmpi-3.0.0/opal/.libs/libopen-pal.a(patch

[OMPI users] Compiling Open MPI for Cross-Compilation

2017-12-15 Thread Benjamin Brock
I'd like to run Open MPI on a cluster of RISC-V machines. These machines are pretty weak cores and so I need to cross-compile. I'd like to do this: Machine 1, which is x86_64-linux-gnu, compiles programs for machine 2. Machine 2, which is riscv64-unknown-linux, will run these programs. It seem

Re: [OMPI users] [EXTERNAL] Re: Using shmem_int_fadd() in OpenMPI\'s SHMEM

2017-11-21 Thread Benjamin Brock
> What version of Open MPI are you trying to use? Open MPI 2.1.1-2 as distributed by Arch Linux. > Also, could you describe something about your system. This is all in shared memory on a MacBook Pro; no networking involved. The seg fault with the code example above looks like this: [xiii@shini

[OMPI users] Using shmem_int_fadd() in OpenMPI's SHMEM

2017-11-20 Thread Benjamin Brock
What's the proper way to use shmem_int_fadd() in OpenMPI's SHMEM? A minimal example seems to seg fault: #include #include #include int main(int argc, char **argv) { shmem_init(); const size_t shared_segment_size = 1024; void *shared_segment = shmem_malloc(shared_segment_size); int *

Re: [OMPI users] MPI_Accumulate() Blocking?

2017-05-04 Thread Benjamin Brock
the operation. > > That said, I think to recall that Open-MPI 1.x did not support > asynchronous target-side progress for passive-target synchronization > (which is used in your benchmark example), so the behavior you > observed is to some extent expected. > > Cheers, > Marc-And

[OMPI users] MPI_Accumulate() Blocking?

2017-05-03 Thread Benjamin Brock
MPI_Accumulate() is meant to be non-blocking, and MPI will block until completion when an MPI_Win_flush() is called, correct? In this (https://hastebin.com/raw/iwakacadey) microbenchmark, MPI_Accumulate() seems to be blocking for me in OpenMPI 1.10.6. I'm seeing timings like [brock@nid00622 junk

[OMPI users] How to Free Memory Allocated with MPI_Win_allocate()?

2017-04-24 Thread Benjamin Brock
How are we meant to free memory allocated with MPI_Win_allocate()? The following crashes for me with OpenMPI 1.10.6: #include #include #include int main(int argc, char **argv) { MPI_Init(&argc, &argv); int n = 1000; int *a; MPI_Win win; MPI_Win_allocate(n*sizeof(int), sizeof(int),