Re: [OMPI devel] seg fault when using yalla, XRC, and yalla
Hi David, I was able to reproduce the issue you reported. When the command line doesn't specify the components to use, ompi will try to load/open all the ones available (and close them in the end) and then choose the components according to their priority and whether or not they were opened successfully. This means that even if pml yalla was the one running, other components were opened and closed as well. The parameter you are using - btl_openib_receive_queues, doesn't have an effect on pml yalla. It only affects the openib btl which is used by pml ob1. Using the verbosity of btl_base_verbose I see that when the segmentation fault happens, the code doesn't reach the phase of unloading the openib btl so perhaps the problem originates there (since pml yalla was already unloaded). Can you please try adding this mca parameter to your command line to specify the HCA you are using? -mca btl_openib_if_include It made the segv go away for me. Can you please attach the output of ibv_devinfo and write the MOFED version you are using? Thank you, Alina. On Wed, Apr 20, 2016 at 2:53 PM, Joshua Ladd wrote: > Hi, David > > We are looking into your report. > > Best, > > Josh > > On Tue, Apr 19, 2016 at 4:41 PM, David Shrader wrote: > >> Hello, >> >> I have been investigating using XRC on a cluster with a mellanox >> interconnect. I have found that in a certain situation I get a seg fault. I >> am using 1.10.2 compiled with gcc 5.3.0, and the simplest configure line >> that I have found that still results in the seg fault is as follows: >> >> $> ./configure --with-hcoll --with-mxm --prefix=... >> >> I do have mxm 3.4.3065 and hcoll 3.3.768 installed in to system space >> (/usr/lib64). If I use '--without-hcoll --without-mxm,' the seg fault does >> not happen. >> >> The seg fault happens even when using examples/hello_c.c, so here is an >> example of the seg fault using it: >> >> $> mpicc hello_c.c -o hello_c.x >> $> mpirun -n 1 ./hello_c.x >> Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI >> dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: >> v1.10.1-145-g799148f, Jan 21, 2016, 135) >> $> mpirun -n 1 -mca btl_openib_receive_queues >> X,4096,1024:X,12288,512:X,65536,512 >> Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI >> dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: >> v1.10.1-145-g799148f, Jan 21, 2016, 135) >> -- >> mpirun noticed that process rank 0 with PID 22819 on node mu0001 exited >> on signal 11 (Segmentation fault). >> -- >> >> The seg fault happens no matter the number of ranks. I have tried the >> above command with '-mca pml_base_verbose,' and it shows that I am using >> the yalla pml: >> >> $> mpirun -n 1 -mca btl_openib_receive_queues >> X,4096,1024:X,12288,512:X,65536,512 -mca pml_base_verbose 100 ./hello_c.x >> ...output snipped... >> [mu0001.localdomain:22825] select: component cm not selected / finalized >> [mu0001.localdomain:22825] select: component ob1 not selected / finalized >> [mu0001.localdomain:22825] select: component yalla selected >> ...output snipped... >> -- >> mpirun noticed that process rank 0 with PID 22825 on node mu0001 exited >> on signal 11 (Segmentation fault). >> -- >> >> Interestingly enough, if I tell mpirun what pml to use, the seg fault >> goes away. The following command does not get the seg fault: >> >> $> mpirun -n 1 -mca btl_openib_receive_queues >> X,4096,1024:X,12288,512:X,65536,512 -mca pml yalla ./hello_c.x >> >> Passing either ob1 or cm to '-mca pml' also works. So it seems that the >> seg fault comes about when the yalla pml is chosen by default, when >> mxm/hcoll is involved, and using XRC. I'm not sure if mxm is to blame, >> however, as using '-mca pml cm -mca mtl mxm' with the XRC parameters >> doesn't throw the seg fault. >> >> Other information... >> OS: RHEL 6.7-based (TOSS) >> OpenFabrics: RedHat provided >> Kernel: 2.6.32-573.8.1.2chaos.ch5.4.x86_64 >> Config.log and 'ompi_info --all' are in the tarball ompi.tar.bz2 which is >> attached. >> >> Is there something else I should be doing with the yalla pml when using >> XRC? Regardless, I hope reporting the seg fault is useful. >> >> Thanks, >> David >> >> -- >> David Shrader >> HPC-ENV High Performance Computer Systems >> Los Alamos National Lab >> Email: dshrader lanl.gov >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2016/04/18786.php >> > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.o
Re: [OMPI devel] seg fault when using yalla, XRC, and yalla
David, thanks for the info you provided. I will try to dig in further to see what might be causing this issue. In the meantime, maybe Nathan can please comment about the openib btl behavior here? Thanks, Alina. On Wed, Apr 20, 2016 at 8:01 PM, David Shrader wrote: > Hello Alina, > > Thank you for the information about how the pml components work. I knew > that the other components were being opened and ultimately closed in favor > of yalla, but I didn't realize that initial open would cause a persistent > change in the ompi runtime. > > Here's the information you requested about the ib network: > > - MOFED version: > We are using the Open Fabrics Software as bundled by RedHat, and my ib > network folks say we're running something close to v1.5.4 > - ibv_devinfo: > [dshrader@mu0001 examples]$ ibv_devinfo > hca_id: mlx4_0 > transport: InfiniBand (0) > fw_ver: 2.9.1000 > node_guid: 0025:90ff:ff16:78d8 > sys_image_guid: 0025:90ff:ff16:78db > vendor_id: 0x02c9 > vendor_part_id: 26428 > hw_ver: 0xB0 > board_id: SM_212101000 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu:4096 (5) > active_mtu: 4096 (5) > sm_lid: 250 > port_lid: 366 > port_lmc: 0x00 > link_layer: InfiniBand > > > I still get the seg fault when specifying the hca: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include mlx4_0 > ./hello_c.x > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > -- > mpirun noticed that process rank 0 with PID 10045 on node mu0001 exited on > signal 11 (Segmentation fault). > -- > > I don't know if this helps, but the first time I tried the command I > mistyped the hca name. This got me a warning, but no seg fault: > > $> mpirun -n 1 -mca btl_openib_receive_queues > X,4096,1024:X,12288,512:X,65536,512 -mca btl_openib_if_include ml4_0 > ./hello_c.x > -- > WARNING: One or more nonexistent OpenFabrics devices/ports were > specified: > > Host: mu0001 > MCA parameter:mca_btl_if_include > Nonexistent entities: ml4_0 > > These entities will be ignored. You can disable this warning by > setting the btl_openib_warn_nonexistent_if MCA parameter to 0. > -- > Hello, world, I am 0 of 1, (Open MPI v1.10.2, package: Open MPI > dshra...@mu-fey.lanl.gov Distribution, ident: 1.10.2, repo rev: > v1.10.1-145-g799148f, Jan 21, 2016, 135) > > So, telling the openib btl to use the actual hca didn't get the seg fault > to go away, but giving it a dummy value did. > > Thanks, > David > > > On 04/20/2016 08:13 AM, Alina Sklarevich wrote: > > Hi David, > > I was able to reproduce the issue you reported. > > When the command line doesn't specify the components to use, ompi will try > to load/open all the ones available (and close them in the end) and then > choose the components according to their priority and whether or not they > were opened successfully. > This means that even if pml yalla was the one running, other components > were opened and closed as well. > > The parameter you are using - btl_openib_receive_queues, doesn't have an > effect on pml yalla. It only affects the openib btl which is used by pml > ob1. > > Using the verbosity of btl_base_verbose I see that when the segmentation > fault happens, the code doesn't reach the phase of unloading the openib btl > so perhaps the problem originates there (since pml yalla was already > unloaded). > > Can you please try adding this mca parameter to your command line to > specify the HCA you are using? > -mca btl_openib_if_include > It made the segv go away for me. > > Can you please attach the output of ibv_devinfo and write the MOFED > version you are using? > > Thank you, > Alin
Re: [OMPI devel] seg fault when using yalla, XRC, and yalla
HI, When the segmentation fault happens, I get the following trace: (gdb) bt #0 0x7fffee4f007d in ibv_close_xrcd (xrcd=0x2) at /usr/include/infiniband/verbs.h:1227 #1 0x7fffee4f055f in mca_btl_openib_close_xrc_domain (device=0xfb20c0) at btl_openib_xrc.c:104 #2 0x7fffee4da073 in device_destruct (device=0xfb20c0) at btl_openib_component.c:978 #3 0x7fffee4ce9f7 in opal_obj_run_destructors (object=0xfb20c0) at ../../../../opal/class/opal_object.h:460 #4 0x7fffee4d4f82 in mca_btl_openib_finalize_resources (btl=0xfbbc40) at btl_openib.c:1703 #5 0x7fffee4d511c in mca_btl_openib_finalize (btl=0xfbbc40) at btl_openib.c:1730 #6 0x776b26d6 in mca_btl_base_close () at base/btl_base_frame.c:192 #7 0x7769c73d in mca_base_framework_close (framework=0x7795bda0) at mca_base_framework.c:214 #8 0x77d448e2 in mca_bml_base_close () at base/bml_base_frame.c:130 #9 0x7769c73d in mca_base_framework_close (framework=0x77fe4f00) at mca_base_framework.c:214 #10 0x77cd4d18 in ompi_mpi_finalize () at runtime/ompi_mpi_finalize.c:415 #11 0x77cfee0b in PMPI_Finalize () at pfinalize.c:47 #12 0x00400880 in main () Looks like the problem originates from the openib btl finalize flow (if openib wasn't chosen for the run). This doesn't happen however when ob1 is specified from the command line as David mentioned. Btl openib behaves differently in these cases - in mca_btl_openib_finalize_resources specifically. When pml yalla is specified from the command line, this flow isn't invoked at all so in this case the segv doesn't happen as well. Thanks, Alina. On Thu, Apr 21, 2016 at 6:55 PM, Nathan Hjelm wrote: > > In 1.10.x is possible for the BTLs to be in use by ether ob1 or an > oshmem component. In 2.x one-sided components can also use BTLs. The MTL > interface doesn't not provide support for accessing hardware atomics and > RDMA. As for UD it stands for Unconnected Datagram. Its usage gets > better messaage rates for small messages but really hurts bandwidth. Our > applications are bandwidth bound and not message rate bound so we should > be using XRC not UD. > > -Nathan > > On Thu, Apr 21, 2016 at 09:33:06AM -0600, David Shrader wrote: > >Hey Nathan, > > > >I thought only 1 pml could be loaded at a time, and the only pml that > >could use btl's was ob1. If that is the case, how can the openib btl > run > >at the same time as cm and yalla? > > > >Also, what is UD? > > > >Thanks, > >David > > > >On 04/21/2016 09:25 AM, Nathan Hjelm wrote: > > > > The openib btl should be able to run alongside cm/mxm or yalla. If I > > have time this weekend I will get on the mustang and see what the > > problem is. The best answer is to change the openmpi-mca-params.conf in > > the install to have pml = ob1. I have seen little to no benefit with > > using MXM on mustang. In fact, the default configuration (which uses UD) > > gets terrible bandwidth. > > > > -Nathan > > > > On Thu, Apr 21, 2016 at 01:48:46PM +0300, Alina Sklarevich wrote: > > > > David, thanks for the info you provided. > > I will try to dig in further to see what might be causing this issue. > > In the meantime, maybe Nathan can please comment about the openib btl > > behavior here? > > Thanks, > > Alina. > > On Wed, Apr 20, 2016 at 8:01 PM, David Shrader > wrote: > > > > Hello Alina, > > > > Thank you for the information about how the pml components work. I > knew > > that the other components were being opened and ultimately closed > in > > favor of yalla, but I didn't realize that initial open would cause > a > > persistent change in the ompi runtime. > > > > Here's the information you requested about the ib network: > > > > - MOFED version: > > We are using the Open Fabrics Software as bundled by RedHat, and > my ib > > network folks say we're running something close to v1.5.4 > > - ibv_devinfo: > > [dshrader@mu0001 examples]$ ibv_devinfo > > hca_id: mlx4_0 > > transport: InfiniBand (0) > > fw_ver: 2.9.1000 > > node_guid: 0025:90ff:ff16:78d8 > > sys_image_guid: 0025:90ff:ff16:78db > > vendor_id: 0x02c9 > > vendor_part_id: 26428 > > hw_ver: 0xB0 &
Re: [OMPI devel] Openmpi support for Mellanox CX4-LX
Hi, For rdmacm to work with openib btl, the first receive queue needs to be a point-to-point queue (and not SRQ which is the default in OMPI v2.x). Can you please try adding this parameter to the command line? -mca btl_openib_receive_queues P,4096,8,6,4 You can change the numbers according to what you need. Alina. On Tue, Jun 13, 2017 at 7:57 PM, Chuanxiong Guo wrote: > here it is: > ~/openmpi/bin/mpirun -np 2 -hostfile hostfile --mca btl openib,self,sm > --mca btl_openib_cpc_include rdmacm --mca btl_openib_rroce_enable 1 > ./sendrecv > > what I got as follows. > > -- > WARNING: There was an error initializing an OpenFabrics device. > Local host: chguo-msr-linux1 > Local device: mlx5_0 > -- > -- > At least one pair of MPI processes are unable to reach each other for > MPI communications. This means that no Open MPI device has indicated > that it can be used to communicate between these processes. This is > an error; Open MPI requires that all MPI processes be able to reach > each other. This error can sometimes be the result of forgetting to > specify the "self" BTL. > Process 1 ([[45408,1],0]) is on host: chguo-msr-linux1 > Process 2 ([[45408,1],1]) is on host: chguo-msr-linux02 > BTLs attempted: self > Your MPI job is now going to abort; sorry. > -- > [chguo-msr-linux1:12690] *** An error occurred in MPI_Send > [chguo-msr-linux1:12690] *** reported by process [140379686961153, > 140376711102464] > [chguo-msr-linux1:12690] *** on communicator MPI_COMM_WORLD > [chguo-msr-linux1:12690] *** MPI_ERR_INTERN: internal error > [chguo-msr-linux1:12690] *** MPI_ERRORS_ARE_FATAL (processes in this > communicator will now abort, > [chguo-msr-linux1:12690] ***and potentially your MPI job) > [chguo-msr-linux1:12684] 1 more process has sent help message > help-mpi-btl-openib.txt / error in device init > [chguo-msr-linux1:12684] Set MCA parameter "orte_base_help_aggregate" to 0 > to see all help / error messages > > > On Tue, Jun 13, 2017 at 5:05 AM, Joshua Ladd wrote: > >> Hi, >> >> Please include your full command line. >> >> Josh >> >> On Mon, Jun 12, 2017 at 6:17 PM, Chuanxiong Guo > > wrote: >> >>> Hi, >>> >>> I have two servers with Mellanox CX4-LX (50GbE Ethernet) back-to-back >>> connected. I am using Ubuntu 14-04. I have made mvapich2 work, and I can >>> confirm both roce and rocev2 work well (by packet capturing). >>> >>> But I still cannot make openmpi work with roce. I am using openmpi >>> 2.1.1. >>> It looks that this version of openmpi does not recognize CX4-LX, which I >>> have added vendor part id 4117 to mca-btl-openib-device-params.ini, and >>> I have also updated opal/mca/common/verbs/common_verbs_port.c to >>> support CX4-LX, which has speed 64 and width 1. >>> >>> But I am still getting: >>> >>> "WARNING: There was an error initializing an OpenFabrics device. >>> Local host: chguo-msr-linux1 >>> >>> Local device: mlx5_0 >>> " >>> Any hint on what are missing? >>> >>> Thanks, >>> CX >>> >>> >>> ___ >>> devel mailing list >>> devel@lists.open-mpi.org >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>> >> >> > > ___ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
[OMPI devel] hwloc error
Hello, I am using ompi-v1.8 and have come across the following error: -- Open MPI tried to bind a new process, but something went wrong. The process was killed without launching the target application. Your job will now abort. Local host:vegas17 Application name: trivial/test_get__trivial/c_hello Error message: hwloc_set_cpubind returned "Error" for bitmap "0,16" Location: odls_default_module.c:551 -- This happens when running a simple trivial test with the following command line: mpirun --map-by node --bind-to core -display-map -np 2 -mca pml ob1 …/trivial/test_get__trivial/c_hello What seems to eliminate this error is changing the binding policy from core to none (--bind-to none). The only nodes which are issuing this error are always the nodes which have GPUs in them. When running the same command line on other non-GPU nodes, there is no error. I’m using Slurm to allocate the nodes. Has anyone seen this issue or knows what’s wrong here? Thanks, Alina.
Re: [OMPI devel] hwloc error
Thanks Ralph, adding --hetero-nodes to the command line solved this issue. Alina. On Mon, Sep 15, 2014 at 6:51 AM, Ralph Castain wrote: > Try adding --hetero-nodes to your mpirun cmd line > > On Sep 14, 2014, at 8:25 AM, Alina Sklarevich > wrote: > > Hello, > > > I am using ompi-v1.8 and have come across the following error: > > > -- > > Open MPI tried to bind a new process, but something went wrong. The > > process was killed without launching the target application. Your job > > will now abort. > > > Local host:vegas17 > > Application name: trivial/test_get__trivial/c_hello > > Error message: hwloc_set_cpubind returned "Error" for bitmap "0,16" > > Location: odls_default_module.c:551 > > -- > > > This happens when running a simple trivial test with the following command > line: > > > mpirun --map-by node --bind-to core -display-map -np 2 -mca pml ob1 > …/trivial/test_get__trivial/c_hello > > > What seems to eliminate this error is changing the binding policy from > core to none (--bind-to none). > > The only nodes which are issuing this error are always the nodes which > have GPUs in them. > > When running the same command line on other non-GPU nodes, there is no > error. > > I’m using Slurm to allocate the nodes. > > > Has anyone seen this issue or knows what’s wrong here? > > > Thanks, > > Alina. > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15824.php > > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15825.php >
Re: [OMPI devel] Need to know your Github ID
ompimtttester On Thu, Sep 18, 2014 at 11:38 AM, Alex Margolin < alex.margo...@mail.huji.ac.il> wrote: > alex -> alex-ma > alinas -> alinask > > amikheev -> alex-mikheev > > vasily -> vasilyMellanox > > > On Wed, Sep 10, 2014 at 1:46 PM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > >> As the next step of the planned migration to Github, I need to know: >> >> - Your Github ID (so that you can be added to the new OMPI git repo) >> - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac >> tickets to appropriate owners) >> >> Here's the list of SVN IDs who have committed over the past year -- I'm >> guessing that most of these people will need Github IDs: >> >> adrian >> alekseys >> alex >> alinas >> amikheev >> bbenton >> bosilca (done) >> bouteill >> brbarret >> bwesarg >> devendar >> dgoodell (done) >> edgar >> eugene >> ggouaillardet >> hadi >> hjelmn >> hpcchris >> hppritcha >> igoru >> jjhursey (done) >> jladd >> jroman >> jsquyres (done) >> jurenz >> kliteyn >> manjugv >> miked (done) >> mjbhaskar >> mpiteam (done) >> naughtont >> osvegis >> pasha >> regrant >> rfaucett >> rhc (done) >> rolfv (done) >> samuel >> shiqing >> swise >> tkordenbrock >> vasily >> vvenkates >> vvenkatesan >> yaeld >> yosefe >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15788.php >> > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15864.php >
[OMPI devel] osu_mbw_mr error
Hi, On 1.8.4rc1 we observe the following assert in the osu_mbw_mr test when using the openib BTL. When compiled in production mode (i.e. no --enable-debug) the test simply hangs. When using either the tcp BTL or the cm PML, the benchmark completes without error. The command line to reproduce this is: $ mpirun --bind-to core -display-map -mca btl_openib_if_include mlx5_0:1 -np 2 -mca pml ob1 -mca btl openib,self,sm ./osu_mbw_mr # OSU MPI Multiple Bandwidth / Message Rate Test v4.4 # [ pairs: 1 ] [ window size: 64 ] # Size MB/sMessages/s osu_mbw_mr: ../../../../opal/class/opal_list.h:547: _opal_list_append: Assertion `0 == item->opal_list_item_refcount' failed. [vegas15:30395] *** Process received signal *** [vegas15:30395] Signal: Aborted (6) [vegas15:30395] Signal code: (-6) [vegas15:30395] [ 0] /lib64/libpthread.so.0[0x30bc40f500] [vegas15:30395] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x30bc0328a5] [vegas15:30395] [ 2] /lib64/libc.so.6(abort+0x175)[0x30bc034085] [vegas15:30395] [ 3] /lib64/libc.so.6[0x30bc02ba1e] [vegas15:30395] [ 4] /lib64/libc.so.6(__assert_perror_fail+0x0)[0x30bc02bae0] [vegas15:30395] [ 5] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_btl_openib.so(+0x9087)[0x73f70087] [vegas15:30395] [ 6] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_btl_openib.so(mca_btl_openib_alloc+0x403)[0x73f754b3] [vegas15:30395] [ 7] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_btl_openib.so(mca_btl_openib_sendi+0xf9e)[0x73f785b4] [vegas15:30395] [ 8] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_pml_ob1.so(+0xed08)[0x73308d08] [vegas15:30395] [ 9] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_pml_ob1.so(+0xf8ba)[0x733098ba] [vegas15:30395] [10] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_isend+0x108)[0x73309a1f] [vegas15:30395] [11] /labhome/alinas/workspace/tt/ompi_rc1/openmpi-1.8.4rc1/install/lib/libmpi.so.1(MPI_Isend+0x2ec)[0x77cff5e8] [vegas15:30395] [12] /hpc/local/benchmarks/hpc-stack-gcc/install/ompi-mellanox-v1.8/tests/osu-micro-benchmarks-4.4/osu_mbw_mr[0x400fa4] [vegas15:30395] [13] /hpc/local/benchmarks/hpc-stack-gcc/install/ompi-mellanox-v1.8/tests/osu-micro-benchmarks-4.4/osu_mbw_mr[0x40167d] [vegas15:30395] [14] /lib64/libc.so.6(__libc_start_main+0xfd)[0x30bc01ecdd] [vegas15:30395] [15] /hpc/local/benchmarks/hpc-stack-gcc/install/ompi-mellanox-v1.8/tests/osu-micro-benchmarks-4.4/osu_mbw_mr[0x400db9] [vegas15:30395] *** End of error message *** -- mpirun noticed that process rank 0 with PID 30395 on node vegas15 exited on signal 6 (Aborted). -- Thanks, Alina.
[OMPI devel] thread-tests hang
Hi,
We observe a hang when running the multi-threading support test "latency.c"
(attached to this report), which uses MPI_THREAD_MULTIPLE.
The hang happens immediately at the begining of the test and is reproduced
in the v1.8 release branch.
The command line to reproduce the behavior is:
$ mpirun --map-by node --bind-to core -display-map -np 2 -mca pml ob1 -mca
btl tcp,self ./thread-tests-1.1/latency
The last commit with which the hang doesn't reproduce is:
commit: e4d4266d9c69e
And problems begin after commit:
commit 09b867374e9618007b81bfaf674ec6df04548bed
Author: Jeff Squyres
List-Post: devel@lists.open-mpi.org
Date: Fri Oct 31 12:42:50 2014 -0700
Revert most of open-mpi/ompi@6ef938de3fa9ca0fed2c5bcb0736f65b0d8803af
Is this expected behavior? In other words, should we not expect any stable
release in the 1.8.x series to be able to use MPI_THREAD_MULTIPLE with even
the TCP and SM BTLs?
Please advise.
Thanks,
Alina.
/* -*- Mode: C; c-basic-offset:4 ; -*- */
/*
* Copyright (C) 2007 University of Chicago
* See COPYRIGHT notice in top-level directory.
*/
#include "mpi.h"
#include
#include
#include
#define MAXSIZE 1024
#define NTIMES 1000
/* Measures concurrent latency (for short messages). All even ranks
send to rank i+1 and get a reply. Run on 2 nodes (with multiple processes).
*/
int main(int argc,char *argv[])
{
int rank, nprocs, src, dest, tag, i, size, incr;
double stime, etime, ttime;
char *sendbuf, *recvbuf;
int provided;
//MPI_Init(&argc,&argv);
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
if (provided != MPI_THREAD_MULTIPLE) {
printf("Thread multiple needed\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
sendbuf = (char *) malloc(MAXSIZE);
if (!sendbuf) {
printf("Cannot allocate buffer\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
recvbuf = (char *) malloc(MAXSIZE);
if (!recvbuf) {
printf("Cannot allocate buffer\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
incr = 16;
if (rank == 0) printf("Size (bytes) \t Time (us)\n");
MPI_Barrier(MPI_COMM_WORLD);
/* All even ranks send to (and recv from) rank i+1 many times */
if ((rank % 2) == 0) { /* even */
dest = rank + 1;
for (size=0; size<=MAXSIZE; size+=incr) {
tag = 0;
stime = MPI_Wtime();
for (i=0; i
[OMPI devel] segmentation fault on an accumulate-fence test
Dear OMPI community, We observe a segmentation fault in our regression testing. Our initial investigation shows that It happens for any 1.8.x release and with any PML/BTL/MTL combo on two processes, when running the MPICH one-sided test, accumulate-fence test, attached to this report with the following command line: $mpirun -np 2 --bind-to core --display-map --map-by node -mca pml ob1 -mca btl self,openib ../test/mpi/rma/accfence1 The initial trace is: Data for JOB [16088,1] offset 0 JOB MAP Data for node: vegas15 Num slots: 16 Max slots: 0Num procs: 1 Process OMPI jobid: [16088,1] App: 0 Process rank: 0 Data for node: vegas16 Num slots: 16 Max slots: 0Num procs: 1 Process OMPI jobid: [16088,1] App: 0 Process rank: 1 = [vegas16:22098] *** Process received signal *** [vegas16:22098] Signal: Segmentation fault (11) [vegas16:22098] Signal code: Address not mapped (1) [vegas16:22098] Failing at address: 0x34 [vegas16:22098] [ 0] /lib64/libpthread.so.0[0x3f6e80f710] [vegas16:22098] [ 1] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_int_free+0x188)[0x7772baa2] [vegas16:22098] [ 2] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_free+0x98)[0x7772a1f5] [vegas16:22098] [ 3] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(+0xd6f59)[0x77728f59] [vegas16:22098] [ 4] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(+0x2f884)[0x77c92884] [vegas16:22098] [ 5] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_attr_delete_all+0x2eb)[0x77c92dbe] [vegas16:22098] [ 6] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_comm_free+0x6a)[0x77c99336] [vegas16:22098] [ 7] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_free+0x921)[0x732ab3bc] [vegas16:22098] [ 8] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_win_free+0x24)[0x77cc0c87] [vegas16:22098] [ 9] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(MPI_Win_free+0xb8)[0x77d2b702] [vegas16:22098] [10] /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402447] [vegas16:22098] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3f6e41ed1d] [vegas16:22098] [12] /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402119] [vegas16:22098] *** End of error message *** And subsequent investigation of the core file generates the following hints: (gdb) bt #0 0x77722a96 in opal_memory_ptmalloc2_int_free (av=0x7796b320, mem=0x7125a0) at malloc.c:4402 #1 0x777211f5 in opal_memory_ptmalloc2_free (mem=0x7125a0) at malloc.c:3511 #2 0x7771ff59 in opal_memory_linux_free_hook (__ptr=0x7125a0, caller=0x7769a8f6) at hooks.c:709 #3 0x7769a8f6 in opal_datatype_destruct (datatype=0x7123b0) at opal_datatype_create.c:59 #4 0x73346ad0 in opal_obj_run_destructors (object=0x7123b0) at ../../../../opal/class/opal_object.h:448 #5 0x7334af68 in process_acc (module=0x70e370, source=0, acc_header=0x70fef0) at osc_rdma_data_move.c:1184 #6 0x7334c752 in process_frag (module=0x70e370, frag=0x70fee0) at osc_rdma_data_move.c:1576 #7 0x7334cafb in ompi_osc_rdma_callback (request=0x700b80) at osc_rdma_data_move.c:1656 #8 0x73db3770 in ompi_request_complete (request=0x700b80, with_signal=true) at ../../../../ompi/request/request.h:402 #9 0x73db3f11 in recv_request_pml_complete (recvreq=0x700b80) at pml_ob1_recvreq.h:181 #10 0x73db5019 in mca_pml_ob1_recv_frag_callback_match (btl=0x741d9c20, tag=65 'A', des=0x7fffd210, cbdata=0x0) at pml_ob1_recvfrag.c:243 #11 0x73fd6c4b in mca_btl_sm_component_progress () at btl_sm_component.c:1087 #12 0x77678d66 in opal_progress () at runtime/opal_progress.c:187 #13 0x73dabb44 in opal_condition_wait (c=0x77ffa120, m=0x77ffa160) at ../../../../opal/threads/condition.h:78 #14 0x73dabcc6 in ompi_request_wait_completion (req=0x7fffd410) at ../../../../ompi/request/request.h:381 #15 0x73dac9da in mca_pml_ob1_recv (addr=0x7fffd9ec, count=1, datatype=0x77fe25c0, src=0, tag=-24, comm=0x70dac0, status=0x0) at pml_ob1_irecv.c:109 #16 0x72cd2868 in ompi_coll_tuned_scatter_intra_basic_linear (sbuf=0x0, scount=1, sdtype=0x77fe25c0, rbuf=0x7fffd9ec, rcount=1, rdtype=0x77fe25c0, root=0, comm=0x70dac0, module=0x70fa20) at coll_tuned_scatter.c:231 #17 0x72cbbd75 in ompi_coll_tuned_scatter_intra_dec_fixed (sbuf=0x0, scount=1, sdtype=0x77fe25c0, rbuf=0x7fffd9ec, rcount=1, rdtype=0x77fe25c0, root=0, comm=
Re: [OMPI devel] segmentation fault on an accumulate-fence test
Attaching the test for reproduction. On Mon, Jan 19, 2015 at 11:48 AM, Alina Sklarevich < ali...@dev.mellanox.co.il> wrote: > Dear OMPI community, > > > > We observe a segmentation fault in our regression testing. Our initial > investigation shows that It happens for any 1.8.x release and with any > PML/BTL/MTL combo on two processes, when running the MPICH one-sided test, > accumulate-fence test, attached to this report with the following command > line: > > > > $mpirun -np 2 --bind-to core --display-map --map-by node -mca pml ob1 -mca > btl self,openib ../test/mpi/rma/accfence1 > > > > The initial trace is: > > > > Data for JOB [16088,1] offset 0 > > > > JOB MAP > > > > Data for node: vegas15 Num slots: 16 Max slots: 0Num procs: 1 > >Process OMPI jobid: [16088,1] App: 0 Process rank: 0 > > > > Data for node: vegas16 Num slots: 16 Max slots: 0Num procs: 1 > >Process OMPI jobid: [16088,1] App: 0 Process rank: 1 > > > > = > > [vegas16:22098] *** Process received signal *** > > [vegas16:22098] Signal: Segmentation fault (11) > > [vegas16:22098] Signal code: Address not mapped (1) > > [vegas16:22098] Failing at address: 0x34 > > [vegas16:22098] [ 0] /lib64/libpthread.so.0[0x3f6e80f710] > > [vegas16:22098] [ 1] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_int_free+0x188)[0x7772baa2] > > [vegas16:22098] [ 2] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_free+0x98)[0x7772a1f5] > > [vegas16:22098] [ 3] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(+0xd6f59)[0x77728f59] > > [vegas16:22098] [ 4] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(+0x2f884)[0x77c92884] > > [vegas16:22098] [ 5] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_attr_delete_all+0x2eb)[0x77c92dbe] > > [vegas16:22098] [ 6] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_comm_free+0x6a)[0x77c99336] > > [vegas16:22098] [ 7] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_free+0x921)[0x732ab3bc] > > [vegas16:22098] [ 8] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_win_free+0x24)[0x77cc0c87] > > [vegas16:22098] [ 9] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(MPI_Win_free+0xb8)[0x77d2b702] > > [vegas16:22098] [10] > /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402447] > > [vegas16:22098] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3f6e41ed1d] > > [vegas16:22098] [12] > /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402119] > > [vegas16:22098] *** End of error message *** > > > > > > And subsequent investigation of the core file generates the following > hints: > > > > (gdb) bt > > #0 0x77722a96 in opal_memory_ptmalloc2_int_free > (av=0x7796b320, mem=0x7125a0) at malloc.c:4402 > > #1 0x777211f5 in opal_memory_ptmalloc2_free (mem=0x7125a0) at > malloc.c:3511 > > #2 0x7771ff59 in opal_memory_linux_free_hook (__ptr=0x7125a0, > caller=0x7769a8f6) at hooks.c:709 > > #3 0x7769a8f6 in opal_datatype_destruct (datatype=0x7123b0) at > opal_datatype_create.c:59 > > #4 0x73346ad0 in opal_obj_run_destructors (object=0x7123b0) at > ../../../../opal/class/opal_object.h:448 > > #5 0x7334af68 in process_acc (module=0x70e370, source=0, > acc_header=0x70fef0) at osc_rdma_data_move.c:1184 > > #6 0x7334c752 in process_frag (module=0x70e370, frag=0x70fee0) at > osc_rdma_data_move.c:1576 > > #7 0x7334cafb in ompi_osc_rdma_callback (request=0x700b80) at > osc_rdma_data_move.c:1656 > > #8 0x73db3770 in ompi_request_complete (request=0x700b80, > with_signal=true) at ../../../../ompi/request/request.h:402 > > #9 0x73db3f11 in recv_request_pml_complete (recvreq=0x700b80) at > pml_ob1_recvreq.h:181 > > #10 0x73db5019 in mca_pml_ob1_recv_frag_callback_match > (btl=0x741d9c20, tag=65 'A', des=0x7fffd210, cbdata=0x0) at > pml_ob1_recvfrag.c:243 > > #11 0x73fd6c4b in mca_btl_sm_component_progress () at > btl_sm_component.c:1087 > > #12 0x77678d66 in opal_progress () at runtime/opal_progress.c:187 > > #13 0x7
Re: [OMPI devel] segmentation fault on an accumulate-fence test
Hi, Sure: https://github.com/open-mpi/ompi-release/issues/178 Thanks, Alina. On Sat, Jan 31, 2015 at 3:39 PM, Jeff Squyres (jsquyres) wrote: > Alina -- > > Sorry; I think this bug report got lost in the run-up to the Open MPI dev > meeting last week, and that fact that Nathan (the primary one-sided > maintainer) is taking a little time off. > > Can you file a Github issue about this? > > Thanks. > > > > On Jan 19, 2015, at 4:50 AM, Alina Sklarevich > wrote: > > > > Attaching the test for reproduction. > > > > On Mon, Jan 19, 2015 at 11:48 AM, Alina Sklarevich < > ali...@dev.mellanox.co.il> wrote: > > Dear OMPI community, > > > > > > We observe a segmentation fault in our regression testing. Our initial > investigation shows that It happens for any 1.8.x release and with any > PML/BTL/MTL combo on two processes, when running the MPICH one-sided test, > accumulate-fence test, attached to this report with the following command > line: > > > > > > $mpirun -np 2 --bind-to core --display-map --map-by node -mca pml ob1 > -mca btl self,openib ../test/mpi/rma/accfence1 > > > > > > The initial trace is: > > > > > > Data for JOB [16088,1] offset 0 > > > > > > JOB MAP > > > > > > Data for node: vegas15 Num slots: 16 Max slots: 0Num procs: 1 > > > >Process OMPI jobid: [16088,1] App: 0 Process rank: 0 > > > > > > Data for node: vegas16 Num slots: 16 Max slots: 0Num procs: 1 > > > >Process OMPI jobid: [16088,1] App: 0 Process rank: 1 > > > > > > = > > > > [vegas16:22098] *** Process received signal *** > > > > [vegas16:22098] Signal: Segmentation fault (11) > > > > [vegas16:22098] Signal code: Address not mapped (1) > > > > [vegas16:22098] Failing at address: 0x34 > > > > [vegas16:22098] [ 0] /lib64/libpthread.so.0[0x3f6e80f710] > > > > [vegas16:22098] [ 1] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_int_free+0x188)[0x7772baa2] > > > > [vegas16:22098] [ 2] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_free+0x98)[0x7772a1f5] > > > > [vegas16:22098] [ 3] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(+0xd6f59)[0x77728f59] > > > > [vegas16:22098] [ 4] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(+0x2f884)[0x77c92884] > > > > [vegas16:22098] [ 5] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_attr_delete_all+0x2eb)[0x77c92dbe] > > > > [vegas16:22098] [ 6] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_comm_free+0x6a)[0x77c99336] > > > > [vegas16:22098] [ 7] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_free+0x921)[0x732ab3bc] > > > > [vegas16:22098] [ 8] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_win_free+0x24)[0x77cc0c87] > > > > [vegas16:22098] [ 9] > /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(MPI_Win_free+0xb8)[0x77d2b702] > > > > [vegas16:22098] [10] > /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402447] > > > > [vegas16:22098] [11] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x3f6e41ed1d] > > > > [vegas16:22098] [12] > /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402119] > > > > [vegas16:22098] *** End of error message *** > > > > > > > > And subsequent investigation of the core file generates the following > hints: > > > > > > (gdb) bt > > > > #0 0x77722a96 in opal_memory_ptmalloc2_int_free > (av=0x7796b320, mem=0x7125a0) at malloc.c:4402 > > > > #1 0x777211f5 in opal_memory_ptmalloc2_free (mem=0x7125a0) at > malloc.c:3511 > > > > #2 0x7771ff59 in opal_memory_linux_free_hook (__ptr=0x7125a0, > caller=0x7769a8f6) at hooks.c:709 > > > > #3 0x7769a8f6 in opal_datatype_destruct (datatype=0x7123b0) at > opal_datatype_create.c:59 > > > > #4 0x73346ad0 in opal_obj_run_destructors (object=0x7123b0) at > ../../../../opal/class/opal_object.h:448 > > > > #5 0x7334af68 in process_acc (module=0x70e370, source=0, > acc_header=0x70fef0) at o
Re: [OMPI devel] Unwanted ibv_fork_init() mess(ages) and complaint for non-IB login node
Hi, I will change the default of the opal_common_verbs_want_fork_support to -1 in order to avoid these messages in case ibv_fork_init() fails. The reason why it is failing is that ibv_fork_init() is being called to late. To avoid this, every component should call ibv_fork_init() early in the init (in this case before oob/ud does) - call the opal_common_verbs_fork_test() function which does just that. Paul, can you please check if adding this call to psm fixes the issue? On Wed, Mar 4, 2015 at 11:40 PM, Dave Goodell (dgoodell) wrote: > On Mar 4, 2015, at 3:25 PM, Paul Hargrove wrote: > > > On Wed, Mar 4, 2015 at 1:04 PM, Dave Goodell (dgoodell) < > dgood...@cisco.com> wrote: > > [...] > > > libibverbs: Warning: couldn't open config directory > '/etc/libibverbs.d'. > > > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > > > > I think that warning is printed by libibverbs itself. Are you 100% sure > there are no IB HCAs sitting in the head node? If there are IB HCAs but > you don't want them to be used, you might want to ensure that the various > verbs kernel modules don't get loaded, which is one half of the mismatch > which confuses libibverbs. > > [...] > > > > FWIW, I can confirm that these two lines are from libibverbs itself: > > $ strings /usr/lib64/libibverbs.a | grep -e 'no userspace' -e 'open > config directory' > > libibverbs: Warning: no userspace device-specific driver found for %s > > libibverbs: Warning: couldn't open config directory '%s'. > > Yes, I think you'd also see the same message if you run "ibv_devices" or > "ibv_devinfo" on the head node. > > > As it happens, the login node *does* have an HCA installed and the > kernel modules appears to be loaded. However, as the "17th node" in the > cluster it was never cabled to the 16-port switch and the package(s) that > should have created/populated /etc/libibverbs.d are *not* present > (specifically the login node has libipathverbs-devel installed but not > libipathverbs). > > > > So, Dave, are you saying that what I describe in the previous paragraph > would be considered "misconfiguration"? I am fine with dropping the > discussion of those first two lines if there is agreement that Open MPI > shouldn't be responsible for handling this case. > > I would consider that to be a lesser misconfiguration, which is only > really an issue because of libibverbs deficiencies. Either the hardware > could be removed from the head node or the kernel modules could be unloaded > / prevented from loading on the head node. > > > Now the ibv_fork_init() warnings are another issue entirely. Since > btl:verbs and mtl:psm both work (at least separately) perfectly fine on the > compute nodes, I don't believe that there are any configuration issues > there. > > Agreed, something needs to be improved there. I assume that Mike D. or > someone from his team will take a look. I don't have any bandwidth to look > at this myself. > > -Dave > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17100.php >
Re: [OMPI devel] Unwanted ibv_fork_init() mess(ages) and complaint for non-IB login node
If oob:ud was disabled then there was no call to ibv_fork_init() anywhere else, right? If so, then this is why the messages went away. The calls to ibv_fork_init() from the opal common verbs were pushed to the master. One of the places a call was set is oob:ud, but if there is a call to memory registering verbs before this place, then the call to it in oob:ud would result in a failure. On Thu, Mar 5, 2015 at 4:21 PM, Jeff Squyres (jsquyres) wrote: > This is not a PSM issue -- I believe Paul said that when he disabled > oob:ud, the messages went away. > > I'm sorry; I'm at the MPI Forum this week and not paying close attention > to master commits. Has your code to ensure to call the opal common verbs > ibv_fork_init() stuff been pushed to master yet? If so, then > ibv_fork_init() *should* be getting called first, and there's something > else going on that needs to be understood. > > > > > On Mar 5, 2015, at 1:57 AM, Alina Sklarevich > wrote: > > > > Hi, > > > > I will change the default of the opal_common_verbs_want_fork_support to > -1 in order to avoid these messages in case ibv_fork_init() fails. > > > > The reason why it is failing is that ibv_fork_init() is being called to > late. To avoid this, every component should call ibv_fork_init() early in > the init (in this case before oob/ud does) - call the > opal_common_verbs_fork_test() function which does just that. > > > > Paul, can you please check if adding this call to psm fixes the issue? > > > > On Wed, Mar 4, 2015 at 11:40 PM, Dave Goodell (dgoodell) < > dgood...@cisco.com> wrote: > > On Mar 4, 2015, at 3:25 PM, Paul Hargrove wrote: > > > > > On Wed, Mar 4, 2015 at 1:04 PM, Dave Goodell (dgoodell) < > dgood...@cisco.com> wrote: > > > [...] > > > > libibverbs: Warning: couldn't open config directory > '/etc/libibverbs.d'. > > > > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > > > > > > I think that warning is printed by libibverbs itself. Are you 100% > sure there are no IB HCAs sitting in the head node? If there are IB HCAs > but you don't want them to be used, you might want to ensure that the > various verbs kernel modules don't get loaded, which is one half of the > mismatch which confuses libibverbs. > > > [...] > > > > > > FWIW, I can confirm that these two lines are from libibverbs itself: > > > $ strings /usr/lib64/libibverbs.a | grep -e 'no userspace' -e 'open > config directory' > > > libibverbs: Warning: no userspace device-specific driver found for %s > > > libibverbs: Warning: couldn't open config directory '%s'. > > > > Yes, I think you'd also see the same message if you run "ibv_devices" or > "ibv_devinfo" on the head node. > > > > > As it happens, the login node *does* have an HCA installed and the > kernel modules appears to be loaded. However, as the "17th node" in the > cluster it was never cabled to the 16-port switch and the package(s) that > should have created/populated /etc/libibverbs.d are *not* present > (specifically the login node has libipathverbs-devel installed but not > libipathverbs). > > > > > > So, Dave, are you saying that what I describe in the previous > paragraph would be considered "misconfiguration"? I am fine with dropping > the discussion of those first two lines if there is agreement that Open MPI > shouldn't be responsible for handling this case. > > > > I would consider that to be a lesser misconfiguration, which is only > really an issue because of libibverbs deficiencies. Either the hardware > could be removed from the head node or the kernel modules could be unloaded > / prevented from loading on the head node. > > > > > Now the ibv_fork_init() warnings are another issue entirely. Since > btl:verbs and mtl:psm both work (at least separately) perfectly fine on the > compute nodes, I don't believe that there are any configuration issues > there. > > > > Agreed, something needs to be improved there. I assume that Mike D. or > someone from his team will take a look. I don't have any bandwidth to look > at this myself. > > > > -Dave > > > > ___ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17100.php > > > > ___ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17101.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17102.php >
Re: [OMPI devel] Unwanted ibv_fork_init() mess(ages) and complaint for non-IB login node
I don't know much about PSM either but shouldn't it be called only after the oob:ud code? If so, then ibv_fork_init() is being called from oob:ud early enough so either there is another reason for ibv_fork_init() failure (like you said), or the reason why this verb failed was the same reason why these warnings appeared? libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0 Also, opal_common_verbs_want_fork_support is now set to -1 like you suggested so these warnings shouldn't appear anymore. On Thu, Mar 5, 2015 at 4:51 PM, Jeff Squyres (jsquyres) wrote: > On Mar 5, 2015, at 6:32 AM, Alina Sklarevich > wrote: > > > > If oob:ud was disabled then there was no call to ibv_fork_init() > anywhere else, right? If so, then this is why the messages went away. > > Right. That's why I'm saying it doesn't seem like a PSM problem. > > (I don't know much about PSM, but I don't think it uses verbs...?) > > > The calls to ibv_fork_init() from the opal common verbs were pushed to > the master. One of the places a call was set is oob:ud, but if there is a > call to memory registering verbs before this place, then the call to it in > oob:ud would result in a failure. > > Yes, I think that is the exact question: why are these messages showing up > because of oob:ud? It seems like the call sequences to ibv_fork_init() are > not as understood as we thought they were. :-( I.e., it was presupposed > that oob_ud was the first entity to call any verbs code (and by your > commits, is supposed to be calling the common verbs code to call > ibv_fork_init() early enough such that it won't be a problem). But either > that is not the case, or ibv_fork_init() is failing for some other reason. > > These are the things that need to be figured out. > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17104.php >
Re: [OMPI devel] Unwanted ibv_fork_init() mess(ages) and complaint for non-IB login node
Jeff, If in this question you were asking me, "Can you confirm that fork support *does* work for you when you set the MCA param to 1?" then the answer is yes. Thanks, Alina. On Fri, Mar 6, 2015 at 4:56 AM, Paul Hargrove wrote: > In "work for you" does "you" == @PHHargrove? > > If YES: > 1) are the changes to be tested reflected in the master tarball yet? > 2) other than presence/absence of the warning how am I testing if the > support "works" when the param is set to 1? > > -Paul > > On Thu, Mar 5, 2015 at 6:29 PM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > >> On Mar 5, 2015, at 7:33 AM, Alina Sklarevich >> wrote: >> > >> > I don't know much about PSM either but shouldn't it be called only >> after the oob:ud code? >> > If so, then ibv_fork_init() is being called from oob:ud early enough so >> either there is another reason for ibv_fork_init() failure (like you said), >> or the reason why this verb failed was the same reason why these warnings >> appeared? >> > libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. >> > libibverbs: Warning: no userspace device-specific driver found for >> /sys/class/infiniband_verbs/uverbs0 >> > >> > Also, opal_common_verbs_want_fork_support is now set to -1 like you >> suggested so these warnings shouldn't appear anymore. >> >> Sorry we went around and around and ended up right back where we >> started. :-( >> >> It might well be worth putting a comment at the location in the code >> where the value is set to -1 explaining *why* it is set to -1 so that we >> don't revisit this circular discussion again someday. :-) >> >> Can you confirm that fork support *does* work for you when you set the >> MCA param to 1? If so, then at least those code paths have been fixed. >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/03/17112.php >> > > > > -- > Paul H. Hargrove phhargr...@lbl.gov > Computer Languages & Systems Software (CLaSS) Group > Computer Science Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/03/17114.php >
