Terry, Please find enclosed the requested check outputs (using -output-filename stdout.tag.null option).
For information, Nysal In his first message referred to ompi/mca/pml/ob1/pml_ob1_hdr.h and said that hdr->tg value was wrnong on receiving side. #define MCA_PML_OB1_HDR_TYPE_MATCH (MCA_BTL_TAG_PML + 1) #define MCA_PML_OB1_HDR_TYPE_RNDV (MCA_BTL_TAG_PML + 2) #define MCA_PML_OB1_HDR_TYPE_RGET (MCA_BTL_TAG_PML + 3) #define MCA_PML_OB1_HDR_TYPE_ACK (MCA_BTL_TAG_PML + 4) #define MCA_PML_OB1_HDR_TYPE_NACK (MCA_BTL_TAG_PML + 5) #define MCA_PML_OB1_HDR_TYPE_FRAG (MCA_BTL_TAG_PML + 6) #define MCA_PML_OB1_HDR_TYPE_GET (MCA_BTL_TAG_PML + 7) #define MCA_PML_OB1_HDR_TYPE_PUT (MCA_BTL_TAG_PML + 8) #define MCA_PML_OB1_HDR_TYPE_FIN (MCA_BTL_TAG_PML + 9) and in ompi/mca/btl/btl.h #define MCA_BTL_TAG_PML 0x40 Eloi On Monday 27 September 2010 14:36:59 Terry Dontje wrote: > I am thinking checking the value of *frag->hdr right before the return > in the post_send function in ompi/mca/btl/openib/btl_openib_endpoint.h. > It is line 548 in the trunk > https://svn.open-mpi.org/source/xref/ompi-trunk/ompi/mca/btl/openib/btl_ope > nib_endpoint.h#548 > > --td > > Eloi Gaudry wrote: > > Hi Terry, > > > > Do you have any patch that I could apply to be able to do so ? I'm > > remotely working on a cluster (with a terminal) and I cannot use any > > parallel debugger or sequential debugger (with a call to xterm...). I > > can track frag->hdr->tag value in > > ompi/mca/btl/openib/btl_openib_component.c::handle_wc in the > > SEND/RDMA_WRITE case, but this is all I can think of alone. > > > > You'll find a stacktrace (receive side) in this thread (10th or 11th > > message) but it might be pointless. > > > > Regards, > > Eloi > > > > On Monday 27 September 2010 11:43:55 Terry Dontje wrote: > >> So it sounds like coalescing is not your issue and that the problem has > >> something to do with the queue sizes. It would be helpful if we could > >> detect the hdr->tag == 0 issue on the sending side and get at least a > >> stack trace. There is something really odd going on here. > >> > >> --td > >> > >> Eloi Gaudry wrote: > >>> Hi Terry, > >>> > >>> I'm sorry to say that I might have missed a point here. > >>> > >>> I've lately been relaunching all previously failing computations with > >>> the message coalescing feature being switched off, and I saw the same > >>> hdr->tag=0 error several times, always during a collective call > >>> (MPI_Comm_create, MPI_Allreduce and MPI_Broadcast, so far). And as > >>> soon as I switched to the peer queue option I was previously using > >>> (--mca btl_openib_receive_queues P,65536,256,192,128 instead of using > >>> --mca btl_openib_use_message_coalescing 0), all computations ran > >>> flawlessly. > >>> > >>> As for the reproducer, I've already tried to write something but I > >>> haven't succeeded so far at reproducing the hdr->tag=0 issue with it. > >>> > >>> Eloi > >>> > >>> On 24/09/2010 18:37, Terry Dontje wrote: > >>>> Eloi Gaudry wrote: > >>>>> Terry, > >>>>> > >>>>> You were right, the error indeed seems to come from the message > >>>>> coalescing feature. If I turn it off using the "--mca > >>>>> btl_openib_use_message_coalescing 0", I'm not able to observe the > >>>>> "hdr->tag=0" error. > >>>>> > >>>>> There are some trac requests associated to very similar error > >>>>> (https://svn.open-mpi.org/trac/ompi/search?q=coalescing) but they are > >>>>> all closed (except https://svn.open-mpi.org/trac/ompi/ticket/2352 > >>>>> that might be related), aren't they ? What would you suggest Terry ? > >>>> > >>>> Interesting, though it looks to me like the segv in ticket 2352 would > >>>> have happened on the send side instead of the receive side like you > >>>> have. As to what to do next it would be really nice to have some > >>>> sort of reproducer that we can try and debug what is really going > >>>> on. The only other thing to do without a reproducer is to inspect > >>>> the code on the send side to figure out what might make it generate > >>>> at 0 hdr->tag. Or maybe instrument the send side to stop when it is > >>>> about ready to send a 0 hdr->tag and see if we can see how the code > >>>> got there. > >>>> > >>>> I might have some cycles to look at this Monday. > >>>> > >>>> --td > >>>> > >>>>> Eloi > >>>>> > >>>>> On Friday 24 September 2010 16:00:26 Terry Dontje wrote: > >>>>>> Eloi Gaudry wrote: > >>>>>>> Terry, > >>>>>>> > >>>>>>> No, I haven't tried any other values than P,65536,256,192,128 yet. > >>>>>>> > >>>>>>> The reason why is quite simple. I've been reading and reading again > >>>>>>> this thread to understand the btl_openib_receive_queues meaning and > >>>>>>> I can't figure out why the default values seem to induce the hdr- > >>>>>>> > >>>>>>>> tag=0 issue > >>>>>>>> (http://www.open-mpi.org/community/lists/users/2009/01/7808.php). > >>>>>> > >>>>>> Yeah, the size of the fragments and number of them really should not > >>>>>> cause this issue. So I too am a little perplexed about it. > >>>>>> > >>>>>>> Do you think that the default shared received queue parameters are > >>>>>>> erroneous for this specific Mellanox card ? Any help on finding the > >>>>>>> proper parameters would actually be much appreciated. > >>>>>> > >>>>>> I don't necessarily think it is the queue size for a specific card > >>>>>> but more so the handling of the queues by the BTL when using > >>>>>> certain sizes. At least that is one gut feel I have. > >>>>>> > >>>>>> In my mind the tag being 0 is either something below OMPI is > >>>>>> polluting the data fragment or OMPI's internal protocol is some how > >>>>>> getting messed up. I can imagine (no empirical data here) the > >>>>>> queue sizes could change how the OMPI protocol sets things up. > >>>>>> Another thing may be the coalescing feature in the openib BTL which > >>>>>> tries to gang multiple messages into one packet when resources are > >>>>>> running low. I can see where changing the queue sizes might > >>>>>> affect the coalescing. So, it might be interesting to turn off the > >>>>>> coalescing. You can do that by setting "--mca > >>>>>> btl_openib_use_message_coalescing 0" in your mpirun line. > >>>>>> > >>>>>> If that doesn't solve the issue then obviously there must be > >>>>>> something else going on :-). > >>>>>> > >>>>>> Note, the reason I am interested in this is I am seeing a similar > >>>>>> error condition (hdr->tag == 0) on a development system. Though my > >>>>>> failing case fails with np=8 using the connectivity test program > >>>>>> which is mainly point to point and there are not a significant > >>>>>> amount of data transfers going on either. > >>>>>> > >>>>>> --td > >>>>>> > >>>>>>> Eloi > >>>>>>> > >>>>>>> On Friday 24 September 2010 14:27:07 you wrote: > >>>>>>>> That is interesting. So does the number of processes affect your > >>>>>>>> runs any. The times I've seen hdr->tag be 0 usually has been due > >>>>>>>> to protocol issues. The tag should never be 0. Have you tried to > >>>>>>>> do other receive_queue settings other than the default and the one > >>>>>>>> you mention. > >>>>>>>> > >>>>>>>> I wonder if you did a combination of the two receive queues causes > >>>>>>>> a failure or not. Something like > >>>>>>>> > >>>>>>>> P,128,256,192,128:P,65536,256,192,128 > >>>>>>>> > >>>>>>>> I am wondering if it is the first queuing definition causing the > >>>>>>>> issue or possibly the SRQ defined in the default. > >>>>>>>> > >>>>>>>> --td > >>>>>>>> > >>>>>>>> Eloi Gaudry wrote: > >>>>>>>>> Hi Terry, > >>>>>>>>> > >>>>>>>>> The messages being send/received can be of any size, but the > >>>>>>>>> error seems to happen more often with small messages (as an int > >>>>>>>>> being broadcasted or allreduced). The failing communication > >>>>>>>>> differs from one run to another, but some spots are more likely > >>>>>>>>> to be failing than another. And as far as I know, there are > >>>>>>>>> always located next to a small message (an int being broadcasted > >>>>>>>>> for instance) communication. Other typical messages size are > >>>>>>>>> > >>>>>>>>>> 10k but can be very much larger. > >>>>>>>>> > >>>>>>>>> I've been checking the hca being used, its' from mellanox (with > >>>>>>>>> vendor_part_id=26428). There is no receive_queues parameters > >>>>>>>>> associated to it. > >>>>>>>>> > >>>>>>>>> $ cat share/openmpi/mca-btl-openib-device-params.ini as well: > >>>>>>>>> [...] > >>>>>>>>> > >>>>>>>>> # A.k.a. ConnectX > >>>>>>>>> [Mellanox Hermon] > >>>>>>>>> vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708,0x03ba,0x15b3 > >>>>>>>>> vendor_part_id = > >>>>>>>>> 25408,25418,25428,26418,26428,25448,26438,26448,26468,26478,264 > >>>>>>>>> 88 use_eager_rdma = 1 > >>>>>>>>> mtu = 2048 > >>>>>>>>> max_inline_data = 128 > >>>>>>>>> > >>>>>>>>> [..] > >>>>>>>>> > >>>>>>>>> $ ompi_info --param btl openib --parsable | grep receive_queues > >>>>>>>>> > >>>>>>>>> mca:btl:openib:param:btl_openib_receive_queues:value:P,128,256,1 > >>>>>>>>> 92 ,128 > >>>>>>>>> > >>>>>>>>> :S ,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32 > >>>>>>>>> > >>>>>>>>> mca:btl:openib:param:btl_openib_receive_queues:data_source:defau > >>>>>>>>> lt value > >>>>>>>>> mca:btl:openib:param:btl_openib_receive_queues:status:writable > >>>>>>>>> mca:btl:openib:param:btl_openib_receive_queues:help:Colon-delimi > >>>>>>>>> t ed, comma delimited list of receive queues: > >>>>>>>>> P,4096,8,6,4:P,32768,8,6,4 > >>>>>>>>> mca:btl:openib:param:btl_openib_receive_queues:deprecated:no > >>>>>>>>> > >>>>>>>>> I was wondering if these parameters (automatically computed at > >>>>>>>>> openib btl init for what I understood) were not incorrect in some > >>>>>>>>> way and I plugged some others values: "P,65536,256,192,128" > >>>>>>>>> (someone on the list used that values when encountering a > >>>>>>>>> different issue) . Since that, I haven't been able to observe the > >>>>>>>>> segfault (occuring as hrd->tag = 0 in > >>>>>>>>> btl_openib_component.c:2881) yet. > >>>>>>>>> > >>>>>>>>> Eloi > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> /home/pp_fr/st03230/EG/Softs/openmpi-custom-1.4.2/bin/ > >>>>>>>>> > >>>>>>>>> On Thursday 23 September 2010 23:33:48 Terry Dontje wrote: > >>>>>>>>>> Eloi, I am curious about your problem. Can you tell me what > >>>>>>>>>> size of job it is? Does it always fail on the same bcast, or > >>>>>>>>>> same process? > >>>>>>>>>> > >>>>>>>>>> Eloi Gaudry wrote: > >>>>>>>>>>> Hi Nysal, > >>>>>>>>>>> > >>>>>>>>>>> Thanks for your suggestions. > >>>>>>>>>>> > >>>>>>>>>>> I'm now able to get the checksum computed and redirected to > >>>>>>>>>>> stdout, thanks (I forgot the "-mca pml_base_verbose 5" option, > >>>>>>>>>>> you were right). I haven't been able to observe the > >>>>>>>>>>> segmentation fault (with hdr->tag=0) so far (when using pml > >>>>>>>>>>> csum) but I 'll let you know when I am. > >>>>>>>>>>> > >>>>>>>>>>> I've got two others question, which may be related to the error > >>>>>>>>>>> observed: > >>>>>>>>>>> > >>>>>>>>>>> 1/ does the maximum number of MPI_Comm that can be handled by > >>>>>>>>>>> OpenMPI somehow depends on the btl being used (i.e. if I'm > >>>>>>>>>>> using openib, may I use the same number of MPI_Comm object as > >>>>>>>>>>> with tcp) ? Is there something as MPI_COMM_MAX in OpenMPI ? > >>>>>>>>>>> > >>>>>>>>>>> 2/ the segfaults only appears during a mpi collective call, > >>>>>>>>>>> with very small message (one int is being broadcast, for > >>>>>>>>>>> instance) ; i followed the guidelines given at > >>>>>>>>>>> http://icl.cs.utk.edu/open- > >>>>>>>>>>> mpi/faq/?category=openfabrics#ib-small-message-rdma but the > >>>>>>>>>>> debug-build of OpenMPI asserts if I use a different min-size > >>>>>>>>>>> that 255. Anyway, if I deactivate eager_rdma, the segfaults > >>>>>>>>>>> remains. Does the openib btl handle very small message > >>>>>>>>>>> differently (even with eager_rdma > >>>>>>>>>>> deactivated) than tcp ? > >>>>>>>>>> > >>>>>>>>>> Others on the list does coalescing happen with non-eager_rdma? > >>>>>>>>>> If so then that would possibly be one difference between the > >>>>>>>>>> openib btl and tcp aside from the actual protocol used. > >>>>>>>>>> > >>>>>>>>>>> is there a way to make sure that large messages and small > >>>>>>>>>>> messages are handled the same way ? > >>>>>>>>>> > >>>>>>>>>> Do you mean so they all look like eager messages? How large of > >>>>>>>>>> messages are we talking about here 1K, 1M or 10M? > >>>>>>>>>> > >>>>>>>>>> --td > >>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Eloi > >>>>>>>>>>> > >>>>>>>>>>> On Friday 17 September 2010 17:57:17 Nysal Jan wrote: > >>>>>>>>>>>> Hi Eloi, > >>>>>>>>>>>> Create a debug build of OpenMPI (--enable-debug) and while > >>>>>>>>>>>> running with the csum PML add "-mca pml_base_verbose 5" to the > >>>>>>>>>>>> command line. This will print the checksum details for each > >>>>>>>>>>>> fragment sent over the wire. I'm guessing it didnt catch > >>>>>>>>>>>> anything because the BTL failed. The checksum verification is > >>>>>>>>>>>> done in the PML, which the BTL calls via a callback function. > >>>>>>>>>>>> In your case the PML callback is never called because the > >>>>>>>>>>>> hdr->tag is invalid. So enabling checksum tracing also might > >>>>>>>>>>>> not be of much use. Is it the first Bcast that fails or the > >>>>>>>>>>>> nth Bcast and what is the message size? I'm not sure what > >>>>>>>>>>>> could be the problem at this moment. I'm afraid you will have > >>>>>>>>>>>> to debug the BTL to find out more. > >>>>>>>>>>>> > >>>>>>>>>>>> --Nysal > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Sep 17, 2010 at 4:39 PM, Eloi Gaudry <e...@fft.be> wrote: > >>>>>>>>>>>>> Hi Nysal, > >>>>>>>>>>>>> > >>>>>>>>>>>>> thanks for your response. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've been unable so far to write a test case that could > >>>>>>>>>>>>> illustrate the hdr->tag=0 error. > >>>>>>>>>>>>> Actually, I'm only observing this issue when running an > >>>>>>>>>>>>> internode computation involving infiniband hardware from > >>>>>>>>>>>>> Mellanox (MT25418, ConnectX IB DDR, PCIe 2.0 > >>>>>>>>>>>>> 2.5GT/s, rev a0) with our time-domain software. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I checked, double-checked, and rechecked again every MPI use > >>>>>>>>>>>>> performed during a parallel computation and I couldn't find > >>>>>>>>>>>>> any error so far. The fact that the very > >>>>>>>>>>>>> same parallel computation run flawlessly when using tcp (and > >>>>>>>>>>>>> disabling openib support) might seem to indicate that the > >>>>>>>>>>>>> issue is somewhere located inside the > >>>>>>>>>>>>> openib btl or at the hardware/driver level. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've just used the "-mca pml csum" option and I haven't seen > >>>>>>>>>>>>> any related messages (when hdr->tag=0 and the segfaults > >>>>>>>>>>>>> occurs). Any suggestion ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards, > >>>>>>>>>>>>> Eloi > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Friday 17 September 2010 16:03:34 Nysal Jan wrote: > >>>>>>>>>>>>>> Hi Eloi, > >>>>>>>>>>>>>> Sorry for the delay in response. I haven't read the entire > >>>>>>>>>>>>>> email thread, but do you have a test case which can > >>>>>>>>>>>>>> reproduce this error? Without that it will be difficult to > >>>>>>>>>>>>>> nail down the cause. Just to clarify, I do not work for an > >>>>>>>>>>>>>> iwarp vendor. I can certainly try to reproduce it on an IB > >>>>>>>>>>>>>> system. There is also a PML called csum, you can use it via > >>>>>>>>>>>>>> "-mca pml csum", which will checksum the MPI messages and > >>>>>>>>>>>>>> verify it at the receiver side for any data > >>>>>>>>>>>>>> corruption. You can try using it to see if it is able > >>>>>>>>>>>>> > >>>>>>>>>>>>> to > >>>>>>>>>>>>> > >>>>>>>>>>>>>> catch anything. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Regards > >>>>>>>>>>>>>> --Nysal > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Sep 16, 2010 at 3:48 PM, Eloi Gaudry <e...@fft.be> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> Hi Nysal, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm sorry to intrrupt, but I was wondering if you had a > >>>>>>>>>>>>>>> chance to look > >>>>>>>>>>>>> > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> this error. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Eloi Gaudry > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Free Field Technologies > >>>>>>>>>>>>>>> Company Website: http://www.fft.be > >>>>>>>>>>>>>>> Company Phone: +32 10 487 959 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ---------- Forwarded message ---------- > >>>>>>>>>>>>>>> From: Eloi Gaudry <e...@fft.be> > >>>>>>>>>>>>>>> To: Open MPI Users <us...@open-mpi.org> > >>>>>>>>>>>>>>> Date: Wed, 15 Sep 2010 16:27:43 +0200 > >>>>>>>>>>>>>>> Subject: Re: [OMPI users] [openib] segfault when using > >>>>>>>>>>>>>>> openib btl Hi, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I was wondering if anybody got a chance to have a look at > >>>>>>>>>>>>>>> this issue. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wednesday 18 August 2010 09:16:26 Eloi Gaudry wrote: > >>>>>>>>>>>>>>>> Hi Jeff, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Please find enclosed the output (valgrind.out.gz) from > >>>>>>>>>>>>>>>> /opt/openmpi-debug-1.4.2/bin/orterun -np 2 --host > >>>>>>>>>>>>>>>> pbn11,pbn10 --mca > >>>>>>>>>>>>> > >>>>>>>>>>>>> btl > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>> openib,self --display-map --verbose --mca mpi_warn_on_fork > >>>>>>>>>>>>>>>> 0 --mca btl_openib_want_fork_support 0 -tag-output > >>>>>>>>>>>>>>>> /opt/valgrind-3.5.0/bin/valgrind --tool=memcheck > >>>>>>>>>>>>>>>> --suppressions=/opt/openmpi-debug-1.4.2/share/openmpi/open > >>>>>>>>>>>>>>>> mp i- valgrind.supp > >>>>>>>>>>>>>>>> --suppressions=./suppressions.python.supp > >>>>>>>>>>>>>>>> /opt/actran/bin/actranpy_mp ... > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Tuesday 17 August 2010 09:32:53 Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>> On Monday 16 August 2010 19:14:47 Jeff Squyres wrote: > >>>>>>>>>>>>>>>>>> On Aug 16, 2010, at 10:05 AM, Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>> I did run our application through valgrind but it > >>>>>>>>>>>>>>>>>>> couldn't find any "Invalid write": there is a bunch of > >>>>>>>>>>>>>>>>>>> "Invalid read" (I'm using > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1.4.2 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> with the suppression file), "Use of uninitialized > >>>>>>>>>>>>>>>>>>> bytes" and "Conditional jump depending on > >>>>>>>>>>>>>>>>>>> uninitialized bytes" in > >>>>>>>>>>>>> > >>>>>>>>>>>>> different > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> ompi > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> routines. Some of them are located in > >>>>>>>>>>>>>>>>>>> btl_openib_component.c. I'll send you an output of > >>>>>>>>>>>>>>>>>>> valgrind shortly. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> A lot of them in btl_openib_* are to be expected -- > >>>>>>>>>>>>>>>>>> OpenFabrics uses OS-bypass methods for some of its > >>>>>>>>>>>>>>>>>> memory, and therefore valgrind is unaware of them (and > >>>>>>>>>>>>>>>>>> therefore incorrectly marks them as > >>>>>>>>>>>>>>>>>> uninitialized). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> would it help if i use the upcoming 1.5 version of > >>>>>>>>>>>>>>>>> openmpi ? i > >>>>>>>>>>>>> > >>>>>>>>>>>>> read > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> a huge effort has been done to clean-up the valgrind > >>>>>>>>>>>>>>>>> output ? but maybe that this doesn't concern this btl > >>>>>>>>>>>>>>>>> (for the reasons you mentionned). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Another question, you said that the callback function > >>>>>>>>>>>>>>>>>>> pointer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> never be 0. But can the tag be null (hdr->tag) ? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The tag is not a pointer -- it's just an integer. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I was worrying that its value could not be null. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'll send a valgrind output soon (i need to build > >>>>>>>>>>>>>>>>> libpython without pymalloc first). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks for your help, > >>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On 16/08/2010 18:22, Jeff Squyres wrote: > >>>>>>>>>>>>>>>>>>>> Sorry for the delay in replying. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Odd; the values of the callback function pointer > >>>>>>>>>>>>>>>>>>>> should never > >>>>>>>>>>>>> > >>>>>>>>>>>>> be > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> 0. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This seems to suggest some kind of memory corruption > >>>>>>>>>>>>>>>>>>>> is occurring. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I don't know if it's possible, because the stack trace > >>>>>>>>>>>>>>>>>>>> looks like you're calling through python, but can you > >>>>>>>>>>>>>>>>>>>> run this application through valgrind, or some other > >>>>>>>>>>>>>>>>>>>> memory-checking debugger? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Aug 10, 2010, at 7:15 AM, Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> sorry, i just forgot to add the values of the > >>>>>>>>>>>>>>>>>>>>> function > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> parameters: > >>>>>>>>>>>>>>>>>>>>> (gdb) print reg->cbdata > >>>>>>>>>>>>>>>>>>>>> $1 = (void *) 0x0 > >>>>>>>>>>>>>>>>>>>>> (gdb) print openib_btl->super > >>>>>>>>>>>>>>>>>>>>> $2 = {btl_component = 0x2b341edd7380, btl_eager_limit > >>>>>>>>>>>>>>>>>>>>> = > >>>>>>>>>>>>> > >>>>>>>>>>>>> 12288, > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> btl_rndv_eager_limit = 12288, btl_max_send_size = > >>>>>>>>>>>>>>>>>>>>> 65536, btl_rdma_pipeline_send_length = 1048576, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> btl_rdma_pipeline_frag_size = 1048576, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> btl_min_rdma_pipeline_size > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> = 1060864, btl_exclusivity = 1024, btl_latency = > >>>>>>>>>>>>>>>>>>>>> 10, btl_bandwidth = 800, btl_flags = 310, > >>>>>>>>>>>>>>>>>>>>> btl_add_procs = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb8ee47<mca_btl_openib_add_procs>, > >>>>>>>>>>>>>>>>>>>>> btl_del_procs = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb90156<mca_btl_openib_del_procs>, > >>>>>>>>>>>>>>>>>>>>> btl_register = 0, btl_finalize = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb93186<mca_btl_openib_finalize>, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> btl_alloc > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> = 0x2b341eb90a3e<mca_btl_openib_alloc>, btl_free = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb91400<mca_btl_openib_free>, > >>>>>>>>>>>>>>>>>>>>> btl_prepare_src = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb91813<mca_btl_openib_prepare_src>, > >>>>>>>>>>>>>>>>>>>>> btl_prepare_dst > >>>>>>>>>>>>> > >>>>>>>>>>>>> = > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb91f2e<mca_btl_openib_prepare_dst>, > >>>>>>>>>>>>>>>>>>>>> btl_send = 0x2b341eb94517<mca_btl_openib_send>, > >>>>>>>>>>>>>>>>>>>>> btl_sendi = 0x2b341eb9340d<mca_btl_openib_sendi>, > >>>>>>>>>>>>>>>>>>>>> btl_put = 0x2b341eb94660<mca_btl_openib_put>, > >>>>>>>>>>>>>>>>>>>>> btl_get = 0x2b341eb94c4e<mca_btl_openib_get>, > >>>>>>>>>>>>>>>>>>>>> btl_dump = 0x2b341acd45cb<mca_btl_base_dump>, > >>>>>>>>>>>>>>>>>>>>> btl_mpool = 0xf3f4110, btl_register_error = > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb90565<mca_btl_openib_register_error_cb>, > >>>>>>>>>>>>>>>>>>>>> btl_ft_event > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> = > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 0x2b341eb952e7<mca_btl_openib_ft_event>} > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> (gdb) print hdr->tag > >>>>>>>>>>>>>>>>>>>>> $3 = 0 '\0' > >>>>>>>>>>>>>>>>>>>>> (gdb) print des > >>>>>>>>>>>>>>>>>>>>> $4 = (mca_btl_base_descriptor_t *) 0xf4a6700 > >>>>>>>>>>>>>>>>>>>>> (gdb) print reg->cbfunc > >>>>>>>>>>>>>>>>>>>>> $5 = (mca_btl_base_module_recv_cb_fn_t) 0 > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Tuesday 10 August 2010 16:04:08 Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Here is the output of a core file generated during a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> segmentation > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> fault observed during a collective call (using > >>>>>>>>>>>>>>>>>>>>>> openib): > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> #0 0x0000000000000000 in ?? () > >>>>>>>>>>>>>>>>>>>>>> (gdb) where > >>>>>>>>>>>>>>>>>>>>>> #0 0x0000000000000000 in ?? () > >>>>>>>>>>>>>>>>>>>>>> #1 0x00002aedbc4e05f4 in btl_openib_handle_incoming > >>>>>>>>>>>>>>>>>>>>>> (openib_btl=0x1902f9b0, ep=0x1908a1c0, > >>>>>>>>>>>>>>>>>>>>>> frag=0x190d9700, byte_len=18) at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:2881 #2 0x00002aedbc4e25e2 in > >>>>>>>>>>>>>>>>>>>>>> handle_wc (device=0x19024ac0, cq=0, > >>>>>>>>>>>>>>>>>>>>>> wc=0x7ffff279ce90) at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:3178 #3 0x00002aedbc4e2e9d > >>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> poll_device > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> (device=0x19024ac0, count=2) at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:3318 > >>>>>>>>>>>>> > >>>>>>>>>>>>> #4 > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 0x00002aedbc4e34b8 in progress_one_device > >>>>>>>>>>>>> > >>>>>>>>>>>>> (device=0x19024ac0) > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> at btl_openib_component.c:3426 #5 > >>>>>>>>>>>>>>>>>>>>>> 0x00002aedbc4e3561 in btl_openib_component_progress > >>>>>>>>>>>>>>>>>>>>>> () at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:3451 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> #6 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 0x00002aedb8b22ab8 in opal_progress () at > >>>>>>>>>>>>>>>>>>>>>> runtime/opal_progress.c:207 #7 0x00002aedb859f497 in > >>>>>>>>>>>>>>>>>>>>>> opal_condition_wait (c=0x2aedb888ccc0, > >>>>>>>>>>>>>>>>>>>>>> m=0x2aedb888cd20) at ../opal/threads/condition.h:99 > >>>>>>>>>>>>>>>>>>>>>> #8 > >>>>>>>>>>>>>>>>>>>>>> 0x00002aedb859fa31 in ompi_request_default_wait_all > >>>>>>>>>>>>> > >>>>>>>>>>>>> (count=2, > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> requests=0x7ffff279d0e0, statuses=0x0) at > >>>>>>>>>>>>>>>>>>>>>> request/req_wait.c:262 #9 0x00002aedbd7559ad in > >>>>>>>>>>>>>>>>>>>>>> ompi_coll_tuned_allreduce_intra_recursivedoubling > >>>>>>>>>>>>>>>>>>>>>> (sbuf=0x7ffff279d444, rbuf=0x7ffff279d440, count=1, > >>>>>>>>>>>>>>>>>>>>>> dtype=0x6788220, op=0x6787a20, > >>>>>>>>>>>>>>>>>>>>>> comm=0x19d81ff0, module=0x19d82b20) at > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> coll_tuned_allreduce.c:223 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> #10 0x00002aedbd7514f7 in > >>>>>>>>>>>>>>>>>>>>>> ompi_coll_tuned_allreduce_intra_dec_fixed > >>>>>>>>>>>>>>>>>>>>>> (sbuf=0x7ffff279d444, rbuf=0x7ffff279d440, count=1, > >>>>>>>>>>>>>>>>>>>>>> dtype=0x6788220, op=0x6787a20, comm=0x19d81ff0, > >>>>>>>>>>>>>>>>>>>>>> module=0x19d82b20) at > >>>>>>>>>>>>>>>>>>>>>> coll_tuned_decision_fixed.c:63 > >>>>>>>>>>>>>>>>>>>>>> #11 0x00002aedb85c7792 in PMPI_Allreduce > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> (sendbuf=0x7ffff279d444, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> recvbuf=0x7ffff279d440, count=1, datatype=0x6788220, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> op=0x6787a20, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> comm=0x19d81ff0) at pallreduce.c:102 #12 > >>>>>>>>>>>>>>>>>>>>>> 0x0000000004387dbf > >>>>>>>>>>>>> > >>>>>>>>>>>>> in > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> FEMTown::MPI::Allreduce (sendbuf=0x7ffff279d444, > >>>>>>>>>>>>>>>>>>>>>> recvbuf=0x7ffff279d440, count=1, datatype=0x6788220, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> op=0x6787a20, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> comm=0x19d81ff0) at stubs.cpp:626 #13 > >>>>>>>>>>>>>>>>>>>>>> 0x0000000004058be8 in FEMTown::Domain::align (itf= > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> {<FEMTown::Boost::shared_base_ptr<FEMTown::Domain::Int > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> er fa ce>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> = {_vptr.shared_base_ptr = 0x7ffff279d620, ptr_ = > >>>>>>>>>>>>>>>>>>>>>> {px = 0x199942a4, pn = {pi_ = 0x6}}},<No data > >>>>>>>>>>>>>>>>>>>>>> fields>}) at interface.cpp:371 #14 > >>>>>>>>>>>>>>>>>>>>>> 0x00000000040cb858 in > >>>>>>>>>>>>>>>>>>>>>> FEMTown::Field::detail::align_itfs_and_neighbhors > >>>>>>>>>>>>>>>>>>>>>> (dim=2, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> set={px > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> = 0x7ffff279d780, pn = {pi_ = 0x2f279d640}}, > >>>>>>>>>>>>>>>>>>>>>> check_info=@0x7ffff279d7f0) at check.cpp:63 #15 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 0x00000000040cbfa8 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> in FEMTown::Field::align_elements (set={px = > >>>>>>>>>>>>>>>>>>>>>> 0x7ffff279d950, pn > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> = > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> {pi_ = 0x66e08d0}}, check_info=@0x7ffff279d7f0) at > >>>>>>>>>>>>>>>>>>>>>> check.cpp:159 #16 0x00000000039acdd4 in > >>>>>>>>>>>>>>>>>>>>>> PyField_align_elements (self=0x0, > >>>>>>>>>>>>>>>>>>>>>> args=0x2aaab0765050, kwds=0x19d2e950) at > >>>>>>>>>>>>>>>>>>>>>> check.cpp:31 #17 > >>>>>>>>>>>>>>>>>>>>>> 0x0000000001fbf76d in > >>>>>>>>>>>>>>>>>>>>>> FEMTown::Main::ExErrCatch<_object* (*)(_object*, > >>>>>>>>>>>>>>>>>>>>>> _object*, _object*)>::exec<_object> > >>>>>>>>>>>>>>>>>>>>>> (this=0x7ffff279dc20, s=0x0, po1=0x2aaab0765050, > >>>>>>>>>>>>>>>>>>>>>> po2=0x19d2e950) at > >>>>>>>>>>>>>>>>>>>>>> /home/qa/svntop/femtown/modules/main/py/exception.hp > >>>>>>>>>>>>>>>>>>>>>> p: 463 > >>>>>>>>>>>>> > >>>>>>>>>>>>> #18 > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 0x00000000039acc82 in PyField_align_elements_ewrap > >>>>>>>>>>>>> > >>>>>>>>>>>>> (self=0x0, > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> args=0x2aaab0765050, kwds=0x19d2e950) at > >>>>>>>>>>>>>>>>>>>>>> check.cpp:39 #19 0x00000000044093a0 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalFrameEx (f=0x19b52e90, throwflag=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>) at Python/ceval.c:3921 #20 > >>>>>>>>>>>>>>>>>>>>>> 0x000000000440aae9 in PyEval_EvalCodeEx > >>>>>>>>>>>>>>>>>>>>>> (co=0x2aaab754ad50, globals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> locals=<value optimized out>, args=0x3, argcount=1, > >>>>>>>>>>>>>>>>>>>>>> kws=0x19ace4a0, kwcount=2, > >>>>>>>>>>>>>>>>>>>>>> defs=0x2aaab75e4800, defcount=2, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 > >>>>>>>>>>>>>>>>>>>>>> #21 0x0000000004408f58 in PyEval_EvalFrameEx > >>>>>>>>>>>>>>>>>>>>>> (f=0x19ace2d0, throwflag=<value optimized out>) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:3802 #22 0x000000000440aae9 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab7550120, globals=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> args=0x7, argcount=1, kws=0x19acc418, kwcount=3, > >>>>>>>>>>>>>>>>>>>>>> defs=0x2aaab759e958, defcount=6, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 > >>>>>>>>>>>>>>>>>>>>>> #23 0x0000000004408f58 in PyEval_EvalFrameEx > >>>>>>>>>>>>>>>>>>>>>> (f=0x19acc1c0, throwflag=<value optimized out>) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:3802 #24 0x000000000440aae9 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab8b5e738, globals=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> args=0x6, argcount=1, kws=0x19abd328, kwcount=5, > >>>>>>>>>>>>>>>>>>>>>> defs=0x2aaab891b7e8, defcount=3, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 > >>>>>>>>>>>>>>>>>>>>>> #25 0x0000000004408f58 in PyEval_EvalFrameEx > >>>>>>>>>>>>>>>>>>>>>> (f=0x19abcea0, throwflag=<value optimized out>) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:3802 #26 0x000000000440aae9 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab3eb4198, globals=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> args=0xb, argcount=1, kws=0x19a89df0, kwcount=10, > >>>>>>>>>>>>>>>>>>>>>> defs=0x0, defcount=0, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 #27 0x0000000004408f58 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalFrameEx > >>>>>>>>>>>>>>>>>>>>>> (f=0x19a89c40, throwflag=<value optimized out>) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:3802 #28 0x000000000440aae9 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab3eb4288, globals=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> args=0x1, argcount=0, kws=0x19a89330, kwcount=0, > >>>>>>>>>>>>>>>>>>>>>> defs=0x2aaab8b66668, defcount=1, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 > >>>>>>>>>>>>>>>>>>>>>> #29 0x0000000004408f58 in PyEval_EvalFrameEx > >>>>>>>>>>>>>>>>>>>>>> (f=0x19a891b0, throwflag=<value optimized out>) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:3802 #30 0x000000000440aae9 in > >>>>>>>>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab8b6a738, globals=<value > >>>>>>>>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, > >>>>>>>>>>>>>>>>>>>>>> defcount=0, closure=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:2968 > >>>>>>>>>>>>>>>>>>>>>> #31 0x000000000440ac02 in PyEval_EvalCode > >>>>>>>>>>>>>>>>>>>>>> (co=0x1902f9b0, globals=0x0, locals=0x190d9700) at > >>>>>>>>>>>>>>>>>>>>>> Python/ceval.c:522 #32 0x000000000442853c in > >>>>>>>>>>>>>>>>>>>>>> PyRun_StringFlags (str=0x192fd3d8 > >>>>>>>>>>>>>>>>>>>>>> "DIRECT.Actran.main()", start=<value optimized out>, > >>>>>>>>>>>>>>>>>>>>>> globals=0x192213d0, locals=0x192213d0, flags=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/pythonrun.c:1335 #33 0x0000000004429690 in > >>>>>>>>>>>>>>>>>>>>>> PyRun_SimpleStringFlags (command=0x192fd3d8 > >>>>>>>>>>>>>>>>>>>>>> "DIRECT.Actran.main()", flags=0x0) at > >>>>>>>>>>>>>>>>>>>>>> Python/pythonrun.c:957 #34 0x0000000001fa1cf9 in > >>>>>>>>>>>>>>>>>>>>>> FEMTown::Python::FEMPy::run_application > >>>>>>>>>>>>> > >>>>>>>>>>>>> (this=0x7ffff279f650) > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> at fempy.cpp:873 #35 0x000000000434ce99 in > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> FEMTown::Main::Batch::run > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> (this=0x7ffff279f650) at batch.cpp:374 #36 > >>>>>>>>>>>>> > >>>>>>>>>>>>> 0x0000000001f9aa25 > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> in main (argc=8, argv=0x7ffff279fa48) at main.cpp:10 > >>>>>>>>>>>>>>>>>>>>>> (gdb) f 1 #1 0x00002aedbc4e05f4 in > >>>>>>>>>>>>>>>>>>>>>> btl_openib_handle_incoming (openib_btl=0x1902f9b0, > >>>>>>>>>>>>>>>>>>>>>> ep=0x1908a1c0, frag=0x190d9700, byte_len=18) at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:2881 2881 reg->cbfunc( > >>>>>>>>>>>>>>>>>>>>>> &openib_btl->super, hdr->tag, des, reg->cbdata > >>>>>>>>>>>>> > >>>>>>>>>>>>> ); > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Current language: auto; currently c > >>>>>>>>>>>>>>>>>>>>>> (gdb) > >>>>>>>>>>>>>>>>>>>>>> #1 0x00002aedbc4e05f4 in btl_openib_handle_incoming > >>>>>>>>>>>>>>>>>>>>>> (openib_btl=0x1902f9b0, ep=0x1908a1c0, > >>>>>>>>>>>>>>>>>>>>>> frag=0x190d9700, byte_len=18) at > >>>>>>>>>>>>>>>>>>>>>> btl_openib_component.c:2881 2881 reg->cbfunc( > >>>>>>>>>>>>>>>>>>>>>> &openib_btl->super, hdr->tag, des, reg->cbdata > >>>>>>>>>>>>> > >>>>>>>>>>>>> ); > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> (gdb) l 2876 > >>>>>>>>>>>>>>>>>>>>>> 2877 if(OPAL_LIKELY(!(is_credit_msg = > >>>>>>>>>>>>>>>>>>>>>> is_credit_message(frag)))) { 2878 /* call > >>>>>>>>>>>>>>>>>>>>>> registered callback */ > >>>>>>>>>>>>>>>>>>>>>> 2879 mca_btl_active_message_callback_t* > >>>>>>>>>>>>>>>>>>>>>> reg; 2880 reg = > >>>>>>>>>>>>>>>>>>>>>> mca_btl_base_active_message_trigger + hdr->tag; 2881 > >>>>>>>>>>>>>>>>>>>>>> reg->cbfunc(&openib_btl->super, hdr->tag, des, > >>>>>>>>>>>>>>>>>>>>>> reg->cbdata ); 2882 > >>>>>>>>>>>>>>>>>>>>>> if(MCA_BTL_OPENIB_RDMA_FRAG(frag)) { 2883 > >>>>>>>>>>>>>>>>>>>>>> cqp > >>>>>>>>>>>>> > >>>>>>>>>>>>> = > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> (hdr->credits>> 11)& 0x0f; > >>>>>>>>>>>>>>>>>>>>>> 2884 hdr->credits&= 0x87ff; > >>>>>>>>>>>>>>>>>>>>>> 2885 } else { > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Friday 16 July 2010 16:01:02 Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>>>> Hi Edgar, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> The only difference I could observed was that the > >>>>>>>>>>>>>>>>>>>>>>> segmentation fault appeared sometimes later during > >>>>>>>>>>>>>>>>>>>>>>> the parallel computation. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I'm running out of idea here. I wish I could use > >>>>>>>>>>>>>>>>>>>>>>> the "--mca > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> coll > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> tuned" with "--mca self,sm,tcp" so that I could > >>>>>>>>>>>>>>>>>>>>>>> check that the issue is not somehow limited to the > >>>>>>>>>>>>>>>>>>>>>>> tuned collective routines. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Thursday 15 July 2010 17:24:24 Edgar Gabriel wrote: > >>>>>>>>>>>>>>>>>>>>>>>> On 7/15/2010 10:18 AM, Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> hi edgar, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> thanks for the tips, I'm gonna try this option as > >>>>>>>>>>>>>>>>>>>>>>>>> well. > >>>>>>>>>>>>> > >>>>>>>>>>>>> the > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> segmentation fault i'm observing always happened > >>>>>>>>>>>>>>>>>>>>>>>>> during a collective communication indeed... does > >>>>>>>>>>>>>>>>>>>>>>>>> it basically > >>>>>>>>>>>>> > >>>>>>>>>>>>> switch > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> collective communication to basic mode, right ? > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> sorry for my ignorance, but what's a NCA ? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> sorry, I meant to type HCA (InifinBand networking > >>>>>>>>>>>>>>>>>>>>>>>> card) > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>> Edgar > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> thanks, > >>>>>>>>>>>>>>>>>>>>>>>>> éloi > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thursday 15 July 2010 16:20:54 Edgar Gabriel > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> you could try first to use the algorithms in the > >>>>>>>>>>>>>>>>>>>>>>>>>> basic > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> module, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> e.g. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> mpirun -np x --mca coll basic ./mytest > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> and see whether this makes a difference. I used > >>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>> > >>>>>>>>>>>>> observe > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> sometimes a (similar ?) problem in the openib > >>>>>>>>>>>>>>>>>>>>>>>>>> btl triggered from the tuned collective > >>>>>>>>>>>>>>>>>>>>>>>>>> component, in cases where the ofed libraries > >>>>>>>>>>>>>>>>>>>>>>>>>> were installed but no NCA was found on a node. > >>>>>>>>>>>>>>>>>>>>>>>>>> It used to work however with the basic > >>>>>>>>>>>>>>>>>>>>>>>>>> component. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>>>> Edgar > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 7/15/2010 3:08 AM, Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Rolf, > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> unfortunately, i couldn't get rid of that > >>>>>>>>>>>>>>>>>>>>>>>>>>> annoying segmentation fault when selecting > >>>>>>>>>>>>>>>>>>>>>>>>>>> another bcast algorithm. i'm now going to > >>>>>>>>>>>>>>>>>>>>>>>>>>> replace MPI_Bcast with a naive > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation (using MPI_Send and MPI_Recv) > >>>>>>>>>>>>>>>>>>>>>>>>>>> and see if > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> helps. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>> éloi > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wednesday 14 July 2010 10:59:53 Eloi Gaudry > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Rolf, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for your input. You're right, I miss > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the coll_tuned_use_dynamic_rules option. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll check if I the segmentation fault > >>>>>>>>>>>>>>>>>>>>>>>>>>>> disappears when > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> using > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the basic bcast linear algorithm using the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> proper command line you provided. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday 13 July 2010 20:39:59 Rolf > >>>>>>>>>>>>>>>>>>>>>>>>>>>> vandeVaart > >>>>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Eloi: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To select the different bcast algorithms, you > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to add an extra mca parameter that tells > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the library to use dynamic selection. --mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coll_tuned_use_dynamic_rules 1 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> One way to make sure you are typing this in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> correctly is > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use it with ompi_info. Do the following: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ompi_info -mca coll_tuned_use_dynamic_rules 1 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --param > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> coll > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> You should see lots of output with all the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different algorithms that can be selected for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the various collectives. Therefore, you need > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --mca coll_tuned_use_dynamic_rules 1 --mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coll_tuned_bcast_algorithm 1 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rolf > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 07/13/10 11:28, Eloi Gaudry wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've found that "--mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coll_tuned_bcast_algorithm 1" allowed to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to the basic linear algorithm. Anyway > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whatever the algorithm used, the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation fault remains. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does anyone could give some advice on ways > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>> > >>>>>>>>>>>>> diagnose > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue I'm facing ? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Monday 12 July 2010 10:53:58 Eloi Gaudry > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm focusing on the MPI_Bcast routine that > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems to randomly segfault when using the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openib btl. I'd > >>>>>>>>>>>>> > >>>>>>>>>>>>> like > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know if there is any way to make OpenMPI > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to > >>>>>>>>>>>>> > >>>>>>>>>>>>> a > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different algorithm than the default one > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> being selected for MPI_Bcast. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your help, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Friday 02 July 2010 11:06:52 Eloi Gaudry > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm observing a random segmentation fault > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during > >>>>>>>>>>>>> > >>>>>>>>>>>>> an > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internode parallel computation involving > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>> > >>>>>>>>>>>>> openib > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> btl > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and OpenMPI-1.4.2 (the same issue can be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> observed with OpenMPI-1.3.3). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mpirun (Open MPI) 1.4.2 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Report bugs to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/help/ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] *** Process received > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signal *** [pbn08:02624] Signal: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Segmentation fault (11) [pbn08:02624] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Signal code: Address not mapped > >>>>>>>>>>>>> > >>>>>>>>>>>>> (1) > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] Failing at address: (nil) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] [ 0] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /lib64/libpthread.so.0 [0x349540e4c0] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] *** End of error > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> message > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sh: line 1: 2624 Segmentation fault > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> \/share\/hpc3\/actran_suite\/Actran_11\.0\.rc2\.41872\/R > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ed Ha tE L\ -5 \/ x 86 _6 4\ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /bin\/actranpy_mp > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> '--apl=/share/hpc3/actran_suite/Actran_11.0.rc2.41872/Re > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dH at EL -5 /x 86 _ 64 /A c > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tran_11.0.rc2.41872' > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> '--inputfile=/work/st25652/LSF_130073_0_47696_0/Case1_3D > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re al _m 4_ n2 .d a t' > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> '--scratch=/scratch/st25652/LSF_130073_0_47696_0/scratch > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ' '--mem=3200' '--threads=1' > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '--errorlevel=FATAL' '--t_max=0.1' > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '--parallel=domain' > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If I choose not to use the openib btl (by > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using --mca btl self,sm,tcp on the command > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line, for instance), I don't encounter any > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem and the parallel computation runs > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flawlessly. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to get some help to be able: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - to diagnose the issue I'm facing with > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the openib btl - understand why this > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue is observed only when > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> using > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the openib btl and not when using > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self,sm,tcp > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Any help would be very much appreciated. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The outputs of ompi_info and the configure > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scripts of OpenMPI are enclosed to this > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> email, and some > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the infiniband drivers as well. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the command line used when > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> launching a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> parallel > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> computation > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using infiniband: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path_to_openmpi/bin/mpirun -np > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $NPROCESS --hostfile host.list --mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btl openib,sm,self,tcp --display-map > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose --version --mca mpi_warn_on_fork > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0 --mca btl_openib_want_fork_support 0 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [...] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and the command line used if not using > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> infiniband: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path_to_openmpi/bin/mpirun -np > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $NPROCESS --hostfile host.list --mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btl self,sm,tcp --display-map --verbose > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --version > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> --mca > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mpi_warn_on_fork 0 --mca > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btl_openib_want_fork_support > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 0 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [...] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eloi > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ____________________________________________ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> __ _ > >>>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> users mailing list > >>>>>>>>>>> us...@open-mpi.org > >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users -- Eloi Gaudry Free Field Technologies Company Website: http://www.fft.be Company Phone: +32 10 487 959
