FWIW: that's exactly how we do it in ORTE
On Aug 4, 2014, at 10:25 PM, Gilles Gouaillardet
<gilles.gouaillar...@iferc.org> wrote:
> George,
>
> i confirm there was a problem when running on an heterogeneous cluster,
> this is now fixed in r32425.
>
> i am not convinced i chose the most elegant way to achieve the desired result
> ...
> could you please double check this commit ?
>
> Thanks,
>
> Gilles
>
> On 2014/08/02 0:14, George Bosilca wrote:
>> Gilles,
>>
>> The design of the BTL move was to let the opal_process_name_t be agnostic to
>> what is stored inside, and all accesses should be done through the provided
>> accessors. Thus, big endian or little endian doesn’t make a difference, as
>> long as everything goes through the accessors.
>>
>> I’m skeptical about the support of heterogeneous environments in the current
>> code, so I didn’t pay much attention to handling the case in the TCP BTL.
>> But in case we do care it is enough to make the 2 macros point to something
>> meaningful instead of being empty (bswap_64 or something).
>>
>> George.
>>
>> On Aug 1, 2014, at 06:52 , Gilles Gouaillardet
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>>> George and Ralph,
>>>
>>> i am very confused whether there is an issue or not.
>>>
>>>
>>> anyway, today Paul and i ran basic tests on big endian machines and did not
>>> face any issue related to big endianness.
>>>
>>> so i made my homework, digged into the code, and basically,
>>> opal_process_name_t is used as an orte_process_name_t.
>>> for example, in ompi_proc_init :
>>>
>>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->jobid =
>>> OMPI_PROC_MY_NAME->jobid;
>>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->vpid = i;
>>>
>>> and with
>>>
>>> #define OMPI_CAST_ORTE_NAME(a) ((orte_process_name_t*)(a))
>>>
>>> so as long as an opal_process_name_t is used as an orte_process_name_t,
>>> there is no problem,
>>> regardless the endianness of the homogenous cluster we are running on.
>>>
>>> for the sake of readability (and for being pedantic too ;-) ) in r32357,
>>> &proc_temp->super.proc_name
>>> could be replaced with
>>> OMPI_CAST_ORTE_NAME(&proc_temp->super.proc_name)
>>>
>>>
>>>
>>> That being said, in btl/tcp, i noticed :
>>>
>>> in mca_btl_tcp_component_recv_handler :
>>>
>>> opal_process_name_t guid;
>>> [...]
>>> /* recv the process identifier */
>>> retval = recv(sd, (char *)&guid, sizeof(guid), 0);
>>> if(retval != sizeof(guid)) {
>>> CLOSE_THE_SOCKET(sd);
>>> return;
>>> }
>>> OPAL_PROCESS_NAME_NTOH(guid);
>>>
>>> and in mca_btl_tcp_endpoint_send_connect_ack :
>>>
>>> /* send process identifier to remote endpoint */
>>> opal_process_name_t guid = btl_proc->proc_opal->proc_name;
>>>
>>> OPAL_PROCESS_NAME_HTON(guid);
>>> if(mca_btl_tcp_endpoint_send_blocking(btl_endpoint, &guid,
>>> sizeof(guid)) !=
>>>
>>> and with
>>>
>>> #define OPAL_PROCESS_NAME_NTOH(guid)
>>> #define OPAL_PROCESS_NAME_HTON(guid)
>>>
>>>
>>> i had no time yet to test yet, but for now, i can only suspect :
>>> - there will be an issue with the tcp btl on an heterogeneous cluster
>>> - for this case, the fix is to have a different version of the
>>> OPAL_PROCESS_NAME_xTOy
>>> on little endian arch if heterogeneous mode is supported.
>>>
>>>
>>>
>>> does that make sense ?
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2014/07/31 1:29, George Bosilca wrote:
>>>> The underlying structure changed, so a little bit of fiddling is normal.
>>>> Instead of using a field in the ompi_proc_t you are now using a field down
>>>> in opal_proc_t, a field that simply cannot have the same type as before
>>>> (orte_process_name_t).
>>>>
>>>> George.
>>>>
>>>>
>>>>
>>>> On Wed, Jul 30, 2014 at 12:19 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>
>>>>> George - my point was that we regularly tested using the method in that
>>>>> routine, and now we have to do something a little different. So it is an
>>>>> "issue" in that we have to make changes across the code base to ensure we
>>>>> do things the "new" way, that's all
>>>>>
>>>>> On Jul 30, 2014, at 9:17 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>
>>>>> No, this is not going to be an issue if the opal_identifier_t is used
>>>>> correctly (aka only via the exposed accessors).
>>>>>
>>>>> George.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 30, 2014 at 12:09 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>
>>>>>> Yeah, my fix won't work for big endian machines - this is going to be an
>>>>>> issue across the code base now, so we'll have to troll and fix it. I was
>>>>>> doing the minimal change required to fix the trunk in the meantime.
>>>>>>
>>>>>> On Jul 30, 2014, at 9:06 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>>
>>>>>> Yes. opal_process_name_t has basically no meaning by itself, it is a 64
>>>>>> bits storage location used by the upper layer to save some local key that
>>>>>> can be later used to extract information. Calling the OPAL level compare
>>>>>> function might be a better fit there.
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 11:50 AM, Gilles Gouaillardet <
>>>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>>>
>>>>>>> Ralph,
>>>>>>>
>>>>>>> was it really that simple ?
>>>>>>>
>>>>>>> proc_temp->super.proc_name has type opal_process_name_t :
>>>>>>> typedef opal_identifier_t opal_process_name_t;
>>>>>>> typedef uint64_t opal_identifier_t;
>>>>>>>
>>>>>>> *but*
>>>>>>>
>>>>>>> item_ptr->peer has type orte_process_name_t :
>>>>>>> struct orte_process_name_t {
>>>>>>> orte_jobid_t jobid;
>>>>>>> orte_vpid_t vpid;
>>>>>>> };
>>>>>>>
>>>>>>> bottom line, is r32357 still valid on a big endian arch ?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Gilles
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 30, 2014 at 11:49 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I just fixed this one - all that was required was an ampersand as the
>>>>>>>> name was being passed into the function instead of a pointer to the
>>>>>>>> name
>>>>>>>>
>>>>>>>> r32357
>>>>>>>>
>>>>>>>> On Jul 30, 2014, at 7:43 AM, Gilles GOUAILLARDET <
>>>>>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Rolf,
>>>>>>>>
>>>>>>>> r32353 can be seen as a suspect...
>>>>>>>> Even if it is correct, it might have exposed the bug discussed in #4815
>>>>>>>> even more (e.g. we hit the bug 100% after the fix)
>>>>>>>>
>>>>>>>> does the attached patch to #4815 fixes the problem ?
>>>>>>>>
>>>>>>>> If yes, and if you see this issue as a showstopper, feel free to commit
>>>>>>>> it and drop a note to #4815
>>>>>>>> ( I am afk until tomorrow)
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Gilles
>>>>>>>>
>>>>>>>> Rolf vandeVaart <rvandeva...@nvidia.com> wrote:
>>>>>>>>
>>>>>>>> Just an FYI that my trunk version (r32355) does not work at all anymore
>>>>>>>> if I do not include "--mca coll ^ml". Here is a stack trace from the
>>>>>>>> ibm/pt2pt/send test running on a single node.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> (gdb) where
>>>>>>>>
>>>>>>>> #0 0x00007f6c0d1321d0 in ?? ()
>>>>>>>>
>>>>>>>> #1 <signal handler called>
>>>>>>>>
>>>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>>>> ../../orte/util/name_fns.c:522
>>>>>>>>
>>>>>>>> #3 0x00007f6c0bea17be in bcol_basesmuma_smcm_allgather_connection
>>>>>>>> (sm_bcol_module=0x7f6bf3b68040, module=0xb3d200,
>>>>>>>> peer_list=0x7f6c0c0a6748,
>>>>>>>> back_files=0x7f6bf3ffd6c8,
>>>>>>>>
>>>>>>>> comm=0x6037a0, input=..., base_fname=0x7f6c0bea2606
>>>>>>>> "sm_payload_mem_", map_all=false) at
>>>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_smcm.c:237
>>>>>>>>
>>>>>>>> #4 0x00007f6c0be98307 in bcol_basesmuma_bank_init_opti
>>>>>>>> (payload_block=0xbc0f60, data_offset=64, bcol_module=0x7f6bf3b68040,
>>>>>>>> reg_data=0xba28c0)
>>>>>>>>
>>>>>>>> at
>>>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_buf_mgmt.c:302
>>>>>>>>
>>>>>>>> #5 0x00007f6c0cced386 in mca_coll_ml_register_bcols
>>>>>>>> (ml_module=0xba5c40) at
>>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:510
>>>>>>>>
>>>>>>>> #6 0x00007f6c0cced68f in ml_module_memory_initialization
>>>>>>>> (ml_module=0xba5c40) at
>>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:558
>>>>>>>>
>>>>>>>> #7 0x00007f6c0ccf06b1 in ml_discover_hierarchy (ml_module=0xba5c40) at
>>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:1539
>>>>>>>>
>>>>>>>> #8 0x00007f6c0ccf4e0b in mca_coll_ml_comm_query (comm=0x6037a0,
>>>>>>>> priority=0x7fffe7991b58) at
>>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:2963
>>>>>>>>
>>>>>>>> #9 0x00007f6c18cc5b09 in query_2_0_0 (component=0x7f6c0cf50940,
>>>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>>>
>>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:372
>>>>>>>>
>>>>>>>> #10 0x00007f6c18cc5ac8 in query (component=0x7f6c0cf50940,
>>>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>>>
>>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:355
>>>>>>>>
>>>>>>>> #11 0x00007f6c18cc59d2 in check_one_component (comm=0x6037a0,
>>>>>>>> component=0x7f6c0cf50940, module=0x7fffe7991b90)
>>>>>>>>
>>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:317
>>>>>>>>
>>>>>>>> #12 0x00007f6c18cc5818 in check_components (components=0x7f6c18f46ef0,
>>>>>>>> comm=0x6037a0) at
>>>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:281
>>>>>>>>
>>>>>>>> #13 0x00007f6c18cbe3c9 in mca_coll_base_comm_select (comm=0x6037a0) at
>>>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:117
>>>>>>>>
>>>>>>>> #14 0x00007f6c18c52301 in ompi_mpi_init (argc=1, argv=0x7fffe79924c8,
>>>>>>>> requested=0, provided=0x7fffe79922e8) at
>>>>>>>> ../../ompi/runtime/ompi_mpi_init.c:918
>>>>>>>>
>>>>>>>> #15 0x00007f6c18c86e92 in PMPI_Init (argc=0x7fffe799234c,
>>>>>>>> argv=0x7fffe7992340) at pinit.c:84
>>>>>>>>
>>>>>>>> #16 0x0000000000401056 in main (argc=1, argv=0x7fffe79924c8) at
>>>>>>>> send.c:32
>>>>>>>>
>>>>>>>> (gdb) up
>>>>>>>>
>>>>>>>> #1 <signal handler called>
>>>>>>>>
>>>>>>>> (gdb) up
>>>>>>>>
>>>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>>>> ../../orte/util/name_fns.c:522
>>>>>>>>
>>>>>>>> 522 if (name1->jobid < name2->jobid) {
>>>>>>>>
>>>>>>>> (gdb) print name1
>>>>>>>>
>>>>>>>> $1 = (const orte_process_name_t *) 0x192350001
>>>>>>>>
>>>>>>>> (gdb) print *name1
>>>>>>>>
>>>>>>>> Cannot access memory at address 0x192350001
>>>>>>>>
>>>>>>>> (gdb) print name2
>>>>>>>>
>>>>>>>> $2 = (const orte_process_name_t *) 0xbaf76c
>>>>>>>>
>>>>>>>> (gdb) print *name2
>>>>>>>>
>>>>>>>> $3 = {jobid = 2452946945, vpid = 1}
>>>>>>>>
>>>>>>>> (gdb)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: devel [mailto:devel-boun...@open-mpi.org
>>>>>>>> <devel-boun...@open-mpi.org>] On Behalf Of Gilles
>>>>>>>>
>>>>>>>>> Gouaillardet
>>>>>>>>> Sent: Wednesday, July 30, 2014 2:16 AM
>>>>>>>>> To: Open MPI Developers
>>>>>>>>> Subject: Re: [OMPI devel] trunk compilation errors in jenkins
>>>>>>>>> George,
>>>>>>>>> #4815 is indirectly related to the move :
>>>>>>>>> in bcol/basesmuma, we used to compare ompi_process_name_t, and now
>>>>>>>>> we (try to) compare an ompi_process_name_t and an opal_process_name_t
>>>>>>>>> (which causes a glory SIGSEGV)
>>>>>>>>> i proposed a temporary patch which is both broken and unelegant, could
>>>>>>>> you
>>>>>>>>
>>>>>>>>> please advise a correct solution ?
>>>>>>>>> Cheers,
>>>>>>>>> Gilles
>>>>>>>>> On 2014/07/27 7:37, George Bosilca wrote:
>>>>>>>>>> If you have any issue with the move, I’ll be happy to help and/or
>>>>>>>> support
>>>>>>>>
>>>>>>>>> you on your last move toward a completely generic BTL. To facilitate
>>>>>>>> your
>>>>>>>>
>>>>>>>>> work I exposed a minimalistic set of OMPI information at the OPAL
>>>>>>>> level. Take
>>>>>>>>
>>>>>>>>> a look at opal/util/proc.h for more info, but please try not to expose
>>>>>>>> more.
>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> devel mailing list
>>>>>>>>> de...@open-mpi.org
>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>> Link to this post: http://www.open-
>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>>>
>>>>>>>>> mpi.org/community/lists/devel/2014/07/15348.php
>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>>> ------------------------------
>>>>>>>> This email message is for the sole use of the intended recipient(s)
>>>>>>>> and may contain confidential information. Any unauthorized review,
>>>>>>>> use,
>>>>>>>> disclosure or distribution is prohibited. If you are not the intended
>>>>>>>> recipient, please contact the sender by reply email and destroy all
>>>>>>>> copies
>>>>>>>> of the original message.
>>>>>>>> ------------------------------
>>>>>>>> _______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> de...@open-mpi.org
>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>> Link to this post:
>>>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15355.php
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> de...@open-mpi.org
>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>> Link to this post:
>>>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15356.php
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> de...@open-mpi.org
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15363.php
>>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15364.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15365.php
>>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15366.php
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15367.php
>>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15368.php
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/08/15446.php
>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/08/15454.php
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/08/15509.php
