Thank you, George!
Sorry that I sent my message too quick, and didn't even write that the
bug report was from MPICH (although the link was to MPICH).
Anyway, glad to know that it helped.
So, for the time being (OMPI 1.8.4 and 1.6.5) is it safer to stick to
MPI_Type_Vector,
instead of venturing into MPI_Create_type_subarray?
How about a vanilla example of MPI_Create_type_subarray with strides?
That would be a great help to MPI pedestrians like me.
Many thanks,
Gus Correa
On Jan 17, 2015, at 11:24 PM, George Bosilca wrote:
> Gus,
>
> Thanks for the pointer to the subarray and darray bug in MPICH. It turns out
> that Open MPI has a similar issue. I guess we both followed the MPI standard
> literally, and the standard specifically mention the LB and UB marker in the
> construction of these two types. I pushed a fix for Open MPI, that will
> eventually make it’s way toward the next stable release.
>
> Thanks,
> George.
>
>
>
>> On Jan 16, 2015, at 17:23 , Gus Correa <g...@ldeo.columbia.edu> wrote:
>>
>> Hi George
>>
>> Many thanks for your answer and interest in my questions.
>> ... so ... more questions inline ...
>>
>> On 01/16/2015 03:41 PM, George Bosilca wrote:
>>> Gus,
>>>
>>> Please see my answers inline.
>>>
>>>> On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote:
>>>>
>>>> Hi George
>>>>
>>>> It is still not clear to me how to deal with strides in
>>>> MPI_Create_type_subarray.
>>>> The function/subroutine interface doesn’t mention strides at all.
>>>
>>> That’s indeed a little tricky.
>>> However, the trick here is that when you try to understand the subarray
>>> type you should think recursively about the datatypes involved in the
>>> operation.
>>>
>>>> It is a pity that there is little literature (books) about MPI,
>>>> and the existing books are lagging behind the new MPI developments and
>>>> standards (MPI-2, MPI-3).
>>>> My most reliable sources so far were the "MPI - The complete reference"
>>>> books, vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).
>>>> However, they do not even mention MPI_Create_type_subarray,
>>>> which is part of the MPI-2 standard.
>>>
>>> Let me do a wild guess: you two guys must be the firsts to ask questions
>>> about it …
>>>
>>
>> Did anybody but the MPI developers actually *used* MPI_Create_type_subarray?
>> Could this explain the scarcity of questions about it? :)
>>
>>>> I found it in the MPI-2 standard on the web, but it is not clear to me
>>>> how to achieve the same effect of strides that are available in
>>>> MPI_Type_vector.
>>>> MPI_Create_type_subarray is in section 4.1.3.
>>>> The OMPI MPI_Create_type_subarray man page says there is an example in
>>>> section 9.9.2 of the MPI-2 standard.
>>>> However, there is no section 9.9.2.
>>>> Chapter 9 is about something else ("The info object"), not derived types.
>>>> No good example of MPI_Create_type_subarray in section 4.1.3 either,
>>>> which is written in the typical terse and hermetic style in which
>>>> standards are.
>>>
>>> No comments on subjective topics … ;)
>>
>> It flows almost as smoothly as Backus-Naur prose.
>> Makes a great reading with some Mozart in the background.
>>
>>> You just blew my day away, I was totally under the impression that the
>>> MPI standard reads like a children’s bedtime story book !!!
>>>
>>
>>
>> Did you write it?
>> Do you read it for your kids at bed time?
>> Do they fall asleep right away?
>>
>> Oh, if AEsop, Grimm Brothers, Charles Perrault, Andersen could only have
>> helped as copy-desks ...
>>
>>
>>>> So, how can one handle strides in MPI_Create_type_subarray?
>>>> Would one have to first create several MPI_Type_vector for the various
>>>> dimensions, then use them as "oldtype" in MPI_Create_type_subarray?
>>>> That sounds awkward, because there is only one “oldtype" in
>>>> MPI_Create_type_subarray, not one for each dimension.
>>>
>>> Exactly. Take a look at how we handle the subarray in Open MPI,
>>> more precisely at the file ompi/datatype/ompi_datatype_create_subarray.c.
>>> My comment from few days ago referred exactly to this code, where the
>>> subarray is basically described in terms of vector
>>> (well maybe vector as I was lazy to check the LB/UB).
>>>
>>
>> When documentation equates to reading actual function code ...
>> ... that is when users drop trying to use new developments ...
>>
>> BTW, ominously a bug report on LB/UB misuse in MPI_Type_struct
>> *and* in MPI_Type_create_subarray ... gosh ...
>>
>> http://lists.mpich.org/pipermail/discuss/2015-January/003608.html
>>
>> But hopefully that doesn't affect Open MPI, right?
>>
>>> As I said above think recursively.
>>> You start with the old type,
>>> then build another try on a dimension,
>>> then you use this to expose the second dimensions and so on.
>>> For each dimension your basic type is not the user provided old type,
>>> but the type you built so far.
>>>
>>> - size_array[i] is the number of elements in big data in the dimension i
>>> - subsize_array[i] is the of element you will include in your datatype in
>>> the dimension i
>>> - start_array[i] is how many elements you will skip in the dimension i
>>> before you start including them in your datatype. start[i] + subside[i]
>>> must be smaller or equal to size[i]
>>>
>>
>> OK, that starts to make more sense than the yawning bedtime story
>> in the MPI-2 standard.
>>
>> I should peel off (that would be recursive)
>> or build up (that would be non-recursive, right?) each dimension,
>> one by one,
>> like an onion,
>> creating new subarrays of increasing dimensions,
>> one by one,
>> based on the subarray previously created.
>> Did I get it right?
>> So, should I peel off or build up the dimensions?
>>
>> In which regard is this any better than using MPI_Type_Vector,
>> which can be setup in a single non-recursive call,
>> as long as the sizes, strides, etc, are properly calculated?
>>
>>>> Is there any simple example of how to achieve stride effect with
>>>> MPI_Create_type_subarray in a multi-dimensional array?
>>>
>>> Not as far as I know.
>>> But now that people expressed interest in this topic,
>>> maybe someone will write a blog or something about.
>>>
>>
>> An example, just a simple example ...
>> ... to help those that have to write all steps from 1 to N,
>> when it comes to thinking recursively ...
>> When it comes to recursion, I stopped at the Fibonacci numbers.
>>
>> Well, even if it is on a blog ...
>> Nobody seem to care about books or printed matter anymore.
>> It is all about blogs, wikis, ..., please don't move the OMPI documentation
>> to Twitter, Facebook, or Instagram OK?
>> I don't have an account on any of those.
>> At least keep the FAQ on the web.
>> Our local library here was phased out,
>> will be replaced with a media center ...
>> When I hear news like those I always think of
>> Ray Bradbury's Fahrenheit 451 (and Franc,ois Truffaut's beautiful movie).
>>
>>>> BTW, when are you gentlemen going to write an updated version of the
>>>> “MPI - The Complete Reference"? :)
>>>
>>> Maybe after the release of MPI 4.0 would be a good target …
>>
>> Not any sooner than that?
>> MPI-2 is already poorly covered in the literature,
>> MPI-3 only by the standard (yawn ...).
>> And when MPI 4 comes, would we have to wait for MPI-5 to get
>> the examples?
>>
>>> A lot of new and exciting technologies will hopefully be going in by then,
>>
>> ... which won't be used if people don't get the word about them,
>> on how to use them effectively ... please, don't let MPI-3-4-5-6-
>> become a [MPI-]chat only amongst the [[[>>>MPI developers<<<]]] ...
>>
>>> writing a new book might be worth the effort.
>>>
>>
>> Always worth it, with examples, etc.
>>
>>> George.
>>>
>>
>> Thank you, George!
>>
>> Gus Correa
>>>
>>>
>>>
>>>
>>>>
>>>> Thank you,
>>>> Gus Correa
>>>> (Hijacking Diego Avesani's thread, apologies to Diego.)
>>>> (Also, I know this question is not about Open MPI, but about MPI in
>>>> general. But the lack of examples may warrant asking the question here.)
>>>>
>>>>
>>>> On 01/16/2015 01:39 AM, George Bosilca wrote:
>>>>> The subarray creation is an multi-dimension extension of the vector type.
>>>> You can see it as a vector of vector of vector and so on, one vector per
>>>> dimension.
>>>> The stride array is used to declare on each dimension what is the relative
>>>> displacement
>>>> (in number of elements) from the beginning of the dimension array.
>>>>>
>>>>> It is important to use regular type creation when you can take advantage
>>>>> of such
>>>> regularity instead of resorting to use of struct or h*. This insure better
>>>> packing/unpacking performance, as well as possible future support for
>>>> one-sided
>>>> communications.
>>>>>
>>>>> George.
>>>>>
>>>>>
>>>>>
>>>>>> On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote:
>>>>>>
>>>>>> I never used MPI_Type_create_subarray, only MPI_Type_Vector.
>>>>>> What I like about MPI_Type_Vector is that you can define a stride,
>>>>>> hence you can address any regular pattern in memory.
>>>>>> However, it envisages the array layout in memory as a big 1-D array,
>>>>>> with a linear index progressing in either Fortran or C order.
>>>>>>
>>>>>> Somebody correct me please if I am wrong, but at first sight
>>>>>> MPI_Type_Vector sounds more flexible to me than
>>>>>> MPI_Type_create_subarray, exactly because the latter doesn't have
>>>>>> strides.
>>>>>>
>>>>>> The downside is that you need to do some index arithmetic to figure
>>>>>> the right strides, etc, to match the corresponding
>>>>>> Fortran90 array sections.
>>>>>>
>>>>>> There are good examples in the "MPI - The complete reference" books I
>>>>>> suggested to you before (actually in vol 1).
>>>>>>
>>>>>> Online I could find the two man pages (good information, but no example):
>>>>>>
>>>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
>>>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php
>>>>>>
>>>>>> There is a very simple 2D example of MPI_Type_vector using strides here:
>>>>>>
>>>>>> https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types
>>>>>>
>>>>>> and a similar one here:
>>>>>>
>>>>>> http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html
>>>>>>
>>>>>> Gus Correa
>>>>>>
>>>>>>> On 01/15/2015 06:53 PM, Diego Avesani wrote:
>>>>>>> dear George, dear Gus, dear all,
>>>>>>> Could you please tell me where I can find a good example?
>>>>>>> I am sorry but I can not understand the 3D array.
>>>>>>>
>>>>>>>
>>>>>>> Really Thanks
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>>
>>>>>>> On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
>>>>>>>> <mailto:diego.aves...@gmail.com>> wrote:
>>>>>>>>
>>>>>>>> Dear Gus, Dear all,
>>>>>>>> Thanks a lot.
>>>>>>>> MPI_Type_Struct works well for the first part of my problem, so I
>>>>>>>> am very happy to be able to use it.
>>>>>>>>
>>>>>>>> Regarding MPI_TYPE_VECTOR.
>>>>>>>>
>>>>>>>> I have studied it and for simple case it is clear to me what id
>>>>>>>> does (at least I believe). Foe example if I have a matrix define as:
>>>>>>>> REAL, ALLOCATABLE (AA(:,:))
>>>>>>>> ALLOCATE AA(100,5)
>>>>>>>>
>>>>>>>> I could send part of it defining
>>>>>>>>
>>>>>>>> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)
>>>>>>>>
>>>>>>>> after that I can send part of it with
>>>>>>>>
>>>>>>>> CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
>>>>>>>> MPI_COMM_WORLD );
>>>>>>>>
>>>>>>>> Have I understood correctly?
>>>>>>>>
>>>>>>>> What I can do in case of three dimensional array? for example
>>>>>>>> AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
>>>>>>>> Is that the correct way?
>>>>>>>>
>>>>>>>> Thanks again
>>>>>>>
>>>>>>> Indeed, using the subarray is the right approach independent on the
>>>>>>> number of dimensions of the data (you can use it instead of
>>>>>>> MPI_TYPE_VECTOR as well).
>>>>>>>
>>>>>>> George.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Diego
>>>>>>>>
>>>>>>>>
>>>>>>>> On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu
>>>>>>>> <mailto:g...@ldeo.columbia.edu>> wrote:
>>>>>>>>
>>>>>>>> Hi Diego
>>>>>>>> I guess MPI_Type_Vector is the natural way to send and receive
>>>>>>>> Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
>>>>>>>> I used that before and it works just fine.
>>>>>>>> I think that is pretty standard MPI programming style.
>>>>>>>> I guess MPI_Type_Struct tries to emulate Fortran90 and C
>>>>>>>> structures
>>>>>>>> (as you did in your previous code, with all the surprises
>>>>>>>> regarding alignment, etc), not array sections.
>>>>>>>> Also, MPI type vector should be more easy going (and probably
>>>>>>>> more efficient) than MPI type struct, with less memory
>>>>>>>> alignment problems.
>>>>>>>> I hope this helps,
>>>>>>>> Gus Correa
>>>>>>>>
>>>>>>>> PS - These books have a quite complete description and several
>>>>>>>> examples
>>>>>>>> of all MPI objects and functions, including MPI types (native
>>>>>>>> and user defined):
>>>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-0
>>>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-0>
>>>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-1
>>>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-1>
>>>>>>>>
>>>>>>>> [They cover MPI 1 and 2. I guess there is a new/upcoming book
>>>>>>>> with MPI 3, but for what you're doing 1 and 2 are more than
>>>>>>>> enough.]
>>>>>>>>
>>>>>>>>
>>>>>>>> On 01/13/2015 09:22 AM, Diego Avesani wrote:
>>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I had some wonderful talking about MPI_type_create_struct adn
>>>>>>>> isend\irecv with
>>>>>>>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
>>>>>>>> more clear and my
>>>>>>>> program works.
>>>>>>>>
>>>>>>>> Now I have another question. In may program I have matrix:
>>>>>>>>
>>>>>>>> /QQMLS(:,:,:) /that is allocate as
>>>>>>>>
>>>>>>>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
>>>>>>>> particles
>>>>>>>>
>>>>>>>> QQMLS is double precision.
>>>>>>>>
>>>>>>>> I would like to sent form a CPU to another part of it, for
>>>>>>>> example,
>>>>>>>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
>>>>>>>> particles
>>>>>>>> between 50 to 100.
>>>>>>>> I suppose that i could use MPI_Type_vector but I am not
>>>>>>>> sure. The
>>>>>>>> particle that I want to sent could be from 25 to 50 ecc..
>>>>>>>> ecc..so
>>>>>>>> blocklength changes everytime.
>>>>>>>>
>>>>>>>> Do I have to use MPI_type_create_struct?
>>>>>>>> Do I have correctly understood MPI_Type_vector?
>>>>>>>>
>>>>>>>> Thanks a lot
>>>>>>>>
>>>>>>>>
>>>>>>>> Diego
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>>> Subscription:
>>>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>>>>> Link to this post:
>>>>>>>>
>>>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26171.php
>>>>>>>>
>>>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26171.php>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>>> Subscription:
>>>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>>>>> Link to this post:
>>>>>>>>
>>>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26172.php
>>>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26172.php>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> Link to this post:
>>>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26184.php
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26192.php
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26193.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26194.php
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26195.php
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26206.php
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2015/01/26208.php
>>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26210.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26212.php
