Dear all, Dear Gus, Dear George, have you seen my example program? (in the attachment) As you suggested I have tried to *think **recursively about the datatypes* but there is something wrong that I am not bale to understand, what do you think?
thanks a lot Diego On 16 January 2015 at 23:23, Gus Correa <g...@ldeo.columbia.edu> wrote: > Hi George > > Many thanks for your answer and interest in my questions. > ... so ... more questions inline ... > > On 01/16/2015 03:41 PM, George Bosilca wrote: > >> Gus, >> >> Please see my answers inline. >> >> On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote: >>> >>> Hi George >>> >>> It is still not clear to me how to deal with strides in >>> MPI_Create_type_subarray. >>> The function/subroutine interface doesn’t mention strides at all. >>> >> >> That’s indeed a little tricky. >> However, the trick here is that when you try to understand the subarray >> type you should think recursively about the datatypes involved in the >> operation. >> >> It is a pity that there is little literature (books) about MPI, >>> and the existing books are lagging behind the new MPI developments and >>> standards (MPI-2, MPI-3). >>> My most reliable sources so far were the "MPI - The complete reference" >>> books, vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2). >>> However, they do not even mention MPI_Create_type_subarray, >>> which is part of the MPI-2 standard. >>> >> >> Let me do a wild guess: you two guys must be the firsts to ask questions >> about it … >> >> > Did anybody but the MPI developers actually *used* > MPI_Create_type_subarray? > Could this explain the scarcity of questions about it? :) > > I found it in the MPI-2 standard on the web, but it is not clear to me >>> how to achieve the same effect of strides that are available in >>> MPI_Type_vector. >>> MPI_Create_type_subarray is in section 4.1.3. >>> The OMPI MPI_Create_type_subarray man page says there is an example in >>> section 9.9.2 of the MPI-2 standard. >>> However, there is no section 9.9.2. >>> Chapter 9 is about something else ("The info object"), not derived types. >>> No good example of MPI_Create_type_subarray in section 4.1.3 either, >>> which is written in the typical terse and hermetic style in which >>> standards are. >>> >> >> No comments on subjective topics … ;) >> > > It flows almost as smoothly as Backus-Naur prose. > Makes a great reading with some Mozart in the background. > > You just blew my day away, I was totally under the impression that the >> MPI standard reads like a children’s bedtime story book !!! >> >> > > Did you write it? > Do you read it for your kids at bed time? > Do they fall asleep right away? > > Oh, if AEsop, Grimm Brothers, Charles Perrault, Andersen could only have > helped as copy-desks ... > > > So, how can one handle strides in MPI_Create_type_subarray? >>> Would one have to first create several MPI_Type_vector for the various >>> dimensions, then use them as "oldtype" in MPI_Create_type_subarray? >>> That sounds awkward, because there is only one “oldtype" in >>> MPI_Create_type_subarray, not one for each dimension. >>> >> >> Exactly. Take a look at how we handle the subarray in Open MPI, >> more precisely at the file ompi/datatype/ompi_datatype_create_subarray.c. >> My comment from few days ago referred exactly to this code, where the >> subarray is basically described in terms of vector >> (well maybe vector as I was lazy to check the LB/UB). >> >> > When documentation equates to reading actual function code ... > ... that is when users drop trying to use new developments ... > > BTW, ominously a bug report on LB/UB misuse in MPI_Type_struct > *and* in MPI_Type_create_subarray ... gosh ... > > http://lists.mpich.org/pipermail/discuss/2015-January/003608.html > > But hopefully that doesn't affect Open MPI, right? > > As I said above think recursively. >> You start with the old type, >> then build another try on a dimension, >> then you use this to expose the second dimensions and so on. >> For each dimension your basic type is not the user provided old type, >> but the type you built so far. >> >> - size_array[i] is the number of elements in big data in the dimension i >> - subsize_array[i] is the of element you will include in your datatype in >> the dimension i >> - start_array[i] is how many elements you will skip in the dimension i >> before you start including them in your datatype. start[i] + subside[i] >> must be smaller or equal to size[i] >> >> > OK, that starts to make more sense than the yawning bedtime story > in the MPI-2 standard. > > I should peel off (that would be recursive) > or build up (that would be non-recursive, right?) each dimension, > one by one, > like an onion, > creating new subarrays of increasing dimensions, > one by one, > based on the subarray previously created. > Did I get it right? > So, should I peel off or build up the dimensions? > > In which regard is this any better than using MPI_Type_Vector, > which can be setup in a single non-recursive call, > as long as the sizes, strides, etc, are properly calculated? > > Is there any simple example of how to achieve stride effect with >>> MPI_Create_type_subarray in a multi-dimensional array? >>> >> >> Not as far as I know. >> But now that people expressed interest in this topic, >> maybe someone will write a blog or something about. >> >> > An example, just a simple example ... > ... to help those that have to write all steps from 1 to N, > when it comes to thinking recursively ... > When it comes to recursion, I stopped at the Fibonacci numbers. > > Well, even if it is on a blog ... > Nobody seem to care about books or printed matter anymore. > It is all about blogs, wikis, ..., please don't move the OMPI > documentation to Twitter, Facebook, or Instagram OK? > I don't have an account on any of those. > At least keep the FAQ on the web. > Our local library here was phased out, > will be replaced with a media center ... > When I hear news like those I always think of > Ray Bradbury's Fahrenheit 451 (and Franc,ois Truffaut's beautiful movie). > > BTW, when are you gentlemen going to write an updated version of the >>> “MPI - The Complete Reference"? :) >>> >> >> Maybe after the release of MPI 4.0 would be a good target … >> > > Not any sooner than that? > MPI-2 is already poorly covered in the literature, > MPI-3 only by the standard (yawn ...). > And when MPI 4 comes, would we have to wait for MPI-5 to get > the examples? > > A lot of new and exciting technologies will hopefully be going in by then, >> > > ... which won't be used if people don't get the word about them, > on how to use them effectively ... please, don't let MPI-3-4-5-6- > become a [MPI-]chat only amongst the [[[>>>MPI developers<<<]]] ... > > writing a new book might be worth the effort. >> >> > Always worth it, with examples, etc. > > George. >> >> > Thank you, George! > > Gus Correa > > >> >> >> >> >>> Thank you, >>> Gus Correa >>> (Hijacking Diego Avesani's thread, apologies to Diego.) >>> (Also, I know this question is not about Open MPI, but about MPI in >>> general. But the lack of examples may warrant asking the question here.) >>> >>> >>> On 01/16/2015 01:39 AM, George Bosilca wrote: >>> >>>> The subarray creation is an multi-dimension extension of the vector >>>> type. >>>> >>> You can see it as a vector of vector of vector and so on, one vector per >>> dimension. >>> The stride array is used to declare on each dimension what is the >>> relative displacement >>> (in number of elements) from the beginning of the dimension array. >>> >>>> >>>> It is important to use regular type creation when you can take >>>> advantage of such >>>> >>> regularity instead of resorting to use of struct or h*. This insure >>> better >>> packing/unpacking performance, as well as possible future support for >>> one-sided >>> communications. >>> >>>> >>>> George. >>>> >>>> >>>> >>>> On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote: >>>>> >>>>> I never used MPI_Type_create_subarray, only MPI_Type_Vector. >>>>> What I like about MPI_Type_Vector is that you can define a stride, >>>>> hence you can address any regular pattern in memory. >>>>> However, it envisages the array layout in memory as a big 1-D array, >>>>> with a linear index progressing in either Fortran or C order. >>>>> >>>>> Somebody correct me please if I am wrong, but at first sight >>>>> MPI_Type_Vector sounds more flexible to me than MPI_Type_create_subarray, >>>>> exactly because the latter doesn't have strides. >>>>> >>>>> The downside is that you need to do some index arithmetic to figure >>>>> the right strides, etc, to match the corresponding >>>>> Fortran90 array sections. >>>>> >>>>> There are good examples in the "MPI - The complete reference" books I >>>>> suggested to you before (actually in vol 1). >>>>> >>>>> Online I could find the two man pages (good information, but no >>>>> example): >>>>> >>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php >>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php >>>>> >>>>> There is a very simple 2D example of MPI_Type_vector using strides >>>>> here: >>>>> >>>>> https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types >>>>> >>>>> and a similar one here: >>>>> >>>>> http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html >>>>> >>>>> Gus Correa >>>>> >>>>> On 01/15/2015 06:53 PM, Diego Avesani wrote: >>>>>> dear George, dear Gus, dear all, >>>>>> Could you please tell me where I can find a good example? >>>>>> I am sorry but I can not understand the 3D array. >>>>>> >>>>>> >>>>>> Really Thanks >>>>>> >>>>>> Diego >>>>>> >>>>>> >>>>>> On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu >>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>> >>>>>> >>>>>> On Jan 15, 2015, at 06:02 , Diego Avesani < >>>>>>> diego.aves...@gmail.com >>>>>>> <mailto:diego.aves...@gmail.com>> wrote: >>>>>>> >>>>>>> Dear Gus, Dear all, >>>>>>> Thanks a lot. >>>>>>> MPI_Type_Struct works well for the first part of my problem, so I >>>>>>> am very happy to be able to use it. >>>>>>> >>>>>>> Regarding MPI_TYPE_VECTOR. >>>>>>> >>>>>>> I have studied it and for simple case it is clear to me what id >>>>>>> does (at least I believe). Foe example if I have a matrix define >>>>>>> as: >>>>>>> REAL, ALLOCATABLE (AA(:,:)) >>>>>>> ALLOCATE AA(100,5) >>>>>>> >>>>>>> I could send part of it defining >>>>>>> >>>>>>> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/) >>>>>>> >>>>>>> after that I can send part of it with >>>>>>> >>>>>>> CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0, >>>>>>> MPI_COMM_WORLD ); >>>>>>> >>>>>>> Have I understood correctly? >>>>>>> >>>>>>> What I can do in case of three dimensional array? for example >>>>>>> AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY. >>>>>>> Is that the correct way? >>>>>>> >>>>>>> Thanks again >>>>>>> >>>>>> >>>>>> Indeed, using the subarray is the right approach independent on >>>>>> the >>>>>> number of dimensions of the data (you can use it instead of >>>>>> MPI_TYPE_VECTOR as well). >>>>>> >>>>>> George. >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Diego >>>>>>> >>>>>>> >>>>>>> On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu >>>>>>> <mailto:g...@ldeo.columbia.edu>> wrote: >>>>>>> >>>>>>> Hi Diego >>>>>>> I guess MPI_Type_Vector is the natural way to send and >>>>>>> receive >>>>>>> Fortran90 array sections (e.g. your QQMLS(:,50:100,:)). >>>>>>> I used that before and it works just fine. >>>>>>> I think that is pretty standard MPI programming style. >>>>>>> I guess MPI_Type_Struct tries to emulate Fortran90 and C >>>>>>> structures >>>>>>> (as you did in your previous code, with all the surprises >>>>>>> regarding alignment, etc), not array sections. >>>>>>> Also, MPI type vector should be more easy going (and probably >>>>>>> more efficient) than MPI type struct, with less memory >>>>>>> alignment problems. >>>>>>> I hope this helps, >>>>>>> Gus Correa >>>>>>> >>>>>>> PS - These books have a quite complete description and >>>>>>> several >>>>>>> examples >>>>>>> of all MPI objects and functions, including MPI types (native >>>>>>> and user defined): >>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-0 >>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-0> >>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-1 >>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-1> >>>>>>> >>>>>>> [They cover MPI 1 and 2. I guess there is a new/upcoming book >>>>>>> with MPI 3, but for what you're doing 1 and 2 are more than >>>>>>> enough.] >>>>>>> >>>>>>> >>>>>>> On 01/13/2015 09:22 AM, Diego Avesani wrote: >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I had some wonderful talking about >>>>>>> MPI_type_create_struct adn >>>>>>> isend\irecv with >>>>>>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is >>>>>>> more clear and my >>>>>>> program works. >>>>>>> >>>>>>> Now I have another question. In may program I have >>>>>>> matrix: >>>>>>> >>>>>>> /QQMLS(:,:,:) /that is allocate as >>>>>>> >>>>>>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of >>>>>>> particles >>>>>>> >>>>>>> QQMLS is double precision. >>>>>>> >>>>>>> I would like to sent form a CPU to another part of it, >>>>>>> for >>>>>>> example, >>>>>>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of >>>>>>> the >>>>>>> particles >>>>>>> between 50 to 100. >>>>>>> I suppose that i could use MPI_Type_vector but I am not >>>>>>> sure. The >>>>>>> particle that I want to sent could be from 25 to 50 ecc.. >>>>>>> ecc..so >>>>>>> blocklength changes everytime. >>>>>>> >>>>>>> Do I have to use MPI_type_create_struct? >>>>>>> Do I have correctly understood MPI_Type_vector? >>>>>>> >>>>>>> Thanks a lot >>>>>>> >>>>>>> >>>>>>> Diego >>>>>>> >>>>>>> >>>>>>> >>>>>>> _________________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: >>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users >>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/ >>>>>>> __26171.php >>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/ >>>>>>> 26171.php> >>>>>>> >>>>>>> >>>>>>> _________________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: >>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users >>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/ >>>>>>> __26172.php >>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/ >>>>>>> 26172.php> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26184.php >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26192.php >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: http://www.open-mpi.org/ >>>>>> community/lists/users/2015/01/26193.php >>>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: http://www.open-mpi.org/ >>>>> community/lists/users/2015/01/26194.php >>>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: http://www.open-mpi.org/ >>>> community/lists/users/2015/01/26195.php >>>> >>>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: http://www.open-mpi.org/ >>> community/lists/users/2015/01/26206.php >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: http://www.open-mpi.org/community/lists/users/2015/01/ >> 26208.php >> >> > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: http://www.open-mpi.org/community/lists/users/2015/01/ > 26210.php
MODULE MOD_PRECISION integer, parameter :: dp = selected_real_kind(15, 307) ENDMODULE MOD_PRECISION PROGRAM test_2Darray_send USE MOD_PRECISION USE MPI IMPLICIT NONE INTEGER :: iCPU,icount,iTag,ip,sendcount,sendrecv,i,iStart,iEnd INTEGER :: NPT INTEGER :: status(MPI_STATUS_SIZE) TYPE tMPI INTEGER :: myrank, nCPU, iErr, status END TYPE tMPI TYPE(tMPI) :: MPIdata INTEGER :: MPI_TYPE_VECTOR_1D,MPI_TYPE_VECTOR_2D REAL(DP),ALLOCATABLE,DIMENSION(:,:) :: AA CALL MPI_INIT(MPIdata%iErr) CALL MPI_COMM_RANK(MPI_COMM_WORLD, MPIdata%myrank, MPIdata%iErr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, MPIdata%nCPU, MPIdata%iErr) NPT=10 ALLOCATE(AA(NPT,3,3)) AA=0.d0 CALL MPI_TYPE_VECTOR(3, 3, 3, MPI_DOUBLE_PRECISION, MPI_TYPE_VECTOR_1D,MPIdata%iErr) CALL MPI_TYPE_COMMIT(MPI_TYPE_VECTOR_1D,MPIdata%iErr) CALL MPI_TYPE_VECTOR(3, 3, 3, MPI_TYPE_VECTOR_1D, MPI_TYPE_VECTOR_2D,MPIdata%iErr) CALL MPI_TYPE_COMMIT(MPI_TYPE_VECTOR_1D,MPIdata%iErr) IF(MPIdata%myrank==0)THEN DO I=1,NPT AA(I,1,1)=REAL(I,DP) AA(I,2,1)=10.d0*REAL(I,DP) AA(I,3,1)=100.d0*REAL(I,DP) ENDDO ! CALL MPI_SEND(AA(1:6,:),3,MPI_TYPE_VECTOR_2D,1,5,MPI_COMM_WORLD,MPIdata%iErr) ! ENDIF CALL MPI_BARRIER(MPI_COMM_WORLD,MPIdata%iErr) IF(MPIdata%myrank==1)THEN ! CALL MPI_RECV(AA(1:6,:),3,MPI_TYPE_VECTOR_2D,0,5,MPI_COMM_WORLD,status,MPIdata%iErr) WRITE(*,*) AA(1:6,:) ! ENDIF CALL MPI_Finalize(MPIdata%iErr) ENDPROGRAM