Hi George
Many thanks for your answer and interest in my questions.
... so ... more questions inline ...
On 01/16/2015 03:41 PM, George Bosilca wrote:
Gus,
Please see my answers inline.
On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote:
Hi George
It is still not clear to me how to deal with strides in
MPI_Create_type_subarray.
The function/subroutine interface doesn’t mention strides at all.
That’s indeed a little tricky.
However, the trick here is that when you try to understand the subarray
type you should think recursively about the datatypes involved in the operation.
It is a pity that there is little literature (books) about MPI,
and the existing books are lagging behind the new MPI developments and
standards (MPI-2, MPI-3).
My most reliable sources so far were the "MPI - The complete reference" books,
vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).
However, they do not even mention MPI_Create_type_subarray,
which is part of the MPI-2 standard.
Let me do a wild guess: you two guys must be the firsts to ask questions about
it …
Did anybody but the MPI developers actually *used* MPI_Create_type_subarray?
Could this explain the scarcity of questions about it? :)
I found it in the MPI-2 standard on the web, but it is not clear to me
how to achieve the same effect of strides that are available in MPI_Type_vector.
MPI_Create_type_subarray is in section 4.1.3.
The OMPI MPI_Create_type_subarray man page says there is an example in section
9.9.2 of the MPI-2 standard.
However, there is no section 9.9.2.
Chapter 9 is about something else ("The info object"), not derived types.
No good example of MPI_Create_type_subarray in section 4.1.3 either,
which is written in the typical terse and hermetic style in which
standards are.
No comments on subjective topics … ;)
It flows almost as smoothly as Backus-Naur prose.
Makes a great reading with some Mozart in the background.
You just blew my day away, I was totally under the impression that the
MPI standard reads like a children’s bedtime story book !!!
Did you write it?
Do you read it for your kids at bed time?
Do they fall asleep right away?
Oh, if AEsop, Grimm Brothers, Charles Perrault, Andersen could only have
helped as copy-desks ...
So, how can one handle strides in MPI_Create_type_subarray?
Would one have to first create several MPI_Type_vector for the various dimensions, then
use them as "oldtype" in MPI_Create_type_subarray?
That sounds awkward, because there is only one “oldtype" in
MPI_Create_type_subarray, not one for each dimension.
Exactly. Take a look at how we handle the subarray in Open MPI,
more precisely at the file ompi/datatype/ompi_datatype_create_subarray.c.
My comment from few days ago referred exactly to this code, where the
subarray is basically described in terms of vector
(well maybe vector as I was lazy to check the LB/UB).
When documentation equates to reading actual function code ...
... that is when users drop trying to use new developments ...
BTW, ominously a bug report on LB/UB misuse in MPI_Type_struct
*and* in MPI_Type_create_subarray ... gosh ...
http://lists.mpich.org/pipermail/discuss/2015-January/003608.html
But hopefully that doesn't affect Open MPI, right?
As I said above think recursively.
You start with the old type,
then build another try on a dimension,
then you use this to expose the second dimensions and so on.
For each dimension your basic type is not the user provided old type,
but the type you built so far.
- size_array[i] is the number of elements in big data in the dimension i
- subsize_array[i] is the of element you will include in your datatype in the
dimension i
- start_array[i] is how many elements you will skip in the dimension i before
you start including them in your datatype. start[i] + subside[i] must be
smaller or equal to size[i]
OK, that starts to make more sense than the yawning bedtime story
in the MPI-2 standard.
I should peel off (that would be recursive)
or build up (that would be non-recursive, right?) each dimension,
one by one,
like an onion,
creating new subarrays of increasing dimensions,
one by one,
based on the subarray previously created.
Did I get it right?
So, should I peel off or build up the dimensions?
In which regard is this any better than using MPI_Type_Vector,
which can be setup in a single non-recursive call,
as long as the sizes, strides, etc, are properly calculated?
Is there any simple example of how to achieve stride effect with
MPI_Create_type_subarray in a multi-dimensional array?
Not as far as I know.
But now that people expressed interest in this topic,
maybe someone will write a blog or something about.
An example, just a simple example ...
... to help those that have to write all steps from 1 to N,
when it comes to thinking recursively ...
When it comes to recursion, I stopped at the Fibonacci numbers.
Well, even if it is on a blog ...
Nobody seem to care about books or printed matter anymore.
It is all about blogs, wikis, ..., please don't move the OMPI
documentation to Twitter, Facebook, or Instagram OK?
I don't have an account on any of those.
At least keep the FAQ on the web.
Our local library here was phased out,
will be replaced with a media center ...
When I hear news like those I always think of
Ray Bradbury's Fahrenheit 451 (and Franc,ois Truffaut's beautiful movie).
BTW, when are you gentlemen going to write an updated version of the
“MPI - The Complete Reference"? :)
Maybe after the release of MPI 4.0 would be a good target …
Not any sooner than that?
MPI-2 is already poorly covered in the literature,
MPI-3 only by the standard (yawn ...).
And when MPI 4 comes, would we have to wait for MPI-5 to get
the examples?
A lot of new and exciting technologies will hopefully be going in by then,
... which won't be used if people don't get the word about them,
on how to use them effectively ... please, don't let MPI-3-4-5-6-
become a [MPI-]chat only amongst the [[[>>>MPI developers<<<]]] ...
writing a new book might be worth the effort.
Always worth it, with examples, etc.
George.
Thank you, George!
Gus Correa
Thank you,
Gus Correa
(Hijacking Diego Avesani's thread, apologies to Diego.)
(Also, I know this question is not about Open MPI, but about MPI in general.
But the lack of examples may warrant asking the question here.)
On 01/16/2015 01:39 AM, George Bosilca wrote:
The subarray creation is an multi-dimension extension of the vector type.
You can see it as a vector of vector of vector and so on, one vector per
dimension.
The stride array is used to declare on each dimension what is the relative
displacement
(in number of elements) from the beginning of the dimension array.
It is important to use regular type creation when you can take advantage of such
regularity instead of resorting to use of struct or h*. This insure better
packing/unpacking performance, as well as possible future support for one-sided
communications.
George.
On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote:
I never used MPI_Type_create_subarray, only MPI_Type_Vector.
What I like about MPI_Type_Vector is that you can define a stride,
hence you can address any regular pattern in memory.
However, it envisages the array layout in memory as a big 1-D array,
with a linear index progressing in either Fortran or C order.
Somebody correct me please if I am wrong, but at first sight MPI_Type_Vector
sounds more flexible to me than MPI_Type_create_subarray, exactly because the
latter doesn't have strides.
The downside is that you need to do some index arithmetic to figure
the right strides, etc, to match the corresponding
Fortran90 array sections.
There are good examples in the "MPI - The complete reference" books I suggested
to you before (actually in vol 1).
Online I could find the two man pages (good information, but no example):
http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php
There is a very simple 2D example of MPI_Type_vector using strides here:
https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types
and a similar one here:
http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html
Gus Correa
On 01/15/2015 06:53 PM, Diego Avesani wrote:
dear George, dear Gus, dear all,
Could you please tell me where I can find a good example?
I am sorry but I can not understand the 3D array.
Really Thanks
Diego
On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>> wrote:
On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
<mailto:diego.aves...@gmail.com>> wrote:
Dear Gus, Dear all,
Thanks a lot.
MPI_Type_Struct works well for the first part of my problem, so I
am very happy to be able to use it.
Regarding MPI_TYPE_VECTOR.
I have studied it and for simple case it is clear to me what id
does (at least I believe). Foe example if I have a matrix define as:
REAL, ALLOCATABLE (AA(:,:))
ALLOCATE AA(100,5)
I could send part of it defining
CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)
after that I can send part of it with
CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
MPI_COMM_WORLD );
Have I understood correctly?
What I can do in case of three dimensional array? for example
AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
Is that the correct way?
Thanks again
Indeed, using the subarray is the right approach independent on the
number of dimensions of the data (you can use it instead of
MPI_TYPE_VECTOR as well).
George.
Diego
On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:
Hi Diego
I guess MPI_Type_Vector is the natural way to send and receive
Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
I used that before and it works just fine.
I think that is pretty standard MPI programming style.
I guess MPI_Type_Struct tries to emulate Fortran90 and C
structures
(as you did in your previous code, with all the surprises
regarding alignment, etc), not array sections.
Also, MPI type vector should be more easy going (and probably
more efficient) than MPI type struct, with less memory
alignment problems.
I hope this helps,
Gus Correa
PS - These books have a quite complete description and several
examples
of all MPI objects and functions, including MPI types (native
and user defined):
http://mitpress.mit.edu/books/__mpi-complete-reference-0
<http://mitpress.mit.edu/books/mpi-complete-reference-0>
http://mitpress.mit.edu/books/__mpi-complete-reference-1
<http://mitpress.mit.edu/books/mpi-complete-reference-1>
[They cover MPI 1 and 2. I guess there is a new/upcoming book
with MPI 3, but for what you're doing 1 and 2 are more than
enough.]
On 01/13/2015 09:22 AM, Diego Avesani wrote:
Dear all,
I had some wonderful talking about MPI_type_create_struct adn
isend\irecv with
Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
more clear and my
program works.
Now I have another question. In may program I have matrix:
/QQMLS(:,:,:) /that is allocate as
/ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
particles
QQMLS is double precision.
I would like to sent form a CPU to another part of it, for
example,
sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
particles
between 50 to 100.
I suppose that i could use MPI_Type_vector but I am not
sure. The
particle that I want to sent could be from 25 to 50 ecc..
ecc..so
blocklength changes everytime.
Do I have to use MPI_type_create_struct?
Do I have correctly understood MPI_Type_vector?
Thanks a lot
Diego
_________________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:
http://www.open-mpi.org/__mailman/listinfo.cgi/users
<http://www.open-mpi.org/mailman/listinfo.cgi/users>
Link to this post:
http://www.open-mpi.org/__community/lists/users/2015/01/__26171.php
<http://www.open-mpi.org/community/lists/users/2015/01/26171.php>
_________________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:
http://www.open-mpi.org/__mailman/listinfo.cgi/users
<http://www.open-mpi.org/mailman/listinfo.cgi/users>
Link to this post:
http://www.open-mpi.org/__community/lists/users/2015/01/__26172.php
<http://www.open-mpi.org/community/lists/users/2015/01/26172.php>
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26184.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26192.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26193.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26194.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26195.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26206.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26208.php
