Hi Derek
Great to read that you parallelized the code.
Sorry to hear about the OO problems,
although I enjoyed to read your characterization of it. :)
We also have plenty of that,
mostly with some Fortran90 codes that go OOverboard.
I think I suggested "YZ-books", i.e., decompose the domain across X,
which I guess would take advantage of the C array "row major order",
and obviate the need for creating MPI vector types.
However, I guess your choice really depends on how your data
is laid out in memory.
I am not sure if I understood the I/O (output) problem you described.
However, here is a suggestion.
I think I sent it in a previous email.
It assumes the global array fits rank 0/master process memory:
A) To input data (at the beginning) ,
rank 0 can read the all the data from a file to a big buffer/global
array, then all processes call MPI_Scatter[v],
which distributes the subarrays
to all ranks/slave processes;
B) To output data (at the end),
all processes call MPI_Gather[v],
which allows rank 0/master to collect the final results on a big
buffer/global array,
and then rank 0 does the output to a file (and in your case,
also converts to "Tecplot", I suppose).
If your domain decomposition took advantage of the array layout
in memory, each process can do a single call to MPI_Scatter
and/or to MPI_Gather[v] to do the job. All you need know is
the pointer to the first element of the (sub)array and its size
(and for the global array on rank0/master).
If the domain decomposition cuts across the array memory layout,
you may need to define an MPI vector type, with strides, etc,
and use it in the MPI functions above, which again can be called
only once.
With MPI type vector it is a bit more work and bookkeeping,
but not too hard.
This master/slave I/O pattern is quite common,
and admittedly old fashioned, since it doesn't take advantage of MPI-IO.
However, it is a reliable workhorse,
particularly if you have a plain NFS
mounted file system (as opposed to a parallel file system).
I hope this helps.
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Cole, Derek E wrote:
Hi all,
I posted before about doing a domain decomposition on a 3D array in C,
and this is sort of a follow up to that. I was able to get the
calculations working correctly by performing the calculations on XZ
sub-domains for all Y dimensions of the space. I think someone referred
to this as a “book.” In the space. Being that I now have an X starting
and ending point, a Z starting and ending point, and a total number of X
and Z points to visit in each direction during the computation, I am now
at another hanging point. First, some background.
I am working on modifying a code that was originally written to be run
serially. That being said, there is a massive amount of object oriented
crap that is making this a total nightmare to work on. All of the
properties that are computed for each point in the 3D mesh are stored in
structures, and those structures are stored in structures, blah blah, it
looks very gross. In order to speed this code up, I was able to pull out
the most computationally sensitive property (potential) and get it set
up in this 3D array that is allocated nicely, etc. The problem is, this
code eventually outputs after all the iterations to a Tecplot format.
The code to do this is very, very contrived.
My idea was to, for the sake of wanting to move on, stuff back all of
these XZ subdomains that I have calculated into a single array on the
first processor, so it can go about its way and do the file output on
the WHOLE domain. I seem to be having problems though, extracting out
these SubX * SubZ * Y sized portions of the original that can be sent to
the first processor. Does anyone have any examples anywhere of code that
does something like that? It appears that my 3D mesh is in X major
format in memory, so I tried to create some loops to extract Y, SubZ
sized columns of X to send back to the zero’th processor but I haven’t
had much luck yet.
Any tips are appreciated…thanks!
------------------------------------------------------------------------
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
