Hi, Rob Latham <[email protected]> writes: > Intel's MPI library does not have any explicit optimizations for GPFS, but the > one optimization you need for GPFS is to align writes to the file system block > size. > > you can do this with an MPI-IO hint: set "striping_unit" to your gpfs block > size (you can determine the gpfs block size via 'stat -f': see the 'Block > size:' > field. > > Setting an MPI-IO hint via HDF5 requires setting up your file access property > list appropriately: you will need a non-null INFO parameter to > H5Pset_fapl_mpio > > http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplMpio > > in C, it's like this: > MPI_Info info; > MPI_Info_create(&info); > MPI_Info_set(info, "striping_unit", "8388608") ; > /* or whatever your GPFS block size actually is*/ > H5Pset_fapl_mpio(fapl, comm, info); > > > If you're with me so far, I think you'll see much better parallel write > performance once the MPI-IO library is trying harder to align writes.
I'm going to try your suggestion and will report back, but do you think this could explain the different performance for PMLX, PMLY, and PMLZ? In the sample code in pastebin, the data written to file in each of these cases only differs by who is writing it (in the case of PMLX, processors that in a Cartesian decomposition would be in the smallest X plane, PMLY in the smallest Y plane, and PMLZ those in the smallest Z plane), but the amount of data, the distribution of that data in memory (for each processor), and the place where it is stored in the actual file is the same... > Are you familiar with the Darshan statistics tool? you can use it to confirm > you are hitting (or not) unaligned writes. Not really. Only heard about it, but I will try it. This issue is proving pretty hard to figure out, and it is a real bottleneck for our code, so I will try anything... Thanks a lot, -- Ángel de Vicente http://www.iac.es/galeria/angelv/ --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci�n de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
