In short : are there things to know / make sure of / be aware of to get
good performance with P-HDF5 ?
My understanding is that "before" HDF5 / P-HDF5, each process of a MPI
code was compelled to write / read sequential (and separate) files :
this was stressing the file system, and, perfomance was poor (I make it
short). My understanding is that P-HDF5 (and MPI-IO) were designed to
cope with this problem (MPI-IO is more "low-level", P-HDF5 is built on
top of MPI-IO and offers flexibility and ways to structure easily the
file).
To test this I wrote a MPI code. There are N MPI processus. For each
MPI process, data to write are just an (1D) array of integers
(initialised with the rank of the MPI process). In the MPI / HDF5 file,
one target to write these data by bunchs one after the other (ordered
bunchs). First each MPI process write a file sequentially (each MPI
process writes his own sequential file : one get N separate files at the
end) : the function used to write is "write" (binary). Then, I use
MPI-IO, each process writes the same bunch of data in one file
(containing all data, bunch after bunch, from all processus - one get
only one file at the end) : the function used to write is
MPI_File_write. Then, I use P-HDF5 to do the same thing than with MPI-IO
(so one get only one file at the end) : the function used to write is
H5Dwrite. I expected to get better performance with MPI-IO and P-HDF5
than with the sequential approach. The spirit of this test code is very
simple / basic (each MPI process writes his own block of data in the
same file, or, in separate files in the sequential approach).
Note : in each case (sequential, MPI-IO, P-HDF5), when I say "write
data in file", I mean writing big blocks / bunch of data at once (I do
not write data one by one - I write the biggest block of data, but
smaller than 2Gb, that is possible to write).
Note : I tried with N = 1, 2, 4, 8, 16.
Note : I generated files (MPI-IO, P-HDF5) whose size scaled from 1Gb to
16 Gb (which looks like a "very big" file to me).
Note : I followed the P-HDF5 documentation (use H5P_FILE_ACCESS and
H5P_DATASET_XFER property list + use hyperslab "by chunks")
Note : the file system is "GPFS" (it has been installed by the cluster
vendor : this is supposed to be ready to get performance out of P-HDF5 -
I am an "application" guy that try to use HDF5, I am not an "admin sys"
that would be familiar with complex related stuffs related to the file
system)
Note : I compiled the HDF5 package like this "./configure
--enable-parallel".
Note : I use CentOS + GNU compilers (for both HDF5 package and my test
code) + hdf5-1.8.13
Note : I use mpic++ (not h5pxx compilers - actually I didn't get why
HDF5 provides compilers) to compile my test code, is this a problem ?
The problem is : in all cases, I get always (sequential, MPI-IO, HDF5)
the same performances : using P-HDF5 or MPI-IO seems useless !?... And I
don't get why : this seems not logical to me. I expected to get some
improvements when using P-HDF5 / MPI-IO. I expected to get something
like this :
http://www.speedup.ch/workshops/w37_2008/HDF5-Tutorial-PDF/PSI-HDF5-PARALLEL.pdf
(slide 30). For instance I get :
============================ mpirun -n 8 ./tstIO.exe --dataSize
536870912 ============================
INFO : data block = 536870912 integers per MPI proc X 8 MPI procs =
16384 Mb = 16 Gb
Seq. : write time = 5.2239 sec, read time = 8.5331 sec
MPI-IO : write time = 6.3301 sec, read time = 5.9788 sec
P-HDF5 : write time = 5.9695 sec, read time = 6.1289 sec
============================ mpirun -n 8 ./tstIO.exe --dataSize
1073741824 ============================
INFO : data block = 1073741824 integers per MPI proc X 8 MPI procs =
32768 Mb = 32 Gb
Seq. : write time = 10.426 sec, read time = 14.353 sec
MPI-IO : write time = 11.305 sec, read time = 11.908 sec
P-HDF5 : write time = 10.886 sec, read time = 16.943 sec
I understand I can not get a clear answer to my question as it is not
sharp enough. I try to post this to get some clue to get some logic out
of the behavior I observe. Does the "code spirit" (compare sequnetial,
MPI-IO, P-HDF5 the way I do it) can not enable to see P-HDF5 performance
? If yes, why ("too simple" data set ? data should be gathered at master
side before to be written collectivelly) ? How to change this ? Should I
look for a problem in the file system ? Or somewhere else ? Missing
option(s) when configuring the HDF5 package ? Are there others things I
should or can check to be sure I am in a situation where I can get
performance out of P-HDF5 ? Are there HDF5 tools / tutorial / benchmark
(I could replay) designed to check for performance ? [just before to
send this mail, I heard about h5perf : I ran h5perf over 4 MPI
processus, I attached the log]
Any relevant clue / information would be appreciated. If what I observe
is logical I would just understand why, and, how / when it is possible
to get performance out of P-HDF5. I just would like to get some logic
out of this.
Thanks for help,
FH
PS : I can give more information and the code, if needed (?)
HDF5 Library: Version 1.8.13
rank 0: ==== Parameters ====
rank 0: IO API=posix mpiio phdf5
rank 0: Number of files=1
rank 0: Number of datasets=1
rank 0: Number of iterations=1
rank 0: Number of processes=1:4
rank 0: Number of bytes per process per dataset=256KB
rank 0: Size of dataset(s)=256KB:1MB
rank 0: File size=256KB:1MB
rank 0: Transfer buffer size=128KB:256KB
rank 0: Block size=128KB
rank 0: Block Pattern in Dataset=Contiguous
rank 0: I/O Method for MPI and HDF5=Independent
rank 0: Geometry=1D
rank 0: VFL used for HDF5 I/O=MPI-IO driver
rank 0: Data storage method in HDF5=Contiguous
rank 0: Env HDF5_PARAPREFIX=not set
rank 0: Dumping MPI Info Object(9183424) (up to 256 bytes per item):
object is MPI_INFO_NULL
rank 0: ==== End of Parameters ====
Number of processors = 4
Transfer Buffer Size: 131072 bytes, File size: 1.00 MBs
# of files: 1, # of datasets: 1, dataset size: 1.00 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 868.75 MB/s
Average Throughput: 868.75 MB/s
Minimum Throughput: 868.75 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 185.67 MB/s
Average Throughput: 185.67 MB/s
Minimum Throughput: 185.67 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2939.25 MB/s
Average Throughput: 2939.25 MB/s
Minimum Throughput: 2939.25 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 2598.70 MB/s
Average Throughput: 2598.70 MB/s
Minimum Throughput: 2598.70 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 1098.85 MB/s
Average Throughput: 1098.85 MB/s
Minimum Throughput: 1098.85 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 9.64 MB/s
Average Throughput: 9.64 MB/s
Minimum Throughput: 9.64 MB/s
Read (1 iteration(s)):
Maximum Throughput: 6213.78 MB/s
Average Throughput: 6213.78 MB/s
Minimum Throughput: 6213.78 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 1157.37 MB/s
Average Throughput: 1157.37 MB/s
Minimum Throughput: 1157.37 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 177.65 MB/s
Average Throughput: 177.65 MB/s
Minimum Throughput: 177.65 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 84.42 MB/s
Average Throughput: 84.42 MB/s
Minimum Throughput: 84.42 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2801.81 MB/s
Average Throughput: 2801.81 MB/s
Minimum Throughput: 2801.81 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 609.73 MB/s
Average Throughput: 609.73 MB/s
Minimum Throughput: 609.73 MB/s
Transfer Buffer Size: 262144 bytes, File size: 1.00 MBs
# of files: 1, # of datasets: 1, dataset size: 1.00 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 1084.64 MB/s
Average Throughput: 1084.64 MB/s
Minimum Throughput: 1084.64 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 539.32 MB/s
Average Throughput: 539.32 MB/s
Minimum Throughput: 539.32 MB/s
Read (1 iteration(s)):
Maximum Throughput: 5714.31 MB/s
Average Throughput: 5714.31 MB/s
Minimum Throughput: 5714.31 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 4604.07 MB/s
Average Throughput: 4604.07 MB/s
Minimum Throughput: 4604.07 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 1068.34 MB/s
Average Throughput: 1068.34 MB/s
Minimum Throughput: 1068.34 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 397.15 MB/s
Average Throughput: 397.15 MB/s
Minimum Throughput: 397.15 MB/s
Read (1 iteration(s)):
Maximum Throughput: 6584.46 MB/s
Average Throughput: 6584.46 MB/s
Minimum Throughput: 6584.46 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 1575.03 MB/s
Average Throughput: 1575.03 MB/s
Minimum Throughput: 1575.03 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 1032.06 MB/s
Average Throughput: 1032.06 MB/s
Minimum Throughput: 1032.06 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 441.32 MB/s
Average Throughput: 441.32 MB/s
Minimum Throughput: 441.32 MB/s
Read (1 iteration(s)):
Maximum Throughput: 3701.95 MB/s
Average Throughput: 3701.95 MB/s
Minimum Throughput: 3701.95 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 825.81 MB/s
Average Throughput: 825.81 MB/s
Minimum Throughput: 825.81 MB/s
Number of processors = 2
Transfer Buffer Size: 131072 bytes, File size: 0.50 MBs
# of files: 1, # of datasets: 1, dataset size: 0.50 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 772.72 MB/s
Average Throughput: 772.72 MB/s
Minimum Throughput: 772.72 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 437.82 MB/s
Average Throughput: 437.82 MB/s
Minimum Throughput: 437.82 MB/s
Read (1 iteration(s)):
Maximum Throughput: 4279.90 MB/s
Average Throughput: 4279.90 MB/s
Minimum Throughput: 4279.90 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 3472.11 MB/s
Average Throughput: 3472.11 MB/s
Minimum Throughput: 3472.11 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 1111.37 MB/s
Average Throughput: 1111.37 MB/s
Minimum Throughput: 1111.37 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 349.64 MB/s
Average Throughput: 349.64 MB/s
Minimum Throughput: 349.64 MB/s
Read (1 iteration(s)):
Maximum Throughput: 3934.62 MB/s
Average Throughput: 3934.62 MB/s
Minimum Throughput: 3934.62 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 1116.10 MB/s
Average Throughput: 1116.10 MB/s
Minimum Throughput: 1116.10 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 854.93 MB/s
Average Throughput: 854.93 MB/s
Minimum Throughput: 854.93 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 283.13 MB/s
Average Throughput: 283.13 MB/s
Minimum Throughput: 283.13 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2499.59 MB/s
Average Throughput: 2499.59 MB/s
Minimum Throughput: 2499.59 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 556.13 MB/s
Average Throughput: 556.13 MB/s
Minimum Throughput: 556.13 MB/s
Transfer Buffer Size: 262144 bytes, File size: 0.50 MBs
# of files: 1, # of datasets: 1, dataset size: 0.50 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 1094.55 MB/s
Average Throughput: 1094.55 MB/s
Minimum Throughput: 1094.55 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 684.00 MB/s
Average Throughput: 684.00 MB/s
Minimum Throughput: 684.00 MB/s
Read (1 iteration(s)):
Maximum Throughput: 3125.41 MB/s
Average Throughput: 3125.41 MB/s
Minimum Throughput: 3125.41 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 2688.66 MB/s
Average Throughput: 2688.66 MB/s
Minimum Throughput: 2688.66 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 1077.67 MB/s
Average Throughput: 1077.67 MB/s
Minimum Throughput: 1077.67 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 396.89 MB/s
Average Throughput: 396.89 MB/s
Minimum Throughput: 396.89 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2995.93 MB/s
Average Throughput: 2995.93 MB/s
Minimum Throughput: 2995.93 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 1118.48 MB/s
Average Throughput: 1118.48 MB/s
Minimum Throughput: 1118.48 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 865.16 MB/s
Average Throughput: 865.16 MB/s
Minimum Throughput: 865.16 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 321.75 MB/s
Average Throughput: 321.75 MB/s
Minimum Throughput: 321.75 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2074.33 MB/s
Average Throughput: 2074.33 MB/s
Minimum Throughput: 2074.33 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 527.45 MB/s
Average Throughput: 527.45 MB/s
Minimum Throughput: 527.45 MB/s
Number of processors = 1
Transfer Buffer Size: 131072 bytes, File size: 0.25 MBs
# of files: 1, # of datasets: 1, dataset size: 0.25 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 708.02 MB/s
Average Throughput: 708.02 MB/s
Minimum Throughput: 708.02 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 477.06 MB/s
Average Throughput: 477.06 MB/s
Minimum Throughput: 477.06 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2803.68 MB/s
Average Throughput: 2803.68 MB/s
Minimum Throughput: 2803.68 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 2502.57 MB/s
Average Throughput: 2502.57 MB/s
Minimum Throughput: 2502.57 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 708.02 MB/s
Average Throughput: 708.02 MB/s
Minimum Throughput: 708.02 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 277.18 MB/s
Average Throughput: 277.18 MB/s
Minimum Throughput: 277.18 MB/s
Read (1 iteration(s)):
Maximum Throughput: 2688.66 MB/s
Average Throughput: 2688.66 MB/s
Minimum Throughput: 2688.66 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 841.55 MB/s
Average Throughput: 841.55 MB/s
Minimum Throughput: 841.55 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 564.36 MB/s
Average Throughput: 564.36 MB/s
Minimum Throughput: 564.36 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 197.47 MB/s
Average Throughput: 197.47 MB/s
Minimum Throughput: 197.47 MB/s
Read (1 iteration(s)):
Maximum Throughput: 1544.29 MB/s
Average Throughput: 1544.29 MB/s
Minimum Throughput: 1544.29 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 322.64 MB/s
Average Throughput: 322.64 MB/s
Minimum Throughput: 322.64 MB/s
Transfer Buffer Size: 262144 bytes, File size: 0.25 MBs
# of files: 1, # of datasets: 1, dataset size: 0.25 MBs
IO API = POSIX
Write (1 iteration(s)):
Maximum Throughput: 735.33 MB/s
Average Throughput: 735.33 MB/s
Minimum Throughput: 735.33 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 494.15 MB/s
Average Throughput: 494.15 MB/s
Minimum Throughput: 494.15 MB/s
Read (1 iteration(s)):
Maximum Throughput: 1774.24 MB/s
Average Throughput: 1774.24 MB/s
Minimum Throughput: 1774.24 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 1643.54 MB/s
Average Throughput: 1643.54 MB/s
Minimum Throughput: 1643.54 MB/s
IO API = MPIO
Write (1 iteration(s)):
Maximum Throughput: 696.27 MB/s
Average Throughput: 696.27 MB/s
Minimum Throughput: 696.27 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 282.79 MB/s
Average Throughput: 282.79 MB/s
Minimum Throughput: 282.79 MB/s
Read (1 iteration(s)):
Maximum Throughput: 1952.66 MB/s
Average Throughput: 1952.66 MB/s
Minimum Throughput: 1952.66 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 752.75 MB/s
Average Throughput: 752.75 MB/s
Minimum Throughput: 752.75 MB/s
IO API = PHDF5 (w/MPI-IO driver)
Write (1 iteration(s)):
Maximum Throughput: 586.78 MB/s
Average Throughput: 586.78 MB/s
Minimum Throughput: 586.78 MB/s
Write Open-Close (1 iteration(s)):
Maximum Throughput: 201.15 MB/s
Average Throughput: 201.15 MB/s
Minimum Throughput: 201.15 MB/s
Read (1 iteration(s)):
Maximum Throughput: 1275.64 MB/s
Average Throughput: 1275.64 MB/s
Minimum Throughput: 1275.64 MB/s
Read Open-Close (1 iteration(s)):
Maximum Throughput: 312.45 MB/s
Average Throughput: 312.45 MB/s
Minimum Throughput: 312.45 MB/s
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
