On 08/30/2016 04:21 PM, jaber javanshir wrote:
https://www.hdfgroup.org/HDF5/faq/parallel.html How do you write to a single file in parallel in which different processes write to separate datasets?Please advise on this matter. The answer is not satisfying for the extreme scale computing where hundreds of thousand cores are involved.
If you take those hundreds of thoudsans of cores and issue I/O to the parallel file system, you will probably break your file system.
So you are imagining, say, 1,000 datasets and 100 cores will write to a dataset? Are you imagining one dataset per core? Can the HDF5 visualization and analysis tools deal reasonably well with 100,000 datasets?
A single shared dataset has a lot of workflow advantages. It also maps nicely to collective MPI-IO optimizations.
If you really need one dataset per process, then you probably also need to use the multi-dataset I/O routines (H5Dread_multi() and H5Dwrite_multi() -- are those released yet? )
==rob _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
