Rob, > On Aug 31, 2016, at 11:36 PM, Rob Latham <[email protected]> wrote: > > > > On 08/30/2016 04:21 PM, jaber javanshir wrote: > >> https://www.hdfgroup.org/HDF5/faq/parallel.html How do you write to a >> single file in parallel in which different processes write to separate >> datasets?Please advise on this matter. >> >> The answer is not satisfying for the extreme scale computing where >> hundreds of thousand cores are involved. > > If you take those hundreds of thoudsans of cores and issue I/O to the > parallel file system, you will probably break your file system. > > So you are imagining, say, 1,000 datasets and 100 cores will write to a > dataset? Are you imagining one dataset per core? Can the HDF5 > visualization and analysis tools deal reasonably well with 100,000 datasets? > > A single shared dataset has a lot of workflow advantages. It also maps > nicely to collective MPI-IO optimizations. > > If you really need one dataset per process, then you probably also need to > use the multi-dataset I/O routines (H5Dread_multi() and H5Dwrite_multi() -- > are those released yet? ) > Not yet... ;-( > ==rob > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
