> It doesn't make any sense to use a chunk size that is as large as the > entire spatial domain. As I understand it, H5Pset_chunk has the
Indeed. And even if the data was 1 TB (in my problematic case it was just 15 GB), a chunk size even approaching 4 GB is unlikely to be optimal: HDF5 will always read at least a chunk, so if you want to do anything with a small subset of the data, say just 10 MB, you'll end up reading 4 GB. > restriction that the chunk size cannot be larger than the data, so to be > able to write small data files, PETSc has to set small chunk sizes for > small data. For small files, chunking is probably not going to change performance in any significant manner, so one option could be to simply not chunk small files at all and then chunk big files "optimally" – whatever that means. HDFgroup seems to think that "the chunk size be approximately equal to the average expected size of the data block needed by the application." (http://www.hdfgroup.org/training/HDFtraining/UsersGuide/Perform.fm2.html) For more chunking stuff: In the case of PETSc I think that means not the WHOLE application, but one MPI rank (or perhaps one SMP host running a mixture of MPI ranks and OpenMP threads), which is probably always going to be < 4 GB (except perhaps in the mixture case). > I think this issue can be fixed by changing > > chunkDims[dim] = dims[dim]; > > to something like > > chunkDims[dim] = PetscMin(dims[dim],dim_for_norminal_chunk_size); I'll see how that affects the performance in my case: turning chunking completely off works too, but I would not expect that to excel in performance. > Does anyone have data on what chunk sizes would make a good default? 10 > MiB? See above, but note also that there can at most be 64k chunks in the file, so fixing the chunk size to 10 MiB means limiting file size to 640 GiB. My suggestion is to give PETSc a little more logic here, something like this: if sizeof(data) > 4GiB * 64k: no chunking # impossible to chunk! elif sizeof(data) < small_file_limit: no chunking # probably best for speed elif current rank's data size < 4 GB: chunk using current ranks data size else divide current rank's data size by 2**(number of dimensions) until < 4 GB and then use that chunk size. It is obvious what dimensions the chunks would be in that case. I hope I covered all contingencies there. Cheers, Juha -- ----------------------------------------------- | Juha Jäykkä, [email protected] | | http://koti.kapsi.fi/~juhaj/ | -----------------------------------------------
signature.asc
Description: This is a digitally signed message part.
