This limit of 4GB seems to be a hard limit set within hdf5 https://support.hdfgroup.org/documentation/hdf5/latest/group___d_c_p_l.html#title22
In this case, how can we save larger files ? The problem we see is with a data of the order of 512^3 grid with 5 unknowns at each node. Should we switch to some other format ? Thanks praveen > On 17 Sep 2025, at 10:29 AM, Jalil Khan <[email protected]> wrote: > > I want to correct the line numbers, it is between 8333 - 8348. > > On Wed, Sep 17, 2025 at 10:26 AM Jalil Khan <[email protected] > <mailto:[email protected]>> wrote: >> Here is how I call my data_out object to write HDF5, >> DataOutBase::DataOutFilter >> data_filter(DataOutBase::DataOutFilterFlags(true,true)); >> >> data_out.write_filtered_data(data_filter); >> >> data_out.write_hdf5_parallel(data_filter, >> write_mesh_file, >> mesh_filename, >> solution_filename, >> mpi_comm); >> >> >> Here are the lines pulled from data_out_base.cc (9.7.0 documentation) >> 833 node_dataset_id = H5Pcreate(H5P_DATASET_CREATE); >> # ifdef DEAL_II_WITH_ZLIB >> H5Pset_deflate(node_dataset_id, >> get_zlib_compression_level(flags.compression_level)); >> H5Pset_chunk(node_dataset_id, 2, cell_ds_dim); //<------- Here it sets >> some chunk size? >> # endif >> cell_dataset = H5Dcreate(h5_mesh_file_id, >> "cells", >> H5T_NATIVE_UINT, >> cell_dataspace, >> H5P_DEFAULT, >> node_dataset_id, >> H5P_DEFAULT); >> H5Pclose(node_dataset_id); >> # endif >> 840 AssertThrow(cell_dataset >= 0, ExcIO()); // <--- line 8330 in my >> error >> >> On Tue, Sep 16, 2025 at 11:29 PM 'Wolfgang Bangerth' via deal.II User Group >> <[email protected] <mailto:[email protected]>> wrote: >>> >>> On 9/16/25 05:04, Jalil Khan wrote: >>> > >>> > From inspecting the source in data_out_base.cc, HDF5 uses chunked >>> > layout, with chunk dimensions set internally in deal.II. At 512³, the >>> > computed chunk size exceeds the HDF5 hard limit of 4 GB per chunk, which >>> > leads to the failure. >>> > >>> > Is there a recommended way for users to avoid this error when writing >>> > large output files? >>> >>> Jalil: >>> that likely falls in the category of "the person who wrote these HDF5 >>> calls didn't think of this scenario, and nobody has run into it since >>> then". Can you identify which line in data_out_base.cc specifically >>> triggers the issue, and can you show how you call deal.II functions to >>> run into this issue? >>> >>> In the end, this ought to be fixed of course. Any help is obviously >>> always welcome! >>> >>> Best >>> Wolfgang >>> >>> >>> -- >>> The deal.II project is located at http://www.dealii.org/ >>> For mailing list/forum options, see >>> https://groups.google.com/d/forum/dealii?hl=en >>> --- >>> You received this message because you are subscribed to the Google Groups >>> "deal.II User Group" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:dealii%[email protected]>. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/dealii/f1bd81c7-9629-460b-977f-9b2c627bdd59%40colostate.edu. >> >> >> >> -- >> Regards >> J.R.Khan >> 8803985667 > > > > -- > Regards > J.R.Khan > 8803985667 > > -- > The deal.II project is located at http://www.dealii.org/ > For mailing list/forum options, see > https://groups.google.com/d/forum/dealii?hl=en > --- > You received this message because you are subscribed to the Google Groups > "deal.II User Group" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion visit > https://groups.google.com/d/msgid/dealii/CAALjy%3DgOi7P8VtEWWgqe%2ByULDV-AD9LYeepzWdsyeycoahx-XQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/dealii/CAALjy%3DgOi7P8VtEWWgqe%2ByULDV-AD9LYeepzWdsyeycoahx-XQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/dealii/962202CC-2EBC-4976-912D-F418616C7CF8%40gmail.com.
