Is there any reason you must allocate that on the stack rather than perform a dynamic allocation? Many people would say you shouldn't attempt to put large datastructures (large is something above a couple hundred K) on the stack. Many OS's stack sizes are in the range of 8K-128K.
Accessing in smaller chunks will not be very efficient as you go towards 1 item at a time but you can take slice of the data to reduce what you are looking at. Take a look at hyperslab selections maybe as a starting point. However if this was a reasonable desktop system I would just allocate this data on the heap and read it in there. -Jason On Thu, Jun 4, 2015 at 11:24 AM, Lance Steele <[email protected]> wrote: > Greetings everyone, > > I am attempting to use the HDF5 C++ API to read data from the Visible > Infrared Imaging Radiometer Suite (VIIRS) satellite. VIIRS Sensor Data > Records (SDR) can be downloaded freely from NOAA’s Comprehensive Large > Array-Data Stewardship System (CLASS) at this URL: > http://www.nsof.class.noaa.gov/saa/products/search?sub_id=0&datatype_family=VIIRS_SDR&submit.x=19&submit.y=3 > . > > The problem I am encountering is that the data is stored in large arrays. > Some data are 6144 by 6400 32-bit floating point format. That requires > 314,573 kilobytes of memory to load into an array. In C++, I must declare > that array in the stack memory, and there is not nearly enough for that > size. > > Here is a condensed section of c++ code: > const int X = 6144; > const int Y = 6400; > float geoArray[X][Y] > ... > dataSet.read(geoArray, PredType::NATIVE_FLOAT, memSpace, dataSpace); > > Even when I change my computer’s stack size limit to 65,532 kilobytes or > attempt to dynamically allocate the array, neither solution is not enough > to prevent a segmentation fault. I am currently using h5dump to create > binary files that I can then read into c++ vectors (which allows the data > to be stored in the heap rather than the limited stack). Does the dataset > read function have any provision or usage to access values individually > rather than as a full array? How else can I access such large amounts of > data using the HDF5 C++ API? > > Thank you and best regards, > Lance > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
