Hi, Unfortunately, this is indeed the worst you can have. It's completely normal that you have the worst performance with slicing in these dimensions. Even with a parallel filesystem, you would need to read EVERYTHING from the dataset, and then the library would pick up the pieces you need. One solution would be to agglomerate several z,w in dimensions 5 and 6, so that you still get some performance, but it will be worse than 1 or even 2.
Cheers, Matthieu 2014-06-12 20:43 GMT+01:00 Martin Sarajærvi <[email protected]>: > Hi all, > > I'm working with floating point data building up a very large dataset > typically >100Gb of four dimensions (x, y, z, w). > Dimensions are of the size (x,y,z,w) = (601, 482, 61, 1501) in my example. > > The aim is to slice (READING ONLY) this dataset in orthogonal directions: > 1) (x, *, *, *) > 2) (*, y, *, *) > 3) (*, *, z, w) > > When using a contiguous layout I naturally get good performance for > directions (1) and (2), however it is very poor for (3). > Using a chunking layout of (8,8,8,8) seem to give the best balance so far > for reasonable access times in all directions. but still not as fast as I > was hoping for. My tests also show that compression improves the read > performance slightly. > > I'm looking for advise on possible optimization techniques to use for this > problem other than what has been mentioned. > Otherwise, is my only option to move to some (expensive?) parallel solution? > > Thanks! > > Regards, > Martin > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
