Hi Nicholas,
I'm working on similar problems of storing big data including
especially but not only large particle sets (point clouds). The layout
we use to store particles with additional fields such as velocity,
temperature etc. is the Fiberbundle HDF5 (F5) model, which is an open
source library, described at http://www.fiberbundle.net/ .
It's just a layout "recommendation" on top of HDF5, it does nothing by
itself in terms of data transformation or storing points as octrees, it
merely suggests a layout for complex data types. An octree can be
imposed on it, as well as a multi-resolution hierarchy of particles
based on clustering. On our side, we're using kD-trees, not octrees, but
actually the idea would be to store points via space-filling curves
which are like an "image" of a kd-tree. This part however is work in
progress and not yet implemented. This could be done via multiple index
sets that provide alternative point iteration schemes. Accessing subsets
based on data queries could be connected with the fastbit library, which
provides bit-optimized indexing of HDF5 datasets; it may be part of a
future HDF5 release, I am just experimenting with it to get first
experiences.
However, these indexing abilities are yet under development on our
side, so it depends on your time scale and urgency how far you could use
it. For the mere data layout, the F5 library is available right now. It
can still be coupled with another indexing library as well.
Cheers,
Werner
On 06.01.2015 05:57, Nicholas Yue wrote:
Hi,
I am writing to find out if there is an open source C/C++ library
which allows one to supply a collection of particle (position +
additional channels like velocity, density, temperatures, vorticity
etc.) and builds/layout them in an Octree fashion and persisting them
to disk based on HDF5, which has corresponding methods to retrieve
those particles at non-empty leaf node, one node at a time.
I am looking for a memory efficient way to retrieve small chunks
(at known bounds, hence octree node) of a larger data sets which has
10^8 particles per time step on commodity PC machines with 8-16Gig of
RAM of which I targeting a memory use budget of 1-2Gig as there are
other geometry in memory.
Cheers
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
