Hi all, On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:
> Hi, > > HDF5 doesn't support searching within datasets. You might want to look into > something like Fastbit: > > http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html > http://crd.lbl.gov/~kewu/fastbit/doc/index.html > > This is an addon-library which builds indices on HDF5 datasets allowing to > select certain regions based on indexed properties. We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming... Quincey > Werner > > > On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_pi...@yahoo.com> wrote: > > Hi, > > But that approach would be slow right? assuming that we are dealing > with the worst possible volume of data. > > From: Francesc Alted <fal...@pytables.org> > To: HDF Users Discussion List <hdf-forum@hdfgroup.org> > Sent: Fri, August 13, 2010 10:30:56 PM > Subject: Re: [Hdf-forum] Select query options > > 2010/8/13 Rodel Piano <rodel_pi...@yahoo.com> > Thanks for thr reply Francesc. > > My codes are in C. Lets say my records are about 12million. I cant use > hyperslab coz i wont know which region to look for. What is the expected > result of my query is only 10 records out 12 million. > > If you don't know which region you should look, then you should look into > your complete table. Just use hyperslabs for reading your dataset in small > chunks (my suggestion is that these chunks should fit your CPU cache, for > improved performance). > > -- > Francesc Alted > > > > > -- > ___________________________________________________________________________ > Dr. Werner Benger Visualization Research > Laboratory for Creative Arts and Technology (LCAT) > Center for Computation & Technology at Louisiana State University (CCT/LSU) > 211 Johnston Hall, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 > _______________________________________________ > Hdf-forum is for HDF software users discussion. > Hdf-forum@hdfgroup.org > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@hdfgroup.org http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
