On Wed, Dec 18, 2013 at 01:14:58AM +0100, Francesco Cattoglio wrote: > On Tuesday, 17 December 2013 at 19:09:49 UTC, H. S. Teoh wrote: > >Another OT thread to pick your brains. :) > > > >What's a good, efficient file structure for storing extremely large > >lookup tables? (Extremely large as in > 10 million entries, with keys > >and values roughly about 100 bytes each.) The structure must support > >efficient adding and lookup of entries, as these two operations will > >be very frequent. > > What kind of key do you have? vector of floats? integers?
Vector of (relatively small) integers. > How about some uberfast compression algorithm (eg: LZ4) to compress > pages when you have to offload data to the disk? That's a good idea, actually, since the keys are very similar to each other, so should compress very well when many are placed together in a single page. T -- The diminished 7th chord is the most flexible and fear-instilling chord. Use it often, use it unsparingly, to subdue your listeners into submission!