Chatting on list, was thought that an InMemoryMapFile would not be too
hard to do.. couple of days, maybe. In HBaseMapFile, could read all of
the data into memory, into arrays (since it already sorted) as we do now
reading in the index. The HBaseMapFile.Reader would be modified to go
get entries from in-memory rather than from disk.
Would be ugly since flags would have to go down through multiple levels
of inheritance -- down through BloomFilterMapFile, HalfMapFile -- and
that it should get cleaned up when we do our own Mapfile.
But was thought, without more intelligent rebalancing of regions over
cluster so they were evenly distributed, with our current lumpy
assignment, it would be easy for a regionserver to have its memory
overstrained; it'd go down with an OOME, regions would be redistributed
lumpy, another would go down and then a downward spiral. Was thought
this had to be addressed first.
(That a fair summary Bryan?)
St.Ack
Bryan Duxbury wrote:
I'm not thinking of hot cell caching. I'm talking about going the
whole way and putting all the data in-memory. So yes, store file
contents would be loaded into memory, though not the memcache, because
that would get really complicated, I think. InMemoryStoreFile would
really be what I was going for, I'd guess. The table wouldn't be
read-only. Writes would go through to disk but reads would come
straight from memory.
On Mar 31, 2008, at 8:42 PM, stack wrote:
A Reference-cache of hot cells would take a day at the outside I'd
guess. The bulk of the work is done.
If you're talking about something else, lets discuss. What would it
look like? Store MapFiles would be floated in memory or copied to
MemCache? We'd need a special In-Memory MapFile? We'd do a bulk
memcopy from HDFS up into mem and then you'd serve from there? Would
the table have to be read-only?
St.Ack
Bryan Duxbury wrote:
Quick poll for us devs - if you had to guess, how long do you think
it would take for the in-memory option of HBase to actually be
implemented to work reliably?
-Bryan
