aakash shah wrote:
We can assume that this record has only one key->value mapping. Value will be updated
every minute. Currently we have 1 Million these ( key->value ) pairs but I have to
make sure that we can scale it upto 10 million of these ( key-> value ) pairs.
Every 10 minute I will be updating all of these value using their keys. This is the reason I cannot go for database as a solution.
I wouldn't be so quick to dismiss a database. All your big telcos run
their mobile phone systems on databases, where the big issue is having
enough memory for the DB to stay in memory; some dedicated databases
(e.g. TimesTen) are designed to have bounded latency on lookup so you
can predict how long operations will take.
That said, if you are only doing atomic updates of a single record,
there's less need for the advanced features. Assuming >1 machine, some
kind of distributed hash table may work
I was thinking about going with memcache pool. In the mean-time I heard
about hadoop and wanted to get advice from this mailing list regarding memcache
pool vs hadoop for this specific problem.
It's not an area Hadoop deals with at all.
