This doesn't answer your question per se, but this is how we dealt with load on HBase at Lithium. We power klout.com with HBase. On a nightly basis, we load user profile data and Klout scores for approx. 600 million users into HBase. We also do maintenance on HBase such as major compactions on a regular basis. When either a load or maintenance is being performed, site performance on klout.com used to degrade pretty severely. In order to mitigate this, we stood up 2 HBase clusters and now power klout.com off both. We run these in a custom built active/passive mode. The application layer uses a zookeeper flag to connect to the active cluster and serves from there. We load data or do maintenance on the passive, then flip the clusters so repeat the load/maintenance on the previously active cluster. This mechanism of active/passive systems has been working pretty well for us. It does however require a significant cost in terms of maintaining 2 clusters.
On Sun Dec 07 2014 at 9:35:00 AM gomes <sankarm...@gmail.com> wrote: > Currently some system cleaning tasks do read all the rows, and then perform > some operations on that. It impacts other users who are being served at the > same time. System cleaning tasks are of lower priority, and I can delay the > requests, but I am just wondering if there is anyway I can hook into hbase > system, and if I can continuously measure the load of the system, and based > on that I can limit the lower priority tasks? How can I do that, if there > are any pointers, or helpful suggestions, please provide them. > > > > -- > View this message in context: http://apache-hbase.679495.n3. > nabble.com/Reduce-load-to-hbase-tp4066727.html > Sent from the HBase User mailing list archive at Nabble.com. >
