On Fri, May 18, 2012 at 4:54 AM, Simon Kelly <simongdke...@gmail.com> wrote: > Cluster details > CPUs: 4 x 2.39 GHz > RAM: 8 GB > Ubuntu 10.04.2 LTS > > 1230 - hbase logs for server1 record: We slept 13318ms instead of 3000ms and > regionserver1 is killed by master, request_processing_time goes way up > 1326 - hbase logs for server3 record: We slept 77377ms instead of 3000ms and > regionserver2 is killed by master
The mailing list software strips attachments, if you want to make files available you'll have to host them somewhere and provide URLs. You are overcommitting this cluster. Add more RAM, consider 3-4 times more. The standard bricks we use in production at this point have 64 GB of RAM, so we can support HBase heaps of 8 GB and all of the other Hadoop and system daemons comfortably, as well as concurrent MapReduce tasks and such. Consider: - The CMS collector basically trades heap space for GC time. Too little space and you are still going to get stop-the-world full GC pauses as fragmentation overwhelms collection. - If any of the JVM pages go to swap then during GC some page access times will go from RAM latency to disk latency, it's pathological. You should provision sufficient RAM so all of the -Xmx of Hadoop/HBase daemons and the memory requirements of all other processes fit comfortably, with allowance for overage for administrative actions. - Writes must be buffered in RAM during housekeeping actions like flushing and compacting. High write load scenarios mean sufficient space for such buffering is critical. - The amount of heap alloted for memstore is fixed by configuration. HBase maintains this global limit as part of a strategy to avoid out of memory conditions. Therefore, as the number of regions grow, the available space for each region's memstore shrinks proportionally. If you have a heap sized too small for region hosting demand, then when the number of regions gets up there, HBase will be flushing constantly tiny files and compacting endlessly. Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
