Hi Pere, I have a few thoughts here:
1.) I would make sure you have these settings - -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled 2.) Does this happen with a lower scanner.caching? 3.) Did it go away when you turned the block cache off? 4.) Do you have MSLab turned on? On Thu, Nov 13, 2014 at 10:12 PM, Dhaval Shah <prince_mithi...@yahoo.co.in> wrote: > You can do scan.setCacheBlocks (false) To disable scanner caching on map > reduce scans. Also use parallel GC for new generation. That will help > reduce stop the world pauses with cms > > > Sent from my T-Mobile 4G LTE Device > > > -------- Original message -------- > From: Pere Kyle <p...@whisper.sh> > Date:11/13/2014 8:54 PM (GMT-05:00) > To: user@hbase.apache.org > Cc: > Subject: Avoid GC Pauses on Scan MapReduces > > Hi there, > > Recently I have been experiencing instability when scanning our HBASE > cluster. The table we are trying to scan is 1.5B records 1TB, we have 12GB > heap and 17 servers. Our GC options are as so: > -XX:OnOutOfMemoryError=kill -9 %p -Xmx12000m -XX:+UseConcMarkSweepGC > -Xmx12g -Xmx12g > > The errors I am seeing in the logs: > 2014-11-14 00:00:32,656 WARN org.apache.hadoop.hbase.util.Sleeper > (regionserver60020): We slept 17355ms instead of 3000ms, this is likely due > to a long garbage collecting pause and it's usually bad, see > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired > 2014-11-14 00:00:32,663 WARN org.apache.hadoop.hbase.util.Sleeper > (regionserver60020.periodicFlusher): We slept 24734ms instead of 10000ms, > this is likely due to a long garbage collecting pause and it's usually bad, > see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired > > We are using hbase.client.scanner.caching=1000. I suspect this may be a > block cache issue. My question is if/how to disable the block cache for the > scan queries? This is taking out writes and causing instability on the > cluster. > > Thanks, > Pere > -- Kevin O'Dell Systems Engineer, Cloudera
