You can do scan.setCacheBlocks (false) To disable scanner caching on map reduce scans. Also use parallel GC for new generation. That will help reduce stop the world pauses with cms
Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Pere Kyle <[email protected]> Date:11/13/2014 8:54 PM (GMT-05:00) To: [email protected] Cc: Subject: Avoid GC Pauses on Scan MapReduces Hi there, Recently I have been experiencing instability when scanning our HBASE cluster. The table we are trying to scan is 1.5B records 1TB, we have 12GB heap and 17 servers. Our GC options are as so: -XX:OnOutOfMemoryError=kill -9 %p -Xmx12000m -XX:+UseConcMarkSweepGC -Xmx12g -Xmx12g The errors I am seeing in the logs: 2014-11-14 00:00:32,656 WARN org.apache.hadoop.hbase.util.Sleeper (regionserver60020): We slept 17355ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2014-11-14 00:00:32,663 WARN org.apache.hadoop.hbase.util.Sleeper (regionserver60020.periodicFlusher): We slept 24734ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired We are using hbase.client.scanner.caching=1000. I suspect this may be a block cache issue. My question is if/how to disable the block cache for the scan queries? This is taking out writes and causing instability on the cluster. Thanks, Pere
