We have a hadoop cluster that is only running hbase. We recently installed ganglia to monitor those
servers. Looking over the network graphs about every 2 hours there is a very large spike. Before
the spike we are pushing about 5mb but during the spikes it is around 60-100mb. I'm pretty sure
Look for the compaction queue metrics, see if it fits. Could be major
compactions.
J-D
On Tue, Nov 22, 2011 at 3:55 PM, Jeff Whiting wrote:
> We have a hadoop cluster that is only running hbase. We recently installed
> ganglia to monitor those servers. Looking over the network graphs about
> e
bingo that was it, thanks! The compaction queue spikes at the same time. I also see the rpc
metrics fluctuate with the major compactions (they don't go unreasonably high but are several times
slower). Is there any way to even those out? Or prevent any compactions from impacting the rpc calls?
How major compactions are run is a bit dumb, just every 24h and it's
done via a counter on each region. If you don't need them to run
(let's say you don't delete stuff that much), then you can disable
them and cron it to run during low traffic periods. An easy script
would be:
echo "major_compact
