One other thought: you might try tracing your requests to see where the
slowness happens. Recent versions of PerformanceEvaluation support this
feature and can be used directly or as an example for adding tracing to
your application.
On Thursday, November 6, 2014, Pere Kyle <p...@whisper.sh> wrote:
> Bryan,
>
> Thanks again for the incredibly useful reply.
>
> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
> in the last week (in ganglia)
>
> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>
> Freezes (laggy responses) on the cluster are frequent and affect both
> reads and writes. I noticed iowait on the nodes that spikes.
>
> The cluster goes between a state of working 100% to nothing
> serving/timeouts for no discernible reason.
>
> Looking through the logs I have tons of responseTooSlow, this is the only
> regular occurrence in the logs:
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
> on 60020): (responseTooSlow):
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.139.198:57223
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
> on 60020): (responseTooSlow):
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.21.106:41126
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
> on 60020): (responseTooSlow):
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.230.130.102:54068
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
> on 60020): (responseTooSlow):
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.227.42.252:60493
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudrea...@hubspot.com
> <javascript:;>> wrote:
>
> > blockingStoreFiles
>
>
