It usually happens in a long running setup (at least for me). Can you throttle your load?
Replication is evolving; I'd say update if you can (or backport the jiras?). Himanshu On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <je...@qualtrics.com> wrote: > hmm. So if we are on 0.92 what suggestion would you have to prevent the > problem? > > ~Jeff > > > On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote: >> >> Number of PRI handlers are governed by >> "hbase.regionserver.metahandler.count"; default is 10. >> >> Increasing their number will not solve it, but will delay its >> occurring (i don't know about your load etc). >> >> Another related jira is HBase-6550. >> >> Some more context for your use case: >> >> http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements >> >> >> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <je...@qualtrics.com> wrote: >>> >>> It looks like that is problem we are having. We are on 0.92 so we don't >>> get >>> the patch. But one solution seems to be increasing the privileged >>> handlers. >>> How do we increase the number of privilege handlers? >>> >>> >>> ~Jeff >>> >>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote: >>>> >>>> Your RS priority handlers are blocked on meta lookup, so it becomes >>>> unresponsive. Looks like you hitting >>>> https://issues.apache.org/jira/browse/HBASE-6165 >>>> You running HBase replication? just confirming. >>>> >>>> Himanshu >>>> >>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <st...@duboce.net> wrote: >>>>> >>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook >>>>> <nathani...@qualtrics.com> >>>>> wrote: >>>>>> >>>>>> We ran a jstack on the both the RS process and the hbase shell process >>>>>> trying to do the scan. >>>>>> >>>>>> Jstack log for RS: >>>>>> http://pastebin.com/9Y9t5ERE >>>>>> >>>>> What JVM (I don't know what (20.10-b01 mixed mode) is). >>>>> >>>>> I see a bunch of this: >>>>> >>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10 >>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry >>>>> [0x000000004ab0f000] >>>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>>> at ..... >>>>> >>>>> But when I go to look for other instances of the object monitor, I >>>>> don't find any. I see this for each instance of BLOCKED (Or at least, >>>>> the two or three I checked). >>>>> >>>>> Whats your OS? >>>>> >>>>> St.Ack >>> >>> >>> -- >>> Jeff Whiting >>> Qualtrics Senior Software Engineer >>> je...@qualtrics.com >>> >>> >>> > > -- > Jeff Whiting > Qualtrics Senior Software Engineer > je...@qualtrics.com > > >
