This really looks like the issue we got that prompted applying HBASE-5898. Never heard about the rumor.
If you can isolate the region that exhibits the issue, put it on a region server by itself and patch that region server to see if it fixes the issue. Also, can you confirm that you are running java 1.6 u16? Thx, J-D On Tue, Dec 11, 2012 at 12:14 PM, Shrijeet Paliwal <shrij...@rocketfuel.com> wrote: > Done https://gist.github.com/4261746 > > > On Tue, Dec 11, 2012 at 12:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Shrijeet: >> The attachment didn't go through. >> >> Can you use pastebin ? >> >> Thanks >> >> On Tue, Dec 11, 2012 at 12:05 PM, Shrijeet Paliwal >> <shrij...@rocketfuel.com>wrote: >> >> > Hello All, >> > >> > If our read of https://issues.apache.org/jira/browse/HBASE-5898 is >> > correct it talks about three issues: >> > >> > #1 Contention on a lock causing bad performance >> > #2 HDFS slowness causing IPC handlers blocked for large times. >> > #3 A mysterious bug which causes what looks like a deadlock >> > >> > In out environment[1] we are seeing #3 (ish) signals. Attached is the >> > stack trace. One of our region server got blocked (almost all of IPC >> > handlers) for 11 hours, so it can not be #1 OR #2. >> > >> > We have not back ported the patch yet so cant say if it fixes the issue >> or >> > not. Also it is difficult to reproduce. There is a rumor that a program >> to >> > reproduce this issue exist, does it? >> > >> > Eager to hear thoughts. >> > >> > [1] HBase version 0.92.2 , Hadoop version CDH3u0 >> > -Shrijeet >> > >>
