This is probably unprotected concurrent access to a HashMap in Hadoop metrics. See comments on https://issues.apache.org/jira/browse/HBASE-8416
On Wed, Apr 24, 2013 at 4:37 AM, Buckley,Ron <buckl...@oclc.org> wrote: > I created https://issues.apache.org/jira/browse/HBASE-8416 > > We're not using OpenTSDB, but we do have something similar grabbing the > jmx data on a regular basis. > > Eventually, we moved all the regions off of that region server. We left > it spinning overnight, going to try to look at it this morning. > > > -----Original Message----- > From: Kevin O'dell [mailto:kevin.od...@cloudera.com] > Sent: Tuesday, April 23, 2013 11:04 PM > To: user@hbase.apache.org; lars hofhansl > Subject: Re: HBase Region Server Spinning on JMX requests > > Hi Ron, > > Are you using OpenTSDB? I have seen: > > https://issues.apache.org/jira/browse/HBASE-6602 (which should be > addressed in your build). One possibility is that the Tcollector is > leaving lots of connections open and causing the spin. Unfortunately, > we have not been able to nail it down further. We are thinking > Metrics2 in trunk might inadvertently take care of this issue. > > On Tue, Apr 23, 2013 at 6:57 PM, lars hofhansl <la...@apache.org> wrote: > > Hmm... That's not good. Would you mind filing a ticket here: > https://issues.apache.org/jira/browse/HBASE ? > > > > -- Lars > > > > > > ________________________________ > > From: "Buckley,Ron" <buckl...@oclc.org> > > To: user@hbase.apache.org > > Sent: Tuesday, April 23, 2013 6:57 AM > > Subject: HBase Region Server Spinning on JMX requests > > > > > > This is with HBase 0.94.4 & CDH 4.1.1 > > > > This morning one our region servers (we have 44) stopped responding to > > the '/jmx' request. (It's working for regular activity.) > Additionally, > > the region server is now using all the CPU on the host, running all 8 > > cores at 100%. > > > > I've got several jstacks, they all look like this: > > http://pastebin.com/dGTmTEN7 > > > > If I do a wget of the /jmx url, it starts responding, but never > > completes, always stopping at the same point: > > http://pastebin.com/qhNvxrQK > > > > Has anyone ever seen this before? If so, Is there a way out of it? > > (other than bouncing the region server). > > > > BTW: There's nothing relevant in the region server log and the garbage > > collector log is normal. > > > > > > ---------------------------------------------------------------------- > > Ron Buckley > > > > -- > Kevin O'Dell > Systems Engineer, Cloudera > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
