[jira] [Updated] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4830:
-
Attachment: 4830.txt
Todd's suggestion. Testing it actually works.
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 4830.txt, hbase-stack-regionserver-sv4r9s38.out
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4830:
-
Attachment: 4830-v2.txt
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 4830-v2.txt, 4830.txt,
hbase-stack-regionserver-sv4r9s38.out
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4830:
-
Attachment: hbase-stack-regionserver-sv4r9s38.out
Two thread dumps of hung regionserver
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: hbase-stack-regionserver-sv4r9s38.out
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
