Hi,
Some data points on this issue.
1) "du" runs for 20-30 secs.
2) after some time , I dont see any activity in datanode logs
3) I cant even jstack the datanode (forced it , gave me a
DebuggerException, double checked the pid), the datanode:50075/stacks
takes forever to respond
I can telnet to datanode:50010
I think, the disk is bad or something
Pl suggest some pointers to analyze this problem
-Sagar
Sagar Naik wrote:
CLIENT EXCEPTION:
2008-12-14 08:41:46,919 [Thread-90] INFO
org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream
java.net.SocketTimeoutException: 69000 millis timeout while waiting
for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.50.80.133:54045
remote=/10.50.80.108:50010] 2008-12-14 08:41:46,919 [Thread-90] INFO
org.apache.hadoop.dfs.DFSClient: Abandoning block
blk_-7364265396616885025_5870078 2008-12-14 08:41:46,920 [Thread-90]
INFO org.apache.hadoop.dfs.DFSClient: Waiting to find target node:
10.50.80.108:50010
DATANODE
2008-12-14 08:40:39,215 INFO org.apache.hadoop.dfs.DataNode: Receiving
block blk_-7364265396616885025_5870078 src: /10.50.80.133:54045 dest:
/10.50.80.133:50010
.
.
.
.
.
I occasionally see the datanode as deadnode. When the datanode is
deadnode, I see the du forked from datanode. The "du" is seen in "D"
state
Any pointers to debug this information would help me
-Sagar
