Does anybody know what might be the root cause of this exception:
11/05/08 17:07:28 WARN hdfs.DFSClient: Problem renewing lease for
DFSClient_1994238103 for a period of 0 seconds. Shutting down HDFS client...
java.net.SocketTimeoutException: Call to dfs-master/10.235.79.159:8020 failed
on socket timeout exception: java.net.SocketTimeoutException: 20000 millis
timeout while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
remote=dfs-master/10.235.79.159:8020]
at org.apache.hadoop.ipc.Client.wrapException(Client.java:936)
at org.apache.hadoop.ipc.Client.call(Client.java:908)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
at $Proxy5.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy5.renewLease(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1366)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1382)
at java.lang.Thread.run(Thread.java:662)
I found this in datanode's log if it is related:
2011-05-08 17:07:28,686 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception in receiveBlock for block blk_-438222431049744157_3318
java.io.EOFException: while trying to read 51692 bytes
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder blk_-438222431049744157_3318 2 : Thread is interrupted.
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder 2 for block blk_-438222431049744157_3318 terminating
2011-05-08 17:07:28,687 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
writeBlock blk_-438222431049744157_3318 received exception
java.io.EOFException: while trying to read 51692 bytes
2011-05-08 17:07:28,687 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.235.79.159:50010,
storageID=DS-1501276445-10.235.79.159-50010-1303773028829, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 51692 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:309)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:353)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:409)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:617)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:352)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:390)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:331)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111)
at java.lang.Thread.run(Thread.java:662)
This pattern in repeated several times within few seconds.
Thanks a lot,
Viliam
