[
https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875427#action_12875427
]
Todd Lipcon commented on HADOOP-6762:
-------------------------------------
Hey Sam,
My cluster test case just spat out a nice warning and then deadlocked my DFS
client:
10/06/03 18:48:54 WARN hdfs.DFSClient: Error Recovery for block
blk_832465809601113490_3336818 in pipeline 192.168.42.40:11072,
192.168.42.41:11072, 192.168.42.43:11072: bad datanode 192.168.42.41:11072
Exception in thread "DataStreamer for file /user/todd/test-sync block
blk_832465809601113490_3336818" java.lang.reflect.UndeclaredThrowableException
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
at
org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:146)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2627)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2139)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2306)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1238)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:526)
at org.apache.hadoop.ipc.Client.call(Client.java:765)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
... 8 more
I think we need to add another catch (InterruptedException) and rethrow as
IOException around the sendParam call in Client.call
> exception while doing RPC I/O closes channel
> --------------------------------------------
>
> Key: HADOOP-6762
> URL: https://issues.apache.org/jira/browse/HADOOP-6762
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: sam rash
> Assignee: sam rash
> Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt,
> hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using
> FileSystem.newInstance(), and one of them issues a close(), the leasechecker
> thread is interrupted. This interrupt races with the rpc namenode.renew()
> and can cause a ClosedByInterruptException. This closes the underlying
> channel and the other filesystem, sharing the connection will get errors.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
