[ https://issues.apache.org/jira/browse/HDFS-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603472#comment-13603472 ]
Alejandro Abdelnur commented on HDFS-4600: ------------------------------------------ Uma, Nicholas, thanks for the details. I was looking at this issue with Roman, why does it work OK in a pseudo cluster setup then? I think we can lower the priority since there is a workaround. > HDFS file append failing in multinode cluster > --------------------------------------------- > > Key: HDFS-4600 > URL: https://issues.apache.org/jira/browse/HDFS-4600 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.0.3-alpha > Reporter: Roman Shaposhnik > Priority: Blocker > Fix For: 2.0.4-alpha > > Attachments: core-site.xml, hdfs-site.xml, X.java > > > NOTE: the following only happens in a fully distributed setup (core-site.xml > and hdfs-site.xml are attached) > Steps to reproduce: > {noformat} > $ javac -cp /usr/lib/hadoop/client/\* X.java > $ echo aaaaa > a.txt > $ hadoop fs -ls /tmp/a.txt > ls: `/tmp/a.txt': No such file or directory > $ HADOOP_CLASSPATH=`pwd` hadoop X /tmp/a.txt > 13/03/13 16:05:14 WARN hdfs.DFSClient: DataStreamer Exception > java.io.IOException: Failed to replace a bad datanode on the existing > pipeline due to no more good datanodes being available to try. (Nodes: > current=[10.10.37.16:50010, 10.80.134.126:50010], > original=[10.10.37.16:50010, 10.80.134.126:50010]). The current failed > datanode replacement policy is DEFAULT, and a client may configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:793) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:858) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:964) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:470) > Exception in thread "main" java.io.IOException: Failed to replace a bad > datanode on the existing pipeline due to no more good datanodes being > available to try. (Nodes: current=[10.10.37.16:50010, 10.80.134.126:50010], > original=[10.10.37.16:50010, 10.80.134.126:50010]). The current failed > datanode replacement policy is DEFAULT, and a client may configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:793) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:858) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:964) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:470) > 13/03/13 16:05:14 ERROR hdfs.DFSClient: Failed to close file /tmp/a.txt > java.io.IOException: Failed to replace a bad datanode on the existing > pipeline due to no more good datanodes being available to try. (Nodes: > current=[10.10.37.16:50010, 10.80.134.126:50010], > original=[10.10.37.16:50010, 10.80.134.126:50010]). The current failed > datanode replacement policy is DEFAULT, and a client may configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:793) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:858) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:964) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:470) > {noformat} > Given that the file actually does get created: > {noformat} > $ hadoop fs -ls /tmp/a.txt > Found 1 items > -rw-r--r-- 3 root hadoop 6 2013-03-13 16:05 /tmp/a.txt > {noformat} > this feels like a regression in APPEND's functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira