[
https://issues.apache.org/jira/browse/HADOOP-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687404#comment-13687404
]
Leo Leung commented on HADOOP-9586:
-----------------------------------
Could be related to DFSClient::getRandomLocalInterfaceAddr() picking up an IPv6
IP for interface "lo" on a dual stack system. Changes seems to be from
[HDFS-3148]
h3. The following fails:
2013-06-18 16:34:30,141 DEBUG hdfs.DFSClient (DFSClient.java:<init>(372)) -
Using local interfaces [lo] with addresses
{color:red}[/0:0:0:0:0:0:0:1%1:0,/127.0.0.1:0]{color}
[...]
2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:51117
2013-06-18 16:34:30,289 DEBUG hdfs.DFSClient
(DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface
{color:red}*/0:0:0:0:0:0:0:1%1:0*{color}
2013-06-18 16:34:30,289 INFO hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3904)) - Exception in
createBlockOutputStream 127.0.0.1:51117 java.net.SocketException: Network is
unreachable
2013-06-18 16:34:30,290 INFO hdfs.DFSClient
(DFSClient.java:nextBlockOutputStream(3810)) - Abandoning
blk_7035064663379527570_1001
2013-06-18 16:34:30,292 INFO hdfs.DFSClient
(DFSClient.java:nextBlockOutputStream(3814)) - Excluding datanode
127.0.0.1:51117
2013-06-18 16:34:30,294 WARN namenode.FSNamesystem
(BlockPlacementPolicyDefault.java:chooseTarget(209)) - Not able to place enough
replicas, still in need of 1 to reach 1
Not able to place enough replicas
2013-06-18 16:34:30,294 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1235)) - PriviledgedActionException as:ddn
cause:java.io.IOException: File /user/ddn/filestatus.dat could only be
replicated to 0 nodes, instead of 1
2013-06-18 16:34:30,294 INFO ipc.Server (Server.java:run(1463)) - IPC Server
handler 8 on 41922, call addBlock(/user/ddn/filestatus.dat,
DFSClient_NONMAPREDUCE_-97075681_1,
[Lorg.apache.hadoop.hdfs.protocol.DatanodeInfo;@b04bce3) from 127.0.0.1:48545:
error: java.io.IOException: File /user/ddn/filestatus.dat could only be
replicated to 0 nodes, instead of 1
java.io.IOException: File /user/ddn/filestatus.dat could only be replicated to
0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1983)
h3. The following works:
[root@host] echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
2013-06-18 16:52:41,890 DEBUG hdfs.DFSClient (DFSClient.java:<init>(372)) -
Using local interfaces [lo] with addresses [/127.0.0.1:0]
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3852)) - Connecting to 127.0.0.1:45078
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:getRandomLocalInterfaceAddr(1204)) - Using local interface
/127.0.0.1:0
2013-06-18 16:52:42,029 DEBUG hdfs.DFSClient
(DFSClient.java:createBlockOutputStream(3856)) - Send buf size 131072
2013-06-18 16:52:42,030 INFO datanode.DataNode
(DataXceiver.java:writeBlock(249)) - Receiving blk_4193380550170530321_1001
src: /127.0.0.1:60010 dest: /127.0.0.1:45078
2013-06-18 16:52:42,032 DEBUG hdfs.DFSClient (DFSClient.java:run(3277)) -
DataStreamer block blk_4193380550170530321_1001 wrote packet seqno:0 size:8281
offsetInBlock:0 lastPacketInBlock:true
h3. Summary/Proposal:
Need a fix for DFSClient to understand and support (or not) IPv6
> unit test failure:
> org.apache.hadoop.hdfs.TestFileCreation.testFileCreationSetLocalInterface
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-9586
> URL: https://issues.apache.org/jira/browse/HADOOP-9586
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
> Affects Versions: 1.3.0
> Reporter: Giridharan Kesavan
>
> https://builds.apache.org/job/Hadoop-branch1/lastCompletedBuild/testReport/org.apache.hadoop.hdfs/TestFileCreation/testFileCreationSetLocalInterface/
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/jenkins/filestatus.dat could only be replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy5.addBlock(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at $Proxy5.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
