[
https://issues.apache.org/jira/browse/HDFS-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029904#comment-13029904
]
ramkrishna.s.vasudevan commented on HDFS-1880:
----------------------------------------------
Hi pls find the logs below
2010-06-06 19:56:45,406 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=DataNode, sessionId=null
2010-06-06 19:56:45,426 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=DataNode, port=50020
2010-06-06 19:56:45,428 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics:
Initializing RPC Metrics with hostName=DataNode, port=50020
2010-06-06 19:56:45,433 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
dnRegistration = DatanodeRegistration(linux112:50010, storageID=,
infoPort=50075, ipcPort=50020)
2010-06-06 19:56:45,437 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 50020
2010-06-06 19:56:47,804 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
New storage id DS-1238806821-10.18.52.112-50010-1275834407685 is assigned to
data-node 10.18.52.112:50010
2010-06-06 19:56:47,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.18.52.112:50010,
storageID=DS-1238806821-10.18.52.112-50010-1275834407685, infoPort=50075,
ipcPort=50020)In DataNode.run, data =
FSDataset{dirpath='/home/ramkrishna/opensrchadoop/hadoop-common-0.23.0-SNAPSHOT/hadoop-root/dfs/data/current/finalized'}
2010-06-06 19:56:47,806 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-06-06 19:56:47,808 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 50020: starting
2010-06-06 19:56:47,808 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 50020: starting
2010-06-06 19:56:47,809 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 50020: starting
2010-06-06 19:56:47,809 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
using BLOCKREPORT_INTERVAL of 60000msec Initial delay: 0msec
2010-06-06 19:56:47,810 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 50020: starting
2010-06-06 19:56:47,839 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0 blocks took 2 msec to generate and 17 msecs for RPC and NN
processing
2010-06-06 19:56:47,840 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Starting Periodic block scanner.
2010-06-06 19:57:32,878 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0 blocks took 0 msec to generate and 4 msecs for RPC and NN
processing
2010-06-06 19:58:32,953 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0 blocks took 0 msec to generate and 3 msecs for RPC and NN
processing
2010-06-06 20:14:40,742 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Call to /10.18.52.181:9000 failed on local exception:
java.io.IOException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1055)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at $Proxy4.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No route to host
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:59)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:132)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:371)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:784)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:722)
2010-06-06 20:14:44,748 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:14:48,756 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:14:52,765 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:14:56,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:15:00,781 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:15:04,789 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:15:08,798 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:15:12,806 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:15:16,814 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:15:20,822 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:15:23,827 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Call to /10.18.52.181:9000 failed on local exception:
java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1055)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at $Proxy4.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
at org.apache.hadoop.ipc.Client.call(Client.java:1032)
... 5 more
2010-06-06 20:15:27,835 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:15:31,843 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:15:35,851 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:15:39,860 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:15:43,868 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:15:47,876 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:15:51,884 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:15:55,893 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:15:59,901 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:16:03,909 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:16:06,914 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Call to /10.18.52.181:9000 failed on local exception:
java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1055)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at $Proxy4.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
at org.apache.hadoop.ipc.Client.call(Client.java:1032)
... 5 more
2010-06-06 20:16:10,922 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:16:14,930 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:16:18,938 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:16:22,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:16:26,955 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:16:30,963 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:16:34,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:16:38,979 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:16:42,988 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:16:46,996 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:16:50,001 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Call to /10.18.52.181:9000 failed on local exception:
java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1055)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at $Proxy4.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
at org.apache.hadoop.ipc.Client.call(Client.java:1032)
... 5 more
2010-06-06 20:16:54,008 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:16:58,017 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:17:02,025 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:17:06,033 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:17:10,041 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:17:14,050 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:17:18,058 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:17:22,066 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 7 time(s).
2010-06-06 20:17:26,074 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 8 time(s).
2010-06-06 20:17:30,083 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 9 time(s).
2010-06-06 20:17:33,088 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Call to /10.18.52.181:9000 failed on local exception:
java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1055)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at $Proxy4.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:933)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:440)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:528)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:209)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)
at org.apache.hadoop.ipc.Client.call(Client.java:1032)
... 5 more
2010-06-06 20:17:37,095 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 0 time(s).
2010-06-06 20:17:41,103 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 1 time(s).
2010-06-06 20:17:45,111 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 2 time(s).
2010-06-06 20:17:49,120 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 3 time(s).
2010-06-06 20:17:53,128 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 4 time(s).
2010-06-06 20:17:57,136 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 5 time(s).
2010-06-06 20:18:01,144 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: /10.18.52.181:9000. Already tried 6 time(s).
2010-06-06 20:18:04,163 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeCommand action: DNA_REGISTER
2010-06-06 20:18:04,178 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0 blocks took 0 msec to generate and 8 msecs for RPC and NN
processing
> When Namenode network is unplugged, DFSClient operations waits for ever
> -----------------------------------------------------------------------
>
> Key: HDFS-1880
> URL: https://issues.apache.org/jira/browse/HDFS-1880
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Uma Maheswara Rao G
>
> When NN/DN is shutdown gracefully, the DFSClient operations which are waiting
> for a response from NN/DN, will throw exception & come out quickly
> But when the NN/DN network is unplugged, the DFSClient operations which are
> waiting for a response from NN/DN, waits for ever.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
