Nilotpal Nandi created HDDS-725:
-----------------------------------
Summary: Exception thrown in loop while trying to write a file in
ozonefs
Key: HDDS-725
URL: https://issues.apache.org/jira/browse/HDDS-725
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Manager
Affects Versions: 0.3.0
Environment: Ran the following command :
----------------------------------------
ozone fs -put 2GB /testdir5/
Exceptions are thrown continuously in loop. Please note that there are 8
datanodes alive in the cluster.
{noformat}
root@ctr-e138-1518143905142-544443-01-000008 logs]# /root/allssh.sh 'jps -l |
grep Datanode'
------------------------
Host::172.27.20.96
------------------------
411564 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.20.91
------------------------
472897 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.38.9
------------------------
351139 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.24.90
------------------------
314304 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.15.139
------------------------
324820 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.10.199
------------------------
------------------------
Host::172.27.15.131
------------------------
------------------------
Host::172.27.57.0
------------------------
------------------------
Host::172.27.23.139
------------------------
627053 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.68.65
------------------------
557443 org.apache.hadoop.ozone.HddsDatanodeService
------------------------
Host::172.27.19.74
------------------------
------------------------
Host::172.27.85.64
------------------------
508121 org.apache.hadoop.ozone.HddsDatanodeService{noformat}
{noformat}
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: Election REJECTED; received 0 response(s)
[] and 2 exception(s); 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:t16296,
leader=null, voted=7c3b2fb1-cf16-4e5f-94dc-8a089492ad57, raftlog=[(t:37,
i:271)], conf=271: [7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:172.27.85.64:9858,
86f9e313-ae49-4675-95d7-27856641aee1:172.27.15.131:9858,
9524f4e2-9031-4852-ab7c-11c2da3460db:172.27.57.0:9858], old=null
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RaftServerImpl:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from CANDIDATE to FOLLOWER at
term 16296 for changeToFollower
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown LeaderElection
2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start FollowerState
2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.FollowerState:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes to CANDIDATE, lastRpcTime:1078,
electionTimeout:1078ms
2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RoleInfo:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown FollowerState
2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RaftServerImpl:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from FOLLOWER to CANDIDATE at
term 16296 for changeToCandidate
2018-10-24 09:49:48,172 INFO org.apache.ratis.server.impl.RoleInfo:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start LeaderElection
2018-10-24 09:49:48,173 INFO org.apache.ratis.server.impl.LeaderElection:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: begin an election in Term 16297
2018-10-24 09:49:48,174 INFO org.apache.ratis.server.impl.LeaderElection:
7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 got exception when requesting votes: {}
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
at
org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at
org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:61)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150)
at
org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: /172.27.15.131:9858
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at
org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at
org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
at
org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at
org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at
org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at
org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
{noformat}
Reporter: Nilotpal Nandi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
