Nilotpal Nandi created HDDS-725: ----------------------------------- Summary: Exception thrown in loop while trying to write a file in ozonefs Key: HDDS-725 URL: https://issues.apache.org/jira/browse/HDDS-725 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.3.0 Environment: Ran the following command :
---------------------------------------- ozone fs -put 2GB /testdir5/ Exceptions are thrown continuously in loop. Please note that there are 8 datanodes alive in the cluster. {noformat} root@ctr-e138-1518143905142-544443-01-000008 logs]# /root/allssh.sh 'jps -l | grep Datanode' ------------------------ Host::172.27.20.96 ------------------------ 411564 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.20.91 ------------------------ 472897 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.38.9 ------------------------ 351139 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.24.90 ------------------------ 314304 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.15.139 ------------------------ 324820 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.10.199 ------------------------ ------------------------ Host::172.27.15.131 ------------------------ ------------------------ Host::172.27.57.0 ------------------------ ------------------------ Host::172.27.23.139 ------------------------ 627053 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.68.65 ------------------------ 557443 org.apache.hadoop.ozone.HddsDatanodeService ------------------------ Host::172.27.19.74 ------------------------ ------------------------ Host::172.27.85.64 ------------------------ 508121 org.apache.hadoop.ozone.HddsDatanodeService{noformat} {noformat} 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: Election REJECTED; received 0 response(s) [] and 2 exception(s); 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:t16296, leader=null, voted=7c3b2fb1-cf16-4e5f-94dc-8a089492ad57, raftlog=[(t:37, i:271)], conf=271: [7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:172.27.85.64:9858, 86f9e313-ae49-4675-95d7-27856641aee1:172.27.15.131:9858, 9524f4e2-9031-4852-ab7c-11c2da3460db:172.27.57.0:9858], old=null 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from CANDIDATE to FOLLOWER at term 16296 for changeToFollower 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown LeaderElection 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start FollowerState 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.FollowerState: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes to CANDIDATE, lastRpcTime:1078, electionTimeout:1078ms 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown FollowerState 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from FOLLOWER to CANDIDATE at term 16296 for changeToCandidate 2018-10-24 09:49:48,172 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start LeaderElection 2018-10-24 09:49:48,173 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: begin an election in Term 16297 2018-10-24 09:49:48,174 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 got exception when requesting votes: {} java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214) at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146) at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102) Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132) at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:61) at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150) at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.15.131:9858 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 11 more {noformat} Reporter: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org