[ https://issues.apache.org/jira/browse/HDFS-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273266#comment-17273266 ]
JiangHua Zhu edited comment on HDFS-15162 at 1/28/21, 6:21 AM: --------------------------------------------------------------- [~ayushtkn] , I noticed your opinion. I agree with what you said. When the DN connects to the NN abnormally, it means that the NN is under pressure or the midway connection fails. Recently I encountered a problem. When DN connected to NN, after frequent retries many times (for example, 50 times), an exception broke out. The log is as follows: 2021-01-01 17:55:21,099 [15993307503]-INFO [clusterxxxx lifeline to xxxx/xxxx:port:Client$Connection@948]-Retrying connect to server: xxxx/xxxx:port. Already tried 49 time(s) ; retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2021-01-01 17:55:21,100 [15993307504]-WARN [clusterxxxx lifeline to xxxx/xxxx:port:BPServiceActor$LifelineSender@1008]-IOException in LifelineSender for Block pool xxxx (Datanode Uuid xxxx) service to xxxx/xxxx: port java.net.ConnectException: Call From xxxx/xxxx to xxxx:port failed on connection exception: java.net.ConnectException: Connection refused; For more details see: [http://wiki.apache.org/hadoop/ConnectionRefused] at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) at org.apache.hadoop.ipc.Client.call(Client.java:1453) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy21.sendLifeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeLifelineProtocolClientSideTranslatorPB.sendLifeline(DatanodeLifelineProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.sendLifeline(BPServiceActor.java:1074) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.sendLifelineIfDue(BPServiceActor.java:1058) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.run(BPServiceActor.java:1003) FBR should not be triggered at this time. was (Author: jianghuazhu): [~ayushtkn] , I noticed your opinion. I agree with what you said. When the DN connects to the NN abnormally, it means that the NN is under pressure or the midway connection fails. Recently I encountered a problem. When DN connected to NN, after frequent retries many times (for example, 50 times), an exception broke out. The log is as follows: 2021-01-01 17:55:21,099 [15993307503]-INFO [clusterxxxx lifeline to xxxx/xxxx:port:Client$Connection@948]-Retrying connect to server: xxxx/xxxx:port. Already tried 49 time(s) ; retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2021-01-12 17:55:21,100 [15993307504]-WARN [clusterxxxx lifeline to xxxx/xxxx:port:BPServiceActor$LifelineSender@1008]-IOException in LifelineSender for Block pool xxxx (Datanode Uuid xxxx) service to xxxx/xxxx: port java.net.ConnectException: Call From xxxx/xxxx to xxxx:port failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) at org.apache.hadoop.ipc.Client.call(Client.java:1453) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy21.sendLifeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeLifelineProtocolClientSideTranslatorPB.sendLifeline(DatanodeLifelineProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.sendLifeline(BPServiceActor.java:1074) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.sendLifelineIfDue(BPServiceActor.java:1058) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$LifelineSender.run(BPServiceActor.java:1003) FBR should not be triggered at this time. > Optimize frequency of regular block reports > ------------------------------------------- > > Key: HDFS-15162 > URL: https://issues.apache.org/jira/browse/HDFS-15162 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Ayush Saxena > Assignee: Ayush Saxena > Priority: Critical > > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. > This JIRA intents to limit the regular block reports to be sent only in case > of the above scenarios and during re-registration of datanode, to eliminate > the overhead of processing BlockReports at Namenode in case of huge clusters. > *Eg.* If a block report was sent at 0000 hours and the next was scheduled at > 0600 hours if there is no above mentioned scenario, it will skip sending the > BR, and schedule it to next 1200 hrs. if something of such sort happens > between 06:- 12: it would send the BR normally. > *NOTE*: This would be optional and can be turned off by default. Would add a > configuration to enable this. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org