Hi all,
I am running a 4 node cluster with 2 Master node( NN1, NN2 with HA using QJM) and 2 Slave nodes(DN1, DN2). I am receiving lots of Exceptions in Datanode logs as shown below 2017-07-26 17:56:00,703 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.9.132:50010, datanodeUuid=5a2e6721-3a9a-43f1-94cc-f58f24b5a15b, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-7aa9fcd4-36fc-4e7b-87cd-d20594774b85;nsid=1753301 932;c=1500696043365):Failed to transfer BP-1085904515-192.168.9.116-1500696043365:blk_1078544770_4804082 to 192.168.9.116:50010 got java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:1 92) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.ja va:2312) at java.lang.Thread.run(Thread.java:745) I have 10 million files in hdfs. All the nodes have same configurations. Above Exception started occurring when I changed the below parameters in hdfs-site.xml file. I made these changes to increase replication rate for under-replicated blocks. dfs.namenode.handler.count=5000 dfs.namenode.replication.work.multiplier.per.iteration=1000 dfs.namenode.replication.max-streams=2000 --> not documented in hdfs.site.xml dfs.namenode.replication.max-streams-hard-limit=4000 ---> not documented in hdfs.site.xml The rate of replication of blocks increased but suddenly the Exception started to appear. Can anybody explain this behavior? Regards Omprakash Paliwal ------------------------------------------------------------------------------------------------------------------------------- [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. -------------------------------------------------------------------------------------------------------------------------------