1.Yes, those will ensure that file will be written to available nodes .
2. BlockManager: defaultReplication = 2 This is the Default block replication which you configured in server (Namenode). The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. 3. "dfs.replication" is client(in your case confluent kafka) side property.May be,you can cross check this configuration in kafka. -Brahma Reddy Battula ________________________________ From: Nishant Verma <nishant.verma0...@gmail.com> Sent: Friday, June 30, 2017 7:50 PM To: common-u...@hadoop.apache.org Subject: Ensure High Availability of Datanodes in a HDFS cluster Hi I have a two master and three datanode HDFS cluster setup. They are AWS EC2 instances. I have to test High Availability of Datanodes i.e., if during load run where data is written on HDFS, a datanode dies then there is no data loss. The two remaning datanodes which are alive should take care of the data writes. I have set below properties in hdfs-site.xml. dfs.replication = 2 (because if any one datanode dies, then there is no issue of not able to meet replication factor) dfs.client.block.write.replace-datanode-on-failure.policy = ALWAYS dfs.client.block.write.replace-datanode-on-failure.enable = true dfs.client.block.write.replace-datanode-on-failure.best-effort = true My questions are: 1 - Does setting up above properties suffice my Datanode High Availability? Or something else is needed? 2 - On dfs service startup, I do see below INFO on namenode logs: 2017-06-27 10:51:52,546 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 2 2017-06-27 10:51:52,546 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512 2017-06-27 10:51:52,546 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1 2017-06-27 10:51:52,546 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2 But I still see that the files being created on HDFS are with replication factor 3. Why is that so? This would hurt my High Availability of Datanodes. -rw-r--r-- 3 hadoopuser supergroup 247373 2017-06-29 09:36 /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+210+0001557358+0001557452 -rw-r--r-- 3 hadoopuser supergroup 1344 2017-06-29 08:33 /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432839+0001432850 -rw-r--r-- 3 hadoopuser supergroup 3472 2017-06-29 09:03 /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432851+0001432881 -rw-r--r-- 3 hadoopuser supergroup 2576 2017-06-29 08:33 /topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+23+0001236477+0001236499 P.S. - My records are written on HDFS by Confluent Kafka Connect HDFS Sink Connector. Thanks Nishant