Hi Mei, Thanks. Please also keep hdfs-dev@hadoop.apache.org on the to: list when replying, so your questions get exposure from other HDFS devs as well :)
Regarding your question, the answer is no, the NN will never violate the block placement policy on its own when choosing block location targets for an allocation. On Thu, Apr 4, 2013 at 2:22 PM, Mei Long <lon...@gmail.com> wrote: > I will open a doc JIRA. Another question: is there a chance that all three > blocks end up on the same rack of no DN exists on the writer node? Or is the > NN knows to put the other two copies on a different rack? > > > On Thu, Apr 4, 2013 at 3:07 AM, Harsh J <ha...@cloudera.com> wrote: >> >> Good to know! Please also consider improving the javadoc on that >> method to reflect this, by logging a doc JIRA under HDFS :) >> >> On Thu, Apr 4, 2013 at 12:33 PM, Mei Long <lon...@gmail.com> wrote: >> > Harsh, >> > >> > Thank you! This is exactly what I was confused about. By looking at the >> > code >> > I was suspecting that it's not looking at rack locality for the second >> > scenario (No DN on writer node.) But I was unable to confirm it without >> > running the code. >> > >> > Your answer makes so much sense. Thank you!! :-) I'm much happier now. >> > >> > On Thu, Apr 4, 2013 at 2:14 AM, Harsh J <ha...@cloudera.com> wrote: >> >> >> >> Hi Mei, >> >> >> >> These questions are best fit for the whole hdfs-dev@ group, which am >> >> adding while responding back. >> >> >> >> You're correct that the local node being not chosen, a random node on >> >> the same rack as the writer may be tried. However, if the writer node >> >> itself had no local DN in the first place, a completely random node >> >> across any rack may be selected. >> >> >> >> This is visible at >> >> >> >> >> >> http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java?view=markup >> >> (~263). >> >> >> >> On Thu, Apr 4, 2013 at 12:53 AM, Mei Long <lon...@gmail.com> wrote: >> >> > Hi Harsh, >> >> > >> >> > I found your post on JIRA regarding the comments. I find them very >> >> > confusing >> >> > as well. I'm trying to figure out the following comment: >> >> > >> >> > * the 1st replica is placed on the local machine, >> >> > * otherwise a random datanode. >> >> > >> >> > Do you know when it says "otherwise a random datanode," does it mean >> >> > any >> >> > random datanode anywhere on the network or a random datanode on the >> >> > same >> >> > rack as the local machine? I've been looking at the code for an hour >> >> > and >> >> > I'm >> >> > getting more confused with the comment and code in chooseLocalNode() >> >> > >> >> > /* choose <i>localMachine</i> as the target. >> >> > >> >> > * if <i>localMachine</i> is not availabe, >> >> > >> >> > * choose a node on the same rack >> >> > >> >> > * @return the choosen node >> >> > >> >> > */ >> >> > >> >> > Your help is much appreciated! >> >> > >> >> > Mei >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J > > -- Harsh J
