Hi Takanobu, sorry for the late reply, I missed your email.
That looks good, I'll take a look at that. Cheers, Lars On Fri, Jul 5, 2019 at 6:47 AM Takanobu Asanuma <tasan...@yahoo-corp.jp> wrote: > Hi Lars, > > I investigated it further. BlockPlacementPolicyWithNodeGroup may achieve > your goal. > > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java > > I don't have any experience of this placement policy, but found a use case > slide from CACIB. > > https://www.slideshare.net/Hadoop_Summit/disaster-recovery-experience-at-cacib-hardening-hadoop-for-critical-financial-applications > > Thanks, > - Takanobu > > ________________________________________ > From: Takanobu Asanuma <tasan...@yahoo-corp.jp> > Sent: Thursday, July 4, 2019 8:29:23 PM > To: Lars Francke > Cc: hdfs-user@hadoop.apache.org > Subject: Re: BlockPlacementPolicy question with hierarchical topology > > Oh, you are right. It doesn't meet your needs. Sorry for the confusion. > Seems it may be difficult to achive it with the existing policies. > > - Takanobu > > ________________________________________ > From: Lars Francke <lars.fran...@gmail.com> > Sent: Thursday, July 4, 2019 7:53:35 PM > To: 浅沼 孝信 > Cc: hdfs-user@hadoop.apache.org > Subject: Re: BlockPlacementPolicy question with hierarchical topology > > Hi Takanobu, > > thanks for the quick reply. I missed that class. > > But does it really do what I need? > If I have these racks: > /dc1/rack1 > /dc1/rack2 > /dc1/rack3 > /dc2/rack1 > /dc2/rack2 > /dc2/rack3 > > And I place a single block in HDFS, couldn't this policy chose /dc1/rack1, > /dc1/rack2, /dc1/rack3 at random? > > Cheers, > Lars > > On Thu, Jul 4, 2019 at 12:46 PM Takanobu Asanuma <tasan...@yahoo-corp.jp > <mailto:tasan...@yahoo-corp.jp>> wrote: > Hi Lars, > > I think BlockPlacementPolicyRackFaultTolerant can do it. > This policy tries to place 3 replica separately in different racks. > > <property> > <name>dfs.block.replicator.classname</name> > > <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant</value> > </property> > > See also: > > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java > > Thanks, > - Takanobu > ________________________________________ > From: Lars Francke <lars.fran...@gmail.com<mailto:lars.fran...@gmail.com>> > Sent: Thursday, July 4, 2019 18:15 > To: hdfs-user@hadoop.apache.org<mailto:hdfs-user@hadoop.apache.org> > Subject: BlockPlacementPolicy question with hierarchical topology > > Hi, > > I have a customer who wants to make sure that copies of his data are > distributed amongst datacenters. So they are using rack names like this > /dc1/rack1, /dc1/rack2, /dc2/rack1 etc. > > Unfortunately, the BlockPlacementPolicyDefault seems to place all blocks > on /dc1/* sometimes. > > Is there a way to guarantee that /dc1/* and /dc2/* will be used in this > scenario? > > Looking at chooseRandomWithStorageTypeTwoTrial it seems to consider the > full "scope" and not its components. I couldn't find anything in the code > but I had hoped I'm missing something: Is there a way to configure HDFS for > the behaviour I'd like? > > Thanks! > > Lars > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hdfs-user-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-user-h...@hadoop.apache.org > >
