Re: BlockPlacementPolicy question with hierarchical topology

2019-07-11 Thread Lars Francke
t; To: Lars Francke > Cc: hdfs-user@hadoop.apache.org > Subject: Re: BlockPlacementPolicy question with hierarchical topology > > Oh, you are right. It doesn't meet your needs. Sorry for the confusion. > Seems it may be difficult to achive it with the existing policies. > > -

Re: BlockPlacementPolicy question with hierarchical topology

2019-07-04 Thread Takanobu Asanuma
rsday, July 4, 2019 8:29:23 PM To: Lars Francke Cc: hdfs-user@hadoop.apache.org Subject: Re: BlockPlacementPolicy question with hierarchical topology Oh, you are right. It doesn't meet your needs. Sorry for the confusion. Seems it may be difficult to achive it with the existing policies.

Re: BlockPlacementPolicy question with hierarchical topology

2019-07-04 Thread Takanobu Asanuma
e.org Subject: Re: BlockPlacementPolicy question with hierarchical topology Hi Takanobu, thanks for the quick reply. I missed that class. But does it really do what I need? If I have these racks: /dc1/rack1 /dc1/rack2 /dc1/rack3 /dc2/rack1 /dc2/rack2 /dc2/rack3 And I place a single block in

Re: BlockPlacementPolicy question with hierarchical topology

2019-07-04 Thread Lars Francke
RackFaultTolerant.java > > Thanks, > - Takanobu > > From: Lars Francke > Sent: Thursday, July 4, 2019 18:15 > To: hdfs-user@hadoop.apache.org > Subject: BlockPlacementPolicy question with hierarchical topology > > Hi, > > I have a customer who wants to m

Re: BlockPlacementPolicy question with hierarchical topology

2019-07-04 Thread Takanobu Asanuma
Subject: BlockPlacementPolicy question with hierarchical topology Hi, I have a customer who wants to make sure that copies of his data are distributed amongst datacenters. So they are using rack names like this /dc1/rack1, /dc1/rack2, /dc2/rack1 etc. Unfortunately, the BlockPlacementPolicyDefault

BlockPlacementPolicy question with hierarchical topology

2019-07-04 Thread Lars Francke
Hi, I have a customer who wants to make sure that copies of his data are distributed amongst datacenters. So they are using rack names like this /dc1/rack1, /dc1/rack2, /dc2/rack1 etc. Unfortunately, the BlockPlacementPolicyDefault seems to place all blocks on /dc1/* sometimes. Is there a way to