[ 
https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956200#comment-14956200
 ] 

Ming Ma commented on HDFS-8647:
-------------------------------

Thanks [~brahmareddy]!

Most of the change looks good. It seems this will also fix HDFS-9083. cc: 
[~shahrs87].

There are some questions about the striped erasure coding block placement 
abstraction.

* The existing {{blockHasEnoughRacksStriped}} compares {{getRealDataBlockNum}} 
(# of data blocks) with the number racks. But after the refactoring, it 
compares {{getRealTotalBlockNum}} (# of total blocks) with the number racks,
* It might be easier to not to pass {{isStriped}} to {{verifyBlockPlacement}}. 
Instead, have {{BlockPlacementPolicyRackFaultTolerant}} implement 
{{verifyBlockPlacement}} which will use {{numberOfReplicas}} as the 
{{minRacks}}. This also makes the patch more applicable to branch-2 which 
doesn't have striped EC.
* The current patch doesn't apply to branch-2. If you agree with the above 
changes, could you try out if it applies to branch-2? If it doesn't apply, you 
will need to provide a separate patch for branch-2 later.
* A general question about striped EC. It uses "# of racks >= # of data blocks" 
to check if a given block has enough racks. But what if "# of racks for the 
whole cluster < # of data blocks"? Say we use RS(6,3) and the cluster has 5 
racks. The write operation will spread the 9 blocks to 5 racks and succeed. But 
it will fail the "enough racks" check later in BM? But that has nothing to with 
the refactoring work here. I just want to bring it up in case others can chime 
in.


> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, 
> HDFS-8647-003.patch, HDFS-8647-004.patch, HDFS-8647-004.patch, 
> HDFS-8647-005.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy 
> such as upgrade domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as 
> useDelHint, blockHasEnoughRacks. That means when we have new block placement 
> policy, we need to modify BlockManager to account for the new policy. Ideally 
> BlockManager should ask BlockPlacementPolicy object instead. That will allow 
> us to provide new BlockPlacementPolicy without changing BlockManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to