[
https://issues.apache.org/jira/browse/HDFS-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doris Gu updated HDFS-10604:
----------------------------
Description:
The biggest difference this feature will bring is *making blocks belong to the
same file to save in the same region(DN group).*
So the process will be:
1.Config DN groups, for example
bq.Region1：dn1,dn2,dn3
bq.Region2：dn4,dn5,dn6
bq.Region3：dn7,dn8,dn9,dn10
2.Client uploads a file, first analyze whether this file has any existed blocks:
bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
bq.ii)No:assign new blocks to a DN group which is chosen by some certain policy
to avoid imbalance.
3.Other related processes,including append,balancer etc. also need to modify as
well.
The benefit we wish is when some DNs are down at the same time, the number of
affected files(miss all replicas) is small.
But we are wondering if this is worth doing or not, or if there are problems we
haven't noticed.
was:
The biggest difference this feature will bring is *strong* making blocks belong
to the same file to save in the same region(DN group).*strong*
So the process will be:
1.Config DN groups, for example
bq.Region1：dn1,dn2,dn3
bq.Region2：dn4,dn5,dn6
bq.Region3：dn7,dn8,dn9,dn10
2.Client uploads a file, first analyze whether this file has any existed blocks:
bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
bq.ii)No:assign new blocks to a DN group which is chosen by some certain policy
to avoid imbalance.
3.Other related processes,including append,balancer etc. also need to modify as
well.
The benefit we wish is when some DNs are down at the same time, the number of
affected files(miss all replicas) is small.
But we are wondering if this is worth doing or not, or if there are problems we
haven't noticed.
> What about this?Group DNs and add DN groups--named region to HDFS model , use
> this region to instead of single DN when saving files.
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10604
> URL: https://issues.apache.org/jira/browse/HDFS-10604
> Project: Hadoop HDFS
> Issue Type: Wish
> Reporter: Doris Gu
>
> The biggest difference this feature will bring is *making blocks belong to
> the same file to save in the same region(DN group).*
> So the process will be:
> 1.Config DN groups, for example
> bq.Region1：dn1,dn2,dn3
> bq.Region2：dn4,dn5,dn6
> bq.Region3：dn7,dn8,dn9,dn10
> 2.Client uploads a file, first analyze whether this file has any existed
> blocks:
> bq.i)Yes:assign new blocks to the DN group where the existed blocks belong to.
> bq.ii)No:assign new blocks to a DN group which is chosen by some certain
> policy to avoid imbalance.
> 3.Other related processes,including append,balancer etc. also need to modify
> as well.
> The benefit we wish is when some DNs are down at the same time, the number of
> affected files(miss all replicas) is small.
> But we are wondering if this is worth doing or not, or if there are problems
> we haven't noticed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
