[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3566: - Hadoop Flags: Reviewed +1 the patch looks good. Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Assignee: Sumadhur Reddy Bolli Fix For: 1-win Attachments: AzureBlockPlacementPolicy.pdf, azurepolicy-branch-1-win.patch Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3566: - Resolution: Fixed Status: Resolved (was: Patch Available) I have committed this. Thanks, Suma! Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Assignee: Sumadhur Reddy Bolli Fix For: 1-win Attachments: AzureBlockPlacementPolicy.pdf, azurepolicy-branch-1-win.patch Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-3566: --- Attachment: azurepolicy-branch-1-win.patch submitted a patch for azure policy in branch-1-win Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Assignee: Sumadhur Reddy Bolli Fix For: 1-win Attachments: azurepolicy-branch-1-win.patch Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-3566: --- Fix Version/s: 1-win Status: Patch Available (was: Open) Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Assignee: Sumadhur Reddy Bolli Fix For: 1-win Attachments: azurepolicy-branch-1-win.patch Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-3566: --- Attachment: AzureBlockPlacementPolicy.pdf Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Assignee: Sumadhur Reddy Bolli Fix For: 1-win Attachments: AzureBlockPlacementPolicy.pdf, azurepolicy-branch-1-win.patch Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3566) Custom Replication Policy for Azure
[ https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-3566: --- Target Version/s: 1-win (was: 1.1.0) Custom Replication Policy for Azure --- Key: HDFS-3566 URL: https://issues.apache.org/jira/browse/HDFS-3566 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Sumadhur Reddy Bolli Azure has logical concepts like fault and upgrade domains. Each fault domain spans multiple upgrade domains and each upgrade domain spans multiple fault domains. Machines are spread typically evenly across both fault and upgrade domains. Fault domain failures are typically catastrophic/unplanned failures and data loss possibility is high. An upgrade domain can be taken down by azure for maintenance periodically. Each time an upgrade domain is taken down a small percentage of machines in the upgrade domain(typically 1-2%) are replaced due to disk failures, thus losing data. Assuming the default replication factor 3, any 3 data nodes going down at the same time would mean potential data loss. So, it is important to have a policy that spreads replicas across both fault and upgrade domains to ensure practically no data loss. The problem here is two dimensional and the default policy in hadoop is one-dimensional. This policy would spread the datanodes across atleast 2 fault domains and three upgrade domains to prevent data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
