[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3912: -- Attachment: HDFS-3912.branch-1.patch Fixed the typo > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch, > HDFS-3912.009.patch, HDFS-3912-010.patch, HDFS-3912-branch-1.1-001.patch, > HDFS-3912-branch-1.patch, HDFS-3912.branch-1.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3912: -- Status: Open (was: Patch Available) Canceling patch to prevent Jenkins from running builds for branch-1 patches. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch, > HDFS-3912.009.patch, HDFS-3912-010.patch, HDFS-3912-branch-1.1-001.patch, > HDFS-3912-branch-1.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912-branch-1.patch The patch for branch-1. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch, > HDFS-3912.009.patch, HDFS-3912-010.patch, HDFS-3912-branch-1.1-001.patch, > HDFS-3912-branch-1.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912-010.patch HDFS-3912-branch-1.1-001.patch Patch for branch 1.1. Also did some cleanup for the test code in the patch for trunk. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch, > HDFS-3912.009.patch, HDFS-3912-010.patch, HDFS-3912-branch-1.1-001.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.009.patch Updated based on Suresh's comments. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch, > HDFS-3912.009.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.008.patch Addressed Nicolas's comments. Now we check if the stale interval is positive instead of the original warning msg. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch, HDFS-3912.008.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.007.patch The DataNode#heartbeatsDisabledForTests should be declared as volatile, and for new test cases in TestReplicaitonPolicy, instead of waiting, I explicitly call the heartbeatCheck() method. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch, HDFS-3912.007.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.006.patch Upload the patch with minor updates. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: nkeywal > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Assignee: Jing Zhao (was: nkeywal) Affects Version/s: 3.0.0 Status: Patch Available (was: Open) > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch, > HDFS-3912.006.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.005.patch Thanks for the comments Suresh! I've addressed most of the comments. I will create separate jiras for DatanodeStatics and metrics issues as well. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: nkeywal > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.004.patch Removed redundant test cases and correct part of the comments in the test. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.003.patch Moved stalenode-related information from FSNameSystem back to DatanodeManager. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.002.patch Some cleanup for the patch. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3912: Attachment: HDFS-3912.001.patch And some initial patch for the simpler solution. > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-3912.001.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
