[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal1.png bal2.png > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: bal1.png, bal2.png, h8818_20150723.patch, > h8818_20150727.patch, HDFS-8818-branch-2.7.00.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.patch) > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8818_20150723.patch, h8818_20150727.patch, > HDFS-8818-branch-2.7.00.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.trunk.patch) > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8818_20150723.patch, h8818_20150727.patch, > HDFS-8818-branch-2.7.00.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.8.patch) > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8818_20150723.patch, h8818_20150727.patch, > HDFS-8818-branch-2.7.00.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal_revert.trunk.patch bal_revert.branch-2.patch bal_revert.branch-2.8.patch Attaching patches for revert of the main body of the change. It leaves config and logging changes as is. > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Kihwal Lee > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: bal_revert.branch-2.8.patch, bal_revert.branch-2.patch, > bal_revert.trunk.patch, h8818_20150723.patch, h8818_20150727.patch, > HDFS-8818-branch-2.7.00.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[
https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8818:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to branch-2.7. Pre-existing flaky tests in {{TestBalancer}} are being
addressed in HDFS-10859.
> Allow Balancer to run faster
>
>
> Key: HDFS-8818
> URL: https://issues.apache.org/jira/browse/HDFS-8818
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
> Attachments: HDFS-8818-branch-2.7.00.patch, h8818_20150723.patch,
> h8818_20150727.patch
>
>
> The original design of Balancer is intentionally to make it run slowly so
> that the balancing activities won't affect the normal cluster activities and
> the running jobs.
> There are new use case that cluster admin may choose to balance the cluster
> when the cluster load is low, or in a maintain window. So that we should
> have an option to allow Balancer to run faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8818: Fix Version/s: 2.7.4 > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: HDFS-8818-branch-2.7.00.patch, h8818_20150723.patch, > h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8818: Status: Patch Available (was: Reopened) > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8818-branch-2.7.00.patch, h8818_20150723.patch, > h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8818: Attachment: HDFS-8818-branch-2.7.00.patch > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8818-branch-2.7.00.patch, h8818_20150723.patch, > h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8818: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Release Note: Add a new conf "dfs.balancer.max-size-to-move" so that Balancer.MAX_SIZE_TO_MOVE becomes configurable. Status: Resolved (was: Patch Available) I have committed this. > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0 > > Attachments: h8818_20150723.patch, h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8818: -- Issue Type: Sub-task (was: Improvement) Parent: HDFS-8825 > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h8818_20150723.patch, h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8818: -- Attachment: h8818_20150727.patch h8818_20150727.patch: fixes the findbugs warning and some trailing whitespaces. > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h8818_20150723.patch, h8818_20150727.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8818: -- Status: Patch Available (was: Open) > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h8818_20150723.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8818) Allow Balancer to run faster
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8818: -- Attachment: h8818_20150723.patch h8818_20150723.patch: changes the global moveExecutor to per datanode moveExecutor(s) and changes MAX_SIZE_TO_MOVE to be configurable. > Allow Balancer to run faster > > > Key: HDFS-8818 > URL: https://issues.apache.org/jira/browse/HDFS-8818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h8818_20150723.patch > > > The original design of Balancer is intentionally to make it run slowly so > that the balancing activities won't affect the normal cluster activities and > the running jobs. > There are new use case that cluster admin may choose to balance the cluster > when the cluster load is low, or in a maintain window. So that we should > have an option to allow Balancer to run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
