[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Attachment: continuousmovement.pdf
Attaching a brief design document to explain the problem, solution and a
proposed high level design. Requesting comments on this.
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: continuousmovement.pdf
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as
any (source , target) slot is available instead of waiting for all the
scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Attachment: (was: continuousmovement.pdf)
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: continuousmovement.pdf
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as
any (source , target) slot is available instead of waiting for all the
scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Attachment: continuousmovement.pdf
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: continuousmovement.pdf
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as
any (source , target) slot is available instead of waiting for all the
scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-7197
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as
any (source , target) slot is available instead of waiting for all the
scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Summary: Reduce impact of laggards on Mover (was: Reduce impact of
laggards on move)
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |=
Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The processPath will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead f waiting for all the scheduled
moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Description:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead of waiting for all the scheduled
moves to finish.
was:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |=
Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The processPath will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead f waiting for all the scheduled
moves to finish.
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead of waiting for all the scheduled
moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Description:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
(source , target) slot is available instead of waiting for all the scheduled
moves to finish.
was:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead of waiting for all the scheduled
moves to finish.
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
(source , target) slot is available instead of waiting for all the scheduled
moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7461) Reduce impact of laggards on Mover
[
https://issues.apache.org/jira/browse/HDFS-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7461:
---
Description:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as any
(source , target) slot is available instead of waiting for all the scheduled
moves to finish.
was:
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
(source , target) slot is available instead of waiting for all the scheduled
moves to finish.
Reduce impact of laggards on Mover
--
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The _processPath_ will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled,
it will wait for ALL scheduled moves to finish in _waitForMoveCompletion_.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the next moves as soon as
any (source , target) slot is available instead of waiting for all the
scheduled moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
