[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549883#comment-14549883
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549882#comment-14549882
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549930#comment-14549930
]
Leitao Guo commented on HDFS-7692:
--
Sorry it's my mistake to comment many times here! It seems that my network
condition is not very good now...
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549901#comment-14549901
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549898#comment-14549898
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549907#comment-14549907
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549903#comment-14549903
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549905#comment-14549905
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549896#comment-14549896
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549894#comment-14549894
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549887#comment-14549887
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549889#comment-14549889
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549908#comment-14549908
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549904#comment-14549904
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549906#comment-14549906
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549892#comment-14549892
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549885#comment-14549885
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549890#comment-14549890
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549886#comment-14549886
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549888#comment-14549888
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Attachment: HDFS-7692.02.patch
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549899#comment-14549899
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549902#comment-14549902
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549893#comment-14549893
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549900#comment-14549900
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549895#comment-14549895
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549897#comment-14549897
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549913#comment-14549913
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549910#comment-14549910
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549914#comment-14549914
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your comments, please have a check of the new patch.
1.In DataStorage#recoverTransitionRead, log the InterruptedException and
rethrow it as InterruptedIOException;
2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then
let the test case fail;
3.The multithread in DataStorage#addStorageLocations() is for one specific
namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is
creating one thread pool for each namespace. Not change here.
4.Re-phrase the parameter successVolumes.
[~szetszwo],thanks for your comments, please have a check of the new patch.
1. InterruptedException re-thrown as InterruptedIOException;
2. I think it's a good idea to log the upgrade progress for each dir, but so
far, we can not get the progress easily from the current api. Do you think it's
necessary to file a new jira to follow this?
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Labels: BB2015-05-TBR
Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310587#comment-14310587
]
Leitao Guo commented on HDFS-7692:
--
[~eddyxu], thanks for your review and comments!
When running tests after patching, I got NullPointerException at this line,
so I add the check of null != datanode.getConf() here.
{code} Executors.newFixedThreadPool(null != datanode.getConf() ? datanode
493 .getConf().getInt(
494 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY,
495
DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT)
496 : dataDirs.size()); {code}
I will update the patch according to your comments, thanks!
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Attachments: HDFS-7692.01.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310591#comment-14310591
]
Leitao Guo commented on HDFS-7692:
--
When upgrading before the patch, I find there is high cpu utilization (~90%) in
our cluster , so I think we'd better control the num of threads here. I will
have a test verify this.
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Attachments: HDFS-7692.01.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310592#comment-14310592
]
Leitao Guo commented on HDFS-7692:
--
[~szetszwo] thanks for your comments, I will update the patch asap.
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Assignee: Leitao Guo
Attachments: HDFS-7692.01.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Attachment: HDFS-7692.01.patch
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Attachments: HDFS-7692.01.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Release Note: Please help review the patch. Thanks!
Status: Patch Available (was: Open)
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
Attachments: HDFS-7692.01.patch
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Description:
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes. MultiThread dataDirs analyzing should be supported here to speedup
upgrade.
was:
{code:title=DataStorage#addStorageLocations...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes. MultiThread dataDirs analyzing should be supported here to speedup
upgrade.
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Summary: DataStorage#addStorageLocations(...) should support MultiThread to
speedup the upgrade of block pool at multi storage directories. (was:
BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread
to speedup the upgrade of block pool at multi storage directories.)
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Description:
{code:title=DataStorage#addStorageLocations...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes. MultiThread dataDirs analyzing should be supported here to speedup
upgrade.
was:
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes. MultiThread dataDirs analyzing should be supported here to speedup
upgrade.
DataStorage#addStorageLocations(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
--
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=DataStorage#addStorageLocations...)|borderStyle=solid}
for (StorageLocation dataDir : dataDirs) {
File root = dataDir.getFile();
... ...
bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs,
startOpt);
addBlockPoolStorage(bpid, bpStorage);
... ...
successVolumes.add(dataDir);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7692) BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295013#comment-14295013
]
Leitao Guo commented on HDFS-7692:
--
When we upgrade HDFS from 2.0.0 to 2.5.0 with datanodes have 12 dataDirs and
each dataDir has 2.5TB data, upgrade of block pool at each dataDir cost us
about 25 minutes and the total time is 5 hours. This is really time consuming,
especially when datanodes have more dataDirs and more data.
I will submit the patch later after some tests.
BlockPoolSliceStorage#loadBpStorageDirectories(...) should support
MultiThread to speedup the upgrade of block pool at multi storage directories.
-
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7692) BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
Leitao Guo created HDFS-7692:
Summary: BlockPoolSliceStorage#loadBpStorageDirectories(...)
should support MultiThread to speedup the upgrade of block pool at multi
storage directories.
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7692) BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leitao Guo updated HDFS-7692:
-
Description:
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes. MultiThread dataDirs analyzing should be supported here to speedup
upgrade.
was:
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which is
really time consuming when upgrading HDFS with datanodes have dozens of large
volumes.
BlockPoolSliceStorage#loadBpStorageDirectories(...) should support
MultiThread to speedup the upgrade of block pool at multi storage directories.
-
Key: HDFS-7692
URL: https://issues.apache.org/jira/browse/HDFS-7692
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.5.2
Reporter: Leitao Guo
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
if (containsStorageDir(dataDir)) {
throw new IOException(
BlockPoolSliceStorage.recoverTransitionRead: +
attempt to load an used block storage: + dataDir);
}
StorageDirectory sd =
loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
succeedDirs.add(sd);
}
{code}
In the above code the storage directories will be analyzed one by one, which
is really time consuming when upgrading HDFS with datanodes have dozens of
large volumes. MultiThread dataDirs analyzing should be supported here to
speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
