[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153250#comment-15153250
]
Chris Trezzo commented on HDFS-8578:
------------------------------------
Thanks [~szetszwo]! I now see that the boolean ensures that properties are only
written in the case where they have not already been written.
One small nit that might make it clearer with respect to the boolean:
In DataStorage.loadStorageDirectory, maybe pull the write properties into the
if statement like so:
{code}
//DataStorage.loadStorageDirectory
if (!doTransition(sd, nsInfo, startOpt, callables, datanode.getConf())){
// 3. Update successfully loaded storage.
setServiceLayoutVersion(getServiceLayoutVersion());
writeProperties(sd);
}
return sd;
{code}
Outside of this small nit, the patch looks good to me. +1.
Thanks [~szetszwo] and [~vinayrpet] for all the work.
> On upgrade, Datanode should process all storage/data dirs in parallel
> ---------------------------------------------------------------------
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Raju Bairishetti
> Assignee: Vinayakumar B
> Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-15.patch, HDFS-8578-16.patch, HDFS-8578-17.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch, HDFS-8578-branch-2.7-003.patch,
> h8578_20151210.patch, h8578_20151211.patch, h8578_20151211b.patch,
> h8578_20151212.patch, h8578_20151213.patch, h8578_20160117.patch,
> h8578_20160128.patch, h8578_20160128b.patch, h8578_20160216.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
> for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
