[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052109#comment-15052109
]
Vinayakumar B commented on HDFS-8578:
-------------------------------------
I think its a good idea to run all {{linkBlocks}} in parallel.
But I do believe patch will not work.
Because, during {{bpStorage.recoverTransitionRead}}, {{linkBlocks}} will be
made async, but while returning, immediately, rename of current-previous
happens, also it logs as upgrade completed, even before all links are created
and actual upgrade is over.
In my v14 patch, synchronization was removed on {{addStorageLocations}}
because, getDatanodeUuid() was also synchronized, and threads were getting
blocked because of this.
But now, since datanodeUuId is updated only once, we can remove synchronization
on this. and can make it volatile.
Will update the patch shortly.
> On upgrade, Datanode should process all storage/data dirs in parallel
> ---------------------------------------------------------------------
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Raju Bairishetti
> Assignee: Vinayakumar B
> Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-15.patch, HDFS-8578-branch-2.6.0.patch,
> HDFS-8578-branch-2.7-001.patch, HDFS-8578-branch-2.7-002.patch,
> HDFS-8578-branch-2.7-003.patch, h8578_20151210.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
> for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
