[
https://issues.apache.org/jira/browse/HDFS-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610827#comment-17610827
]
ASF GitHub Bot commented on HDFS-16785:
---------------------------------------
ZanderXu commented on PR #4945:
URL: https://github.com/apache/hadoop/pull/4945#issuecomment-1261833655
@MingXiangLi @Hexiaoqiao Sir, thanks for the warm discussions.
> 1、When fsVolume.getVolumeMap() scan the bock from disk to add block
metadata，it may add new block metadata when another thread add block.
This volume is still not added into the FsVolumeList here, means another
thread can not add new block into this volume? So this case not exists?
> 2、How we deal conflict when remove BlockPool operating occur at same time.
RemoveBlockPool just remove the blocks in the memory replicasMap, will not
delete blocks on the disk. So remove block pool operation will not affect
scanning disk. Let's see some stages:
Case 1: It's ok.
1. fsVolume.addBlockPool();
2. volumeMap.cleanUpBlockPool(bpid);
3. volumes.removeBlockPool(bpid, blocksPerVolume); Here will remove the
blockPoolSlice
4. fsVolume.getVolumeMap(); here will throw IOException, because the
blockPoolSlice is null.
Case2:
1. addVolume get NamespaceInfo
2. volumeMap.cleanUpBlockPool(bpid);
3. volumes.removeBlockPool(bpid, blocksPerVolume); Here will remove the
blockPoolSlice
4. fsVolume.addBlockPool();
5. fsVolume.getVolumeMap(); here will be ok
6. activateVolume(); here will add the removed bpid into the replicamap
again. Maybe is not expected. But it's another problem, should be fixed by
other PR.
From my point, scanning the disk with a lock for a long time is not allowed.
If there are some conflict here, we can find them and fix by other solution.
> DataNode hold BP write lock to scan disk
> ----------------------------------------
>
> Key: HDFS-16785
> URL: https://issues.apache.org/jira/browse/HDFS-16785
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
>
> When patching the fine-grained locking of datanode, I found that `addVolume`
> will hold the write block of the BP lock to scan the new volume to get the
> blocks. If we try to add one full volume that was fixed offline before, i
> will hold the write lock for a long time.
> The related code as bellows:
> {code:java}
> for (final NamespaceInfo nsInfo : nsInfos) {
> String bpid = nsInfo.getBlockPoolID();
> try (AutoCloseDataSetLock l = lockManager.writeLock(LockLevel.BLOCK_POOl,
> bpid)) {
> fsVolume.addBlockPool(bpid, this.conf, this.timer);
> fsVolume.getVolumeMap(bpid, tempVolumeMap, ramDiskReplicaTracker);
> } catch (IOException e) {
> LOG.warn("Caught exception when adding " + fsVolume +
> ". Will throw later.", e);
> exceptions.add(e);
> }
> } {code}
> And I noticed that this lock is added by HDFS-15382, means that this logic is
> not in lock before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
