[
https://issues.apache.org/jira/browse/HDFS-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000790#comment-17000790
]
Stephen O'Donnell commented on HDFS-15000:
------------------------------------------
In the createRbw() method, there is still a race condition with your method.
The current implementation checks the volumeMap to ensure the block does not
already exist to avoid a duplicate getting created. If you drop the lock before
updating the volume map, then a duplicate can come in during the async IO phase
but before the volume map is updated.
I agree my suggestion has the flaw you pointed out, that the volume map would
now have a reference to a block on disk which does not exist yet and that could
be a problem too.
With your idea, why do we need the IO to be async? Could it be simplified to:
1. Existing steps in the method
2. Drop the lock, perform the IO in the current thread
3. Re-take the lock and perform the remaining steps (update volume map)
The current thread cannot do anything else while it waits for the IO anyway, so
perhaps it is simpler to just to the IO on that thread.
We still need to think of a way to keep things consistent when the lock is
released, which is the hard part.
> Improve FsDatasetImpl to avoid IO operation in datasetLock
> ----------------------------------------------------------
>
> Key: HDFS-15000
> URL: https://issues.apache.org/jira/browse/HDFS-15000
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Xiaoqiao He
> Assignee: Aiphago
> Priority: Major
> Attachments: HDFS-15000.001.patch
>
>
> As HDFS-14997 mentioned, some methods in #FsDatasetImpl such as
> #finalizeBlock, #finalizeReplica, #createRbw includes IO operation in the
> datasetLock, It will block some logic when IO load is very high. We should
> reduce grain fineness or move IO operation out of datasetLock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
