[ https://issues.apache.org/jira/browse/HDFS-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381522#comment-17381522 ]
Ahmed Hussein commented on HDFS-15160: -------------------------------------- Backporting HDFS-15160 to branch-3.2 * Created [PR-3200|https://github.com/apache/hadoop/pull/3200] for branch-3.2 ** First: HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell `git cherry-pick -x 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718` *** main conflicts: **** `FsDatasetImpl.java.deepCopyReplica` **** `FoldedTreeSet` is not used in branch-3.2 *** testing: ``` mvn test -Dtest=TestWriteToReplica,TestReplicaMap,\ TestProvidedImpl,TestFsVolumeList,TestInterDatanodeProtocol ``` ** Second: HDFS-15160 `git cherry-pick -x 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718` *** main conflicts: **** `ReplicaMap.java` **** `FsDatasetImpl.java.deepCopyReplica` **** `DirectoryScanner.scan()` conflict in the entire loop. HDFS-15415. _Reduce locking in Datanode DirectoryScanner_. Which is already committed to branch-2.10 removes what HDFS-15160 has added. So, I skipped that. ** Third: HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) and HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig (#2679) `git cherry-pick -x 98097b8f19789605b9697f6a959da57261e0fe19 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4` *** main conflicts: Java imports > ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl > methods should use datanode readlock > ----------------------------------------------------------------------------------------------------------- > > Key: HDFS-15160 > URL: https://issues.apache.org/jira/browse/HDFS-15160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 3.3.0 > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15160-branch-3.3-001.patch, HDFS-15160.001.patch, > HDFS-15160.002.patch, HDFS-15160.003.patch, HDFS-15160.004.patch, > HDFS-15160.005.patch, HDFS-15160.006.patch, HDFS-15160.007.patch, > HDFS-15160.008.patch, HDFS-15160.branch-3-3.001.patch, > image-2020-04-10-17-18-08-128.png, image-2020-04-10-17-18-55-938.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Now we have HDFS-15150, we can start to move some DN operations to use the > read lock rather than the write lock to improve concurrence. The first step > is to make the changes to ReplicaMap, as many other methods make calls to it. > This Jira switches read operations against the volume map to use the readLock > rather than the write lock. > Additionally, some methods make a call to replicaMap.replicas() (eg > getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result > in a read only fashion, so they can also be switched to using a readLock. > Next is the directory scanner and disk balancer, which only require a read > lock. > Finally (for this Jira) are various "low hanging fruit" items in BlockSender > and fsdatasetImpl where is it fairly obvious they only need a read lock. > For now, I have avoided changing anything which looks too risky, as I think > its better to do any larger refactoring or risky changes each in their own > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org