[ https://issues.apache.org/jira/browse/HDFS-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363289#comment-17363289 ]
Hadoop QA commented on HDFS-15150: ---------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 37s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} branch-2.10 Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 18s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 51s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 58s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 9s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 6s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 24s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 14m 37s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 2m 2s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html{color} | {color:red} hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings. {color} | | {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 2m 36s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant spotbugs warnings. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 20s{color} | {color:green}{color} | {color:green} the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 20s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 15s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 15s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 5s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m 35s{color} | {color:green}{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 39s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 18s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 36s{color} | {color:green}{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}218m 43s{color} | {color:black}{color} | {color:black}{color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.log.TestLogLevel | | | hadoop.hdfs.TestDecommission | | | hadoop.hdfs.TestPread | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys | | | hadoop.hdfs.server.datanode.TestBlockRecovery | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/artifact/out/Dockerfile | | JIRA Issue | HDFS-15150 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13026820/HDFS-15150-branch-2.10.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml | | uname | Linux e6b24b28b3aa 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | branch-2.10 / f22ff63 | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 | | Test Results | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/testReport/ | | Max. process+thread count | 2119 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . | | Console output | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/621/console | | versions | git=2.7.4 maven=3.3.9 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. > Introduce read write lock to Datanode > ------------------------------------- > > Key: HDFS-15150 > URL: https://issues.apache.org/jira/browse/HDFS-15150 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 3.3.0 > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15150-branch-2.10.001.patch, HDFS-15150.001.patch, > HDFS-15150.002.patch, HDFS-15150.003.patch > > > HDFS-9668 pointed out the issues around the DN lock being a point of > contention some time ago, but that Jira went in a direction of creating a new > FSDataset implementation which is very risky, and activity on the Jira has > stalled for a few years now. Edit: Looks like HDFS-9668 eventually went in a > similar direction to what I was thinking, so I will review that Jira in more > detail to see if this one is necessary. > I feel there could be significant gains by moving to a ReentrantReadWrite > lock within the DN. The current implementation is simply a ReentrantLock so > any locker blocks all others. > Once place I think a read lock would benefit us significantly, is when the DN > is serving a lot of small blocks and there are jobs which perform a lot of > reads. The start of reading any blocks right now takes the lock, but if we > moved this to a read lock, many reads could do this at the same time. > The first conservative step, would be to change the current lock and then > make all accesses to it obtain the write lock. That way, we should keep the > current behaviour and then we can selectively move some lock accesses to the > readlock in separate Jiras. > I would appreciate any thoughts on this, and also if anyone has attempted it > before and found any blockers. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org