[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215640#comment-15215640 ] Hadoop QA commented on HDFS-5242: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s {color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client: patch generated 3 new + 41 unchanged - 3 fixed = 44 total (was 44) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 18s {color} | {color:red} Patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 19m 47s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:fbe3e86 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12795795/HDFS-5242.patch | | JIRA Issue | HDFS-5242 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 47c6f2ffae11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | |
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549192#comment-14549192 ] Hadoop QA commented on HDFS-5242: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 49s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | | {color:green}+1{color} | javac | 7m 37s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 20s | The applied patch generated 56 new checkstyle issues (total was 0, now 56). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 36s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 0m 42s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | native | 3m 20s | Pre-build of native portion | | {color:green}+1{color} | hdfs tests | 0m 14s | Tests passed in hadoop-hdfs-client. | | | | 39m 32s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12733606/HDFS-5242.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / cdfae44 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/11035/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt | | hadoop-hdfs-client test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11035/artifact/patchprocess/testrun_hadoop-hdfs-client.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11035/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11035/console | This message was automatically generated. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Labels: BB2015-05-TBR Attachments: HDFS-5242.patch, HDFS-5242.patch, HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524748#comment-14524748 ] Hadoop QA commented on HDFS-5242: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12623196/HDFS-5242.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / f1a152c | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/10587/console | This message was automatically generated. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch, HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524829#comment-14524829 ] Hadoop QA commented on HDFS-5242: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12623196/HDFS-5242.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / f1a152c | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/10622/console | This message was automatically generated. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch, HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872661#comment-13872661 ] Hadoop QA commented on HDFS-5242: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623196/HDFS-5242.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestPersistBlocks {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5884//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5884//console This message is automatically generated. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch, HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850805#comment-13850805 ] Suresh Srinivas commented on HDFS-5242: --- Alternatively, this can be done in a lockless fashion using AtomicReference. Let me know if you want me to upload a patch for that. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851206#comment-13851206 ] Daryn Sharp commented on HDFS-5242: --- An atomic reference would be fine but I'm not sure it adds value. The setter is called when adding the node to the network topology - after which, it is never changed. Another thread can't get the instance w/o going through the topology, and the instance isn't in the topology until after the network location is set. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851208#comment-13851208 ] Daryn Sharp commented on HDFS-5242: --- For the impact, it's been so long I don't have numbers handy. I was profiling a 85 node cluster being hammered by s-live and {{getNetworkLocation}} was showing up as a hot spot. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849771#comment-13849771 ] Suresh Srinivas commented on HDFS-5242: --- I am not sure how much performance impact this synchronization has. Do you have have numbers about the impact of this change? bq. As mentioned in the first comment, the set is called only once at DN registration so there is actually no race to avoid. @Daryn, I think if a setter is called by one thread and getter is called by another, then synchronized is still needed. Without that it is possible that another thread may never see the changed value. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845675#comment-13845675 ] Kihwal Lee commented on HDFS-5242: -- +1 for the patch. However, the contention might have seen unusually high if only a small number of data nodes were involved. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13840205#comment-13840205 ] Daryn Sharp commented on HDFS-5242: --- [~sureshms] Are you ok with this patch? Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822750#comment-13822750 ] Daryn Sharp commented on HDFS-5242: --- If one thread calls get concurrently with another thread calling set, then yes, the result is nondeterministic and the loser has a stale ref. However with or w/o sync, that's an unavoidable race. But... As mentioned in the first comment, the set is called only once at DN registration so there is actually no race to avoid. Thus the sync on the get causes unnecessary contention during normal operation. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1310#comment-1310 ] Suresh Srinivas commented on HDFS-5242: --- [~daryn], without synchronized access threads can hold stale references. Is that not the case here? Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5242) Reduce contention on DatanodeInfo instances
[ https://issues.apache.org/jira/browse/HDFS-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775927#comment-13775927 ] Hadoop QA commented on HDFS-5242: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604625/HDFS-5242.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5024//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5024//console This message is automatically generated. Reduce contention on DatanodeInfo instances --- Key: HDFS-5242 URL: https://issues.apache.org/jira/browse/HDFS-5242 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5242.patch Synchronization in {{DatanodeInfo}} instances causes unnecessary contention between call handlers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira