[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841183#comment-13841183 ] Hudson commented on HDFS-5514: -- FAILURE: Integrated in Hadoop-Yarn-trunk #413 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/413/]) Neglected to add new file in HDFS-5514 (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548167) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548161) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.4.0 Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841250#comment-13841250 ] Hudson commented on HDFS-5514: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1630 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1630/]) Neglected to add new file in HDFS-5514 (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548167) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548161) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.4.0 Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841267#comment-13841267 ] Hudson commented on HDFS-5514: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1604 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1604/]) Neglected to add new file in HDFS-5514 (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548167) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548161) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.4.0 Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13840177#comment-13840177 ] Hudson commented on HDFS-5514: -- FAILURE: Integrated in Hadoop-trunk-Commit #4833 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4833/]) HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548161) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13840193#comment-13840193 ] Hudson commented on HDFS-5514: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4834 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4834/]) Neglected to add new file in HDFS-5514 (daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1548167) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.4.0 Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837952#comment-13837952 ] Daryn Sharp commented on HDFS-5514: --- Yes, I meant coarse, nice catch. :) I think I can quickly explain the basic approach w/o a couple page doc, if not, let me know and I'll write something up. The reason for creating a separate class is to move towards providing a pluggable lock context for ops to use. The default will be coarse as it is today to not destabilize the NN. The general principle is changing the pattern of: {code} readLock(); try { ... } finally { readUnlock(); } {code} to something more like: {code} LockContext lockContext = fsLock.getLockContext(LockState.READ); try { ... lockContext.readLock(path); lockContext.writeLock(path); lockContext.writeLockParent(path); ... } finally { lockContext.unlock(); } {code} Use of the context is optional. The existing fsn readLock/writeLock methods will continue to exist to avoid changing anything but {{FSNamesystem}}. This also means not every fsn method needs to be converted immediately. For a coarse lock context, the initial lock state is applied to the global rw lock now wrapped by the {{FSNamesystemLock}} in this patch. The path locking methods are no-ops. Existing fsn readLock/writeLock methods will use the same global lock - which is how I localize the changes to the fsn. For the initial finer grain lock context, which I'll post in yet another future subtask, the context's initial lock state is effectively always a read lock on the global lock to prevent safemode/HA transitions. The path locking methods associate unique rw locks with inodes on a demand basis (the NN won't create or maintain a lock for every single inode). These locks are tracked in the context. However, these are only implementation details of a specific context for finer grain locking. Other implementations may do something completely different. I hope to use a lock-free scoreboard in the future to control/schedule which handlers are allowed to execute - but again just an implementation detail. The initial prototype I'm hoping to implement via the parent jira will not require lock changes to classes other than {{FSNamesystem}} as described above. I just need the {{FSNameSystemLock}} abstraction in this jira, and the addition of the aforementioned path locking apis I'll post in another jira. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837962#comment-13837962 ] Daryn Sharp commented on HDFS-5514: --- [~cnauroth] Regarding #2, that's technically it's a pre-existing gap in test coverage if not already there. Actually, those two methods were recently added by Kihwal's getContentSummary change and don't appear to be used by anything else. I'll probably discard those methods when I adapt the content summary to work with a lock context so there's likely not much value in adding transient tests but I will certainly add them if you like. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837980#comment-13837980 ] Chris Nauroth commented on HDFS-5514: - Thanks, Daryn. On #2, I think verifying that the lock implementation supports reentrancy is the more significant thing than the counts returned from {{getReadHoldCount}} and {{getWriteHoldCount}}. (i.e. Locking twice in a row is expected to succeed with no exception thrown.) I'll defer to you on whether or not you think that's a valuable test here. If not, then I think the only remaining thing is the {{coarseLock}} rename. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837991#comment-13837991 ] Daryn Sharp commented on HDFS-5514: --- I completely agree that reentrancy testing is critical for a new lock implementation. In this jira, there is no change in existing functionality other than the fsn lock becomes a new class that delegates to a lock. Although, the patch has gone stale and I need to update it to add the hold count methods to the fsnl. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838184#comment-13838184 ] Hadoop QA commented on HDFS-5514: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12616827/HDFS-5514.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5625//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5625//console This message is automatically generated. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch, HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823912#comment-13823912 ] Colin Patrick McCabe commented on HDFS-5514: Thanks for working on this, Daryn. It is definitely going to be a big improvement over what we have now. Can we have a design document about coarse-grained locking? Nothing fancy, just a page or two. Right now I don't understand how this fits into the bigger design. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822860#comment-13822860 ] Hadoop QA commented on HDFS-5514: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613879/HDFS-5514.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5434//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5434//console This message is automatically generated. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823172#comment-13823172 ] Chris Nauroth commented on HDFS-5514: - Hi, Daryn. It looks like a good change. A couple of comments: # Was {{courseLock}} meant to be named {{coarseLock}}? # Should we add tests that lock twice in a row and then assert that {{getReadHoldCount}} or {{getWriteHoldCount}} is 2? I'm pretty sure we have code paths that do this and rely on the reentrancy. Unless you anticipate discarding reentrancy (and changing the code paths that rely on it), then it would be a good thing to catch with a quick test. FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5514) FSNamesystem's fsLock should allow custom implementation
[ https://issues.apache.org/jira/browse/HDFS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823259#comment-13823259 ] Jonathan Eagles commented on HDFS-5514: --- Hi, Daryn. I like what is going on here. One minor nit in addition to the above feedback. Please remove the unnecessary _import java.util.concurrent.locks.ReadWriteLock_ in TestFSNamesystem.java FSNamesystem's fsLock should allow custom implementation Key: HDFS-5514 URL: https://issues.apache.org/jira/browse/HDFS-5514 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-5514.patch Changing {{fsLock}} from a {{ReentrantReadWriteLock}} to an API compatible class that encapsulates the rwLock will allow for more sophisticated locking implementations such as fine grain locking. -- This message was sent by Atlassian JIRA (v6.1#6144)