[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851597#comment-13851597 ] Hudson commented on HDFS-5431: -- FAILURE: Integrated in Hadoop-Yarn-trunk #425 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/425/]) HDFS-5431. Support cachepool-based limit management in path-based caching. (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1551651) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/CacheFlag.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolStats.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851710#comment-13851710 ] Hudson commented on HDFS-5431: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1616 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1616/]) HDFS-5431. Support cachepool-based limit management in path-based caching. (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1551651) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/CacheFlag.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolStats.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851817#comment-13851817 ] Hudson commented on HDFS-5431: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1642 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1642/]) HDFS-5431. Support cachepool-based limit management in path-based caching. (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1551651) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/CacheFlag.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolStats.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850771#comment-13850771 ] Hudson commented on HDFS-5431: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4900 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4900/]) HDFS-5431. Support cachepool-based limit management in path-based caching. (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1551651) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/CacheFlag.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolStats.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849744#comment-13849744
]
Andrew Wang commented on HDFS-5431:
---
Thanks for the reviews, v6 attached. Changes are only to CRM, so you can just
review that.
I thought about it, and went way back to the condition variable proposal. I got
rid of the object lock, favoring a new CRM lock with two cvars and some status
variables. The synchronization here is unfortunately still kind of complicated,
but I think overall more better.
I also want to note this isn't actually read-after-write consistency since the
{{wait}} methods just wait for the scan count to advance, not for all pending
updates to be flushed. I think this could be fixed (basically, if a scan is
ongoing and CRM is dirty, wait for scanCount to advance by 2 rather than 1),
but as this is also a performance tradeoff, I'd really like to just get this
committed and fix it later if necessary.
I also re-examined {{close}} since I was re-doing the synchronization. This
snippet looks like it makes the thread calling {{close}} wait for 60 seconds,
where the intent is probably to interrupt the CRM thread and make it cleanly
exit:
{code}
try {
if (this.isAlive()) {
this.join(6);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
{code}
If you agree, I'll file a follow-on to fix this up. It could affect HA state
transitions.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849796#comment-13849796
]
Colin Patrick McCabe commented on HDFS-5431:
Thanks. This looks a lot better with just one lock, and without the volatiles.
Do we still need both waitForRescan and waitForRescanInternal, given that one
just calls the other?
+1.
bq. I also re-examined close since I was re-doing the synchronization. This
snippet looks like it makes the thread calling close wait for 60 seconds, where
the intent is probably to interrupt the CRM thread and make it cleanly exit:
It probably should not have a timeout, but just use
{{Uninterruptibles#joinUninterruptibly}}. Otherwise we might get into a weird
state where close had been called, but the CRM thread is still running. The
use of {{Thread#isAlive}} shouldn't be required either. Of course, this is
just used in tests anyway, so I'm not too worried about it. But by all means,
file a follow-on.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849875#comment-13849875
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618975/hdfs-5431-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5735//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5735//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5735//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849989#comment-13849989 ] Colin Patrick McCabe commented on HDFS-5431: +1 pending jenkins. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch, hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch We should support cachepool-based limit management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850116#comment-13850116
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619015/hdfs-5431-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5738//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5738//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch, hdfs-5431-6.patch, hdfs-5431-7.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848765#comment-13848765
]
Colin Patrick McCabe commented on HDFS-5431:
bq. With the new EnumSet, I also added a version of add/modify directive
without a flags argument, since it's really annoying to put
EnumSet.noneOf(CacheFlag.class everywhere.
Yeah, I agree. That is the one thing I never liked about EnumSets. I suppose
we could declare a constant insider DFS like NONE, but it's probably not worth
it.
bq. Tweaked the kick behavior somewhat. We now try to do read-after-write
consistency with the dirty bit. add/modify with force is essentially a read and
a write.
Great.
bq. I don't like putting bytesOverlimit in CachePoolEntry because it feels
inconsistent with CachePoolStats. Left it as is, unless you feel strongly on
this one.
I think it's fine as-is. Avoiding duplication is good, but I think you are
right that putting it into entry would add too much complexity.
{code}
this.cachedBlocks = cacheManager.getCachedBlocks();
this.intervalMs = intervalMs;
+this.needsRefresh = true;
}
{code}
We don't need this here, since it's initialized in {{run}}.
{code}
synchronized (this) {
needsRefresh = true;
this.notifyAll();
}
{code}
I would move {{needsRefresh}} out of the synchronized block. Since it's
volatile, you don't need to protect it (and it's nice to make that clear).
+1 once those changes are addressed.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848783#comment-13848783
]
Colin Patrick McCabe commented on HDFS-5431:
I looked at this again, and I think I found a bug.
{code}
startTimeMs = curTimeMs;
mark = !mark;
rescan();
curTimeMs = Time.monotonicNow(); === POINT 1
needsRefresh = false;
endTimeMs.updateTime(curTimeMs);
LOG.info(Scanned + scannedDirectives + directive(s) and +
scannedBlocks + block(s) in + (curTimeMs - startTimeMs) + +
millisecond(s).);
}
{code}
What if the CRM thread gets to point 1 right before {{waitForRescan}} is
called? Perhaps {{waitForRescan}} will set the volatile boolean to true, but
it will be immediately overwritten by the CRM thread. And then
{{waitForRescan}} will have to wait a full 5 minutes (or whatever) for the next
update.
I think the best solution here is using an {{AtomicBoolean}}. Then the
original check in the loop can be a compare-and-swap which atomically replaces
true with false, avoiding the lost updates problem.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848323#comment-13848323
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618766/hdfs-5431-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5721//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5721//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch, hdfs-5431-5.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848212#comment-13848212
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618742/hdfs-5431-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.cli.TestCacheAdminCLI
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5718//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5718//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5718//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch,
hdfs-5431-4.patch
We should support cachepool-based limit management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846187#comment-13846187
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618372/hdfs-5431-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5702//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5702//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5702//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846198#comment-13846198
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618372/hdfs-5431-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5703//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5703//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5703//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846203#comment-13846203
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618372/hdfs-5431-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5704//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5704//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5704//console
This message is automatically generated.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846682#comment-13846682
]
Colin Patrick McCabe commented on HDFS-5431:
API:
* I realize this might be an annoying request -- I know redoing that API code
is a pain-- but shouldn't we use an {{EnumSet}} rather than a boolean in
{{addDirective}}? We've had so many problems with proliferating function
overloads in the past that I just lean towards starting with an {{EnumSet}}. I
also think it's more readable. Seeing {{addDirective(foo, FORCE)}} is a lot
clearer than {{addDirective(foo, true)}}.
CacheReplicationMonitor:
* It's good that we are now using
{{INodeFile#computeFileSizeNotIncludingLastUcBlock}} in
{{CacheReplicationMonitor}}.
* I think you have the right general idea here. But it would be nice to do
this in a more object-oriented way. How about something like this:
{code}
private static final class ScanTime {
private long timeMs;
synchronized void waitForTime(long time) {
while (true) {
if (timeMs = time) {
return;
}
this.wait();
}
}
synchronized void updateTime(long time) {
this.timeMs = time;
this.notifyAll();
}
}
{code}
Then CRM calls {{updateTime}} when a scan finishes. And
{{kickAndWaitForRescan}} calls {{waitForTime(Time.now())}} or something. That
way, you don't need volatile variables or variables of type Object floating
around-- it's all encapsulated in {{ScanTime}}.
Stats
* Rather than putting {{bytesOverlimit}} in {{CachePoolStats}}, why not add a
method in {{CachePoolEntry}} that just subtracts {{stats.bytesNeeded}} by
{{info.limit}}? It bloats the RPCs to transmit essentially the same
information twice, and also opens up the possibility of divergence if someone
updates one thing and not another.
* I don't think we should change
{{dfs.namenode.path.based.cache.refresh.interval.ms}}. We selected that value
earlier because we felt the cached items would not change that often. It seems
like the motivation here behind changing it is to make the case of adding a
directive with force=true be a better interface, but that seems like a hack.
Why not just have {{listCacheDirectives}} trigger a cache replication monitor
run and then wait for it to complete? In basically the same way addDirective
without force does now. That way we could support the use case of adding a
bunch of directives with {{addDirective(force)}} and then checking to see if we
went over with {{listDirectives}}.
{code}
throw new InvalidRequestException(Caching path + path + of size
+ stats.getBytesNeeded() + bytes at replication + replication
+ would exceed pool + pool.getPoolName() + 's remaining
+ capacity of + (pool.getLimit() - pool.getBytesNeeded())
+ bytes.);
{code}
This is just a small nit, but doesn't {{CacheDirectiveStats#getBytesNeeded}}
include the replication factor in it? It might be confusing to say the path
has size 30 when it's really a size 10 path with 3x replication. I'm not sure
what the remedy is here. You could stop talking about the replication (other
than to suggest turning it down). You could also divide by replication, but
that feels a little dirty... or is it?
* Moving the serialization stuff to {{FSImageSerialization}} seems like a good
idea-- thanks for that.
Tests
* I see you refactored TestCacheDirectives to avoid creating two DFSClusters.
Does this accomplish everything we wanted to do in HDFS-5564? That would be
good if so.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846732#comment-13846732
]
Andrew Wang commented on HDFS-5431:
---
Thanks Colin, I'll get to all the things you mentioned along with rebasing for
the HDFS-2832 changes. One note though:
On changing the refresh interval, I can see monitoring and apps polling
listPools/listDirectives pretty frequently. If the expected rate of external
triggers of CRM#refresh is higher than our own refresh interval, we might as
well just do it asynchronously at that expected rate rather than doing it
synchronously on every RPC. I know Cloudera Manager typically does these things
on a 1 minute tick, which is why I chose this value. A faster refresh interval
will also make caching more reactive, which is also nice, and we might be able
to get rid of the retry logic for resending caching commands too.
Another thing is the delay for updating the {{*Cached}} stats. It takes one
refresh to schedule caching work, and another refresh to update the stats.
Doing this incrementally means mapping blocks - pools or directives, which
probably isn't going to happen.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847059#comment-13847059 ] Colin Patrick McCabe commented on HDFS-5431: How about if we have a dirty bit that gets set when you add/modify/delete a pool/directive. A CRM run clears the bit. Then listStatus/listPools only blocks on a CRM run if that dirty bit is set. That way, if an operator adds a directive from the command-line, s/he can see the effective immediately with the next list operation. It's more intuitive. It also avoids problems with frequently called list comands. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845756#comment-13845756 ] Andrew Wang commented on HDFS-5431: --- The checkLimit flag makes sense to me, except I'd prefer force, or if you'd like it flipped, enforce or strict. This is pretty easy. I agree on synchronously waiting on the CRM in the listed scenarios, and a CV would be a good way of doing this. It's a bit complicated though, since I don't think we can get a FSN CV, especially with the new lock context in HDFS-5453 coming down the pipe I think kicking the CRM, releasing the FSN lock, waiting on the CRM CV, then regetting the FSN lock should be okay, but it might be simpler to just call into CRM directly to do the rescan. I'll try the CV version, but if it looks too messy, we can go with a direct call. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846101#comment-13846101
]
Andrew Wang commented on HDFS-5431:
---
Here's a new version. I made the suggested changes as follows:
- force RPC parameter, which forces a directive to be added or modified
regardless of pool limits. Propagated this out to CacheAdmin and redid all the
usages. When {{force=false}}, we synchronously wait for a CRM rescan before
checking limits for more accuracy.
- {{CRM#kickAndWaitForRescan}} tries to be fancy with volatile longs and a
condition variable, checking this would be appreciated
- Do incremental stats updates on add/remove/modify of a directive, with
caveats about consistency and usage of {{force}}
- Made it so CRM is only kicked for {{force=false}} operations and
modify/remove pool. I feel like the normal usage pattern will be to do a bunch
of caching commands at once, so it'd be bad to do rescans during this. Instead,
I lowered the default refresh time for CRM to 1 min, and maybe we should lower
it even more once we benchmark it.
- Rejiggered overlimit bytes to simply be Math.max(bytesNeeded-limit, 0) for a
pool, simpler.
support cachepool-based limit management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch, hdfs-5431-3.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844883#comment-13844883 ] Colin Patrick McCabe commented on HDFS-5431: I think we should make checking against the cache pool limit optional in addDirective and modifyDirective. Basically have a parameter like checkLimit. If checkLimit = false, we allow the addition/modification to happen without respect to the limit. If it is true, then we throw an exception if the proposed change would exceed the cache pool limits. One detail: we should update the bytesNeeded and filesAffected after addDirective/modifyDirective, so that multiple calls in a row have the expected semantics. It would also be good to ensure that an add plus a list delivered the correct result. A condition variable could be used to do this I think. As a bonus, this would avoid the need to kick the CRM thread after every add/modify/etc. We probably still want to kick it after deletes and cache pool add/mod/dels, though. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based limit management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844885#comment-13844885 ] Colin Patrick McCabe commented on HDFS-5431: bq. We probably still want to kick it after deletes and cache pool add/mod/dels, though. Actually, we could probably pull the same trick and update bytesNeeded and filesAffected before returning from a delete operation. This might lead to having more bytesCached than bytesNeeded, but that's ok. That's the normal scenario when uncaching takes a bit of time. support cachepool-based limit management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
