[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859509#comment-13859509 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1655/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859488#comment-13859488 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1630 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1630/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859439#comment-13859439 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Yarn-trunk #438 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/438/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859005#comment-13859005 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4936 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4936/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13858995#comment-13858995 ] Andrew Wang commented on HDFS-5636: --- I just updated the editsStored files, which should fix TestOEV. This is standard procedure whenever an edit changes, it was just forgotten in the initial commit. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855855#comment-13855855 ] Haohui Mai commented on HDFS-5636: -- Looking at the code, I think TestOfflineEditsViewer.testStore() has been covered by other test cases. I think we can delete the unit test. I created HDFS-5695 to clean up the code. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855817#comment-13855817 ] Jing Zhao commented on HDFS-5636: - The patch got committed without Jenkins +1? TestOfflineEditsViewer is still broken in trunk. We need to create a new jira to fix it. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855032#comment-13855032 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1619 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1619/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855020#comment-13855020 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1645/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854998#comment-13854998 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Yarn-trunk #428 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/428/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854989#comment-13854989 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4924 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4924/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854636#comment-13854636 ] Colin Patrick McCabe commented on HDFS-5636: +1. thanks, Andrew. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854417#comment-13854417 ] Colin Patrick McCabe commented on HDFS-5636: I think forcing users to do {{directiveBuilder.setExpiry(Expiry.newRelative(CachePoolInfo.RELATIVE_EXPIRY_NEVER))}} to set "no expiration time" is kind of gross. Why does the user need to know how we're representing "never" in the protobuf? What about having clearMaxRelativeExpiryMs, etc. functions in CacheDirectiveInfo#Builder and CachePoolInfo#Builder so people don't have to wrestle with our internal constants? Maybe all it has to do is set the field to null. I also feel like MAX_RELATIVE_EXPIRY_MS should be a constant in Expiration, and enforced in Expiration. You fixed the overflow for your code path-- but let's fix it for every code path, rather than potentially letting users overflow us with wacky values. Change to TestFsDatasetCache#BLOCK_SIZE seems unrelated to the other changes. would prefer to do this separately > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854278#comment-13854278 ] Andrew Wang commented on HDFS-5636: --- Thanks for the review Colin, fixed the comment and functionized CacheAdmin, the rest inline: bq. I think we should have a constant MAX_TTL_MS... I added this, the enforcement was already in place, so this was just adding another variable and tweaking usages a bit. bq. Then we should not accept either relative or absolute limits... I agree we should evaluate both relative and absolute limits against the same relative max, but this means when converting to an absolute expiration for the fsimage, the value could be greater than the relative max. Hope that's suitable. bq. Can we have a static Expiration.NEVER final constant...? I'm not sure that's much better, since it conflates Expiration's ability to do absolute and relative expirations with this, which is just a relative. It's also already the case that users need to use big longs for {{Expiration.newRelative(long)}}, so it's not that different from the Java API perspective. CacheAdmin wraps this up nicely of course. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853471#comment-13853471 ] Hadoop QA commented on HDFS-5636: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619642/hdfs-5636-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5773//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5773//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5773//console This message is automatically generated. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-5636-1.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853257#comment-13853257 ] Colin Patrick McCabe commented on HDFS-5636: I think we should have a constant MAX_TTL_MS which is Long.MAX_VALUE / 4. Then we should not accept either relative or absolute limits which are higher than that. As you mentioned, these may cause overflows. We should forbid them the same way we forbid negative times. Then you could define RELATIVE_EXPIRY_NEVER_MS in terms MAX_TTL_MS. We already encompass millions of years with MAX_VALUE / 4 milliseconds, so that limit should be fine... {code} @Nullable Long maxRelativeExpiry; {code} should have "ms" at the end to indicate units. Same goes for a lot of other variables here. {code} // Allow if pool does not have a max expiry set, // fail if the requested expiry is greater than the max if (relExpiryTime > maxRelativeExpiryTime) { throw new InvalidRequestException("Expiration " + expiry.toString() + " exceeds the max relative expiration time of " + maxRelativeExpiryTime + " ms."); } {code} This comment seems a bit confusing. Based on the first part, I looked for special handling for pools that don't have a max expiry, and didn't find it. Maybe it would be better to get rid of the first part of the comment, since "pools without a max expiry" are just pools with max expiry set to MAX, right? Can we have a static {{Expiration.NEVER}} final constant which is an instance of {{Expiration}}? Since Expiration is immutable, it would be easy, I think. Much nicer than making API users set the time to a huge value. {code} if (maxTtlString != null) { long maxTtl; if (maxTtlString.equalsIgnoreCase("never")) { maxTtl = CachePoolInfo.RELATIVE_EXPIRY_NEVER; } else { try { maxTtl = DFSUtil.parseRelativeTime(maxTtlString); } catch (IOException e) { System.err.println( "Error while parsing maxTtl value: " + e.getMessage()); return 1; } } info.setMaxRelativeExpiry(maxTtl); {code} This is duplicated a few times in CacheAdmin. Seems like it should be a function. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-5636-1.patch > > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852285#comment-13852285 ] Colin Patrick McCabe commented on HDFS-5636: It might better to leave existing cache directives as-is, similar to how setting a quota or changing permissions doesn't undo things done in the past. Aside from that, looks reasonable. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852261#comment-13852261 ] Andrew Wang commented on HDFS-5636: --- I actually thought about this a bit more, and in terms of user experience, I propose the following: * If an expiry is specified in the CDInfo, check it against the pool's max expiration and error if invalid. * If an expiry is not specified, use the pool's max expiration. By default, the max expiration is never/infinite. * If a pool is modified to a lower max expiration, for all directives in the pool, set their expiration to min(oldExpiry, newMaxExpiration). * If a directive is modified to a new pool and a new expiration is not specified, set its expiration to min(oldExpiration, maxExpirationOfNewPool). The modify behavior will be kinda warty unfortunately when listCachePools is combined with modifyPool, since the CachePoolInfos you get out will have the expiry set (so behavior #4 won't happen). It'll be fine in CacheAdmin though, which is probably good enough since setting up pools is probably happening through that. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848090#comment-13848090 ] Andrew Wang commented on HDFS-5636: --- I was thinking a hard error. It's better for users too since the default expiry is "never". > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848041#comment-13848041 ] Colin Patrick McCabe commented on HDFS-5636: OK, I misinterpreted the proposal here. I thought you wanted a single TTL which would be applied to all directives in the pool. Instead, you want a max (relative) TTL that can be set on anything added to the pool, at the time of addition. Are you proposing to silently add these TTLs when they're not present in the request? Or hard error when we try to add something with a TTL that's "too high"? The latter seems simpler. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841754#comment-13841754 ] Andrew Wang commented on HDFS-5636: --- The issue is that if a pool is hanging on to forgotten cached data, that prevents another pool from using it. This is true even with quotas/limits. It's somewhat fixed if we also add minimum reservations for pools, but even then you might want max-TTLs so that forgotten cache is returned to be used by fair share. Basically, right now TTLs are opt in, rather than opt out, and admins might sometimes instead want them to be opt out. There's an impedance mismatch here between admins and the users of a pool; just because a user has access to a cache pool, doesn't necessarily mean they should be able to do whatever they want with it because it might be bad for overall system performance. I see max-TTL as an admin-friendly feature, since it'll help them avoid manually cleaning up cache pools. A couple imagined use cases: * A scratch / temp cache pool with a low max TTL (say, 1 hr) and 0777 permissions, so all users can do some adhoc data exploration. The admin doesn't need to worry about constantly cleaning up forgotten directives. * When caching time-series data, you might only care about caching the last day of data. Thus, the admin could set a max TTL of 24H to enforce this. Sort of related, we might also want a command like {{hdfs cacheadmin -removeExpiredDirectives [-pool ]}} to help people clean up their expired directives. Maybe even a trash-like functionality where directives that have been expired for so long are automatically removed. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841653#comment-13841653 ] Colin Patrick McCabe commented on HDFS-5636: I'm not sure about this one. Generally different cache pools are associated with different users, and users don't "age out" in most of the clusters I'm familiar with. Let's see how people make use of the existing TTL stuff. We can always add this later if need be. > Enforce a max TTL per cache pool > > > Key: HDFS-5636 > URL: https://issues.apache.org/jira/browse/HDFS-5636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, namenode >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice for administrators to be able to specify a maximum TTL for > directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1#6144)