[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859439#comment-13859439 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Yarn-trunk #438 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/438/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859488#comment-13859488 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1630 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1630/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859509#comment-13859509 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1655/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858995#comment-13858995 ] Andrew Wang commented on HDFS-5636: --- I just updated the editsStored files, which should fix TestOEV. This is standard procedure whenever an edit changes, it was just forgotten in the initial commit. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859005#comment-13859005 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4936 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4936/]) Add updated editsStored files missing from initial HDFS-5636 commit. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554293) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855817#comment-13855817 ] Jing Zhao commented on HDFS-5636: - The patch got committed without Jenkins +1? TestOfflineEditsViewer is still broken in trunk. We need to create a new jira to fix it. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855855#comment-13855855 ] Haohui Mai commented on HDFS-5636: -- Looking at the code, I think TestOfflineEditsViewer.testStore() has been covered by other test cases. I think we can delete the unit test. I created HDFS-5695 to clean up the code. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854989#comment-13854989 ] Hudson commented on HDFS-5636: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4924 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4924/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854998#comment-13854998 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Yarn-trunk #428 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/428/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855020#comment-13855020 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1645/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855032#comment-13855032 ] Hudson commented on HDFS-5636: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1619 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1619/]) HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1552841) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirective.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CachePool.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CacheAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCacheAdminConf.xml Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 3.0.0 Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[
https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854278#comment-13854278
]
Andrew Wang commented on HDFS-5636:
---
Thanks for the review Colin, fixed the comment and functionized CacheAdmin, the
rest inline:
bq. I think we should have a constant MAX_TTL_MS...
I added this, the enforcement was already in place, so this was just adding
another variable and tweaking usages a bit.
bq. Then we should not accept either relative or absolute limits...
I agree we should evaluate both relative and absolute limits against the same
relative max, but this means when converting to an absolute expiration for the
fsimage, the value could be greater than the relative max. Hope that's suitable.
bq. Can we have a static Expiration.NEVER final constant...?
I'm not sure that's much better, since it conflates Expiration's ability to do
absolute and relative expirations with this, which is just a relative. It's
also already the case that users need to use big longs for
{{Expiration.newRelative(long)}}, so it's not that different from the Java API
perspective. CacheAdmin wraps this up nicely of course.
Enforce a max TTL per cache pool
Key: HDFS-5636
URL: https://issues.apache.org/jira/browse/HDFS-5636
Project: Hadoop HDFS
Issue Type: Sub-task
Components: caching, namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch
It'd be nice for administrators to be able to specify a maximum TTL for
directives in a cache pool. This forces all directives to eventually age out.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854636#comment-13854636 ] Colin Patrick McCabe commented on HDFS-5636: +1. thanks, Andrew. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-5636-1.patch, hdfs-5636-2.patch, hdfs-5636-3.patch It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[
https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13853257#comment-13853257
]
Colin Patrick McCabe commented on HDFS-5636:
I think we should have a constant MAX_TTL_MS which is Long.MAX_VALUE / 4. Then
we should not accept either relative or absolute limits which are higher than
that. As you mentioned, these may cause overflows. We should forbid them the
same way we forbid negative times. Then you could define
RELATIVE_EXPIRY_NEVER_MS in terms MAX_TTL_MS.
We already encompass millions of years with MAX_VALUE / 4 milliseconds, so that
limit should be fine...
{code}
@Nullable
Long maxRelativeExpiry;
{code}
should have ms at the end to indicate units. Same goes for a lot of other
variables here.
{code}
// Allow if pool does not have a max expiry set,
// fail if the requested expiry is greater than the max
if (relExpiryTime maxRelativeExpiryTime) {
throw new InvalidRequestException(Expiration + expiry.toString()
+ exceeds the max relative expiration time of
+ maxRelativeExpiryTime + ms.);
}
{code}
This comment seems a bit confusing. Based on the first part, I looked for
special handling for pools that don't have a max expiry, and didn't find it.
Maybe it would be better to get rid of the first part of the comment, since
pools without a max expiry are just pools with max expiry set to MAX, right?
Can we have a static {{Expiration.NEVER}} final constant which is an instance
of {{Expiration}}? Since Expiration is immutable, it would be easy, I think.
Much nicer than making API users set the time to a huge value.
{code}
if (maxTtlString != null) {
long maxTtl;
if (maxTtlString.equalsIgnoreCase(never)) {
maxTtl = CachePoolInfo.RELATIVE_EXPIRY_NEVER;
} else {
try {
maxTtl = DFSUtil.parseRelativeTime(maxTtlString);
} catch (IOException e) {
System.err.println(
Error while parsing maxTtl value: + e.getMessage());
return 1;
}
}
info.setMaxRelativeExpiry(maxTtl);
{code}
This is duplicated a few times in CacheAdmin. Seems like it should be a
function.
Enforce a max TTL per cache pool
Key: HDFS-5636
URL: https://issues.apache.org/jira/browse/HDFS-5636
Project: Hadoop HDFS
Issue Type: Sub-task
Components: caching, namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hdfs-5636-1.patch
It'd be nice for administrators to be able to specify a maximum TTL for
directives in a cache pool. This forces all directives to eventually age out.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[
https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13853471#comment-13853471
]
Hadoop QA commented on HDFS-5636:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619642/hdfs-5636-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5773//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5773//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5773//console
This message is automatically generated.
Enforce a max TTL per cache pool
Key: HDFS-5636
URL: https://issues.apache.org/jira/browse/HDFS-5636
Project: Hadoop HDFS
Issue Type: Sub-task
Components: caching, namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hdfs-5636-1.patch
It'd be nice for administrators to be able to specify a maximum TTL for
directives in a cache pool. This forces all directives to eventually age out.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13852261#comment-13852261 ] Andrew Wang commented on HDFS-5636: --- I actually thought about this a bit more, and in terms of user experience, I propose the following: * If an expiry is specified in the CDInfo, check it against the pool's max expiration and error if invalid. * If an expiry is not specified, use the pool's max expiration. By default, the max expiration is never/infinite. * If a pool is modified to a lower max expiration, for all directives in the pool, set their expiration to min(oldExpiry, newMaxExpiration). * If a directive is modified to a new pool and a new expiration is not specified, set its expiration to min(oldExpiration, maxExpirationOfNewPool). The modify behavior will be kinda warty unfortunately when listCachePools is combined with modifyPool, since the CachePoolInfos you get out will have the expiry set (so behavior #4 won't happen). It'll be fine in CacheAdmin though, which is probably good enough since setting up pools is probably happening through that. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13852285#comment-13852285 ] Colin Patrick McCabe commented on HDFS-5636: It might better to leave existing cache directives as-is, similar to how setting a quota or changing permissions doesn't undo things done in the past. Aside from that, looks reasonable. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848090#comment-13848090 ] Andrew Wang commented on HDFS-5636: --- I was thinking a hard error. It's better for users too since the default expiry is never. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[ https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841653#comment-13841653 ] Colin Patrick McCabe commented on HDFS-5636: I'm not sure about this one. Generally different cache pools are associated with different users, and users don't age out in most of the clusters I'm familiar with. Let's see how people make use of the existing TTL stuff. We can always add this later if need be. Enforce a max TTL per cache pool Key: HDFS-5636 URL: https://issues.apache.org/jira/browse/HDFS-5636 Project: Hadoop HDFS Issue Type: Sub-task Components: caching, namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang It'd be nice for administrators to be able to specify a maximum TTL for directives in a cache pool. This forces all directives to eventually age out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5636) Enforce a max TTL per cache pool
[
https://issues.apache.org/jira/browse/HDFS-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841754#comment-13841754
]
Andrew Wang commented on HDFS-5636:
---
The issue is that if a pool is hanging on to forgotten cached data, that
prevents another pool from using it. This is true even with quotas/limits. It's
somewhat fixed if we also add minimum reservations for pools, but even then you
might want max-TTLs so that forgotten cache is returned to be used by fair
share.
Basically, right now TTLs are opt in, rather than opt out, and admins might
sometimes instead want them to be opt out. There's an impedance mismatch here
between admins and the users of a pool; just because a user has access to a
cache pool, doesn't necessarily mean they should be able to do whatever they
want with it because it might be bad for overall system performance. I see
max-TTL as an admin-friendly feature, since it'll help them avoid manually
cleaning up cache pools.
A couple imagined use cases:
* A scratch / temp cache pool with a low max TTL (say, 1 hr) and 0777
permissions, so all users can do some adhoc data exploration. The admin doesn't
need to worry about constantly cleaning up forgotten directives.
* When caching time-series data, you might only care about caching the last day
of data. Thus, the admin could set a max TTL of 24H to enforce this.
Sort of related, we might also want a command like {{hdfs cacheadmin
-removeExpiredDirectives [-pool pool]}} to help people clean up their expired
directives. Maybe even a trash-like functionality where directives that have
been expired for so long are automatically removed.
Enforce a max TTL per cache pool
Key: HDFS-5636
URL: https://issues.apache.org/jira/browse/HDFS-5636
Project: Hadoop HDFS
Issue Type: Sub-task
Components: caching, namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
It'd be nice for administrators to be able to specify a maximum TTL for
directives in a cache pool. This forces all directives to eventually age out.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
