[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[ https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834471#comment-13834471 ] Colin Patrick McCabe commented on HDFS-5430: Looks good. +1 pending jenkins (and manual testing of TestOEV, since patch won't pick up the binary change as we know) Support TTL on CacheBasedPathDirectives --- Key: HDFS-5430 URL: https://issues.apache.org/jira/browse/HDFS-5430 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Andrew Wang Priority: Minor Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch, hdfs-5430-3.patch, hdfs-5430-4.patch It would be nice if CacheBasedPathDirectives would support an expiration time, after which they would be automatically removed by the NameNode. This time would probably be in wall-block time for the convenience of system administrators. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[
https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834488#comment-13834488
]
Hadoop QA commented on HDFS-5430:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616162/hdfs-5430-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.web.TestWebHDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5598//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5598//console
This message is automatically generated.
Support TTL on CacheBasedPathDirectives
---
Key: HDFS-5430
URL: https://issues.apache.org/jira/browse/HDFS-5430
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch, hdfs-5430-3.patch,
hdfs-5430-4.patch
It would be nice if CacheBasedPathDirectives would support an expiration
time, after which they would be automatically removed by the NameNode. This
time would probably be in wall-block time for the convenience of system
administrators.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[ https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834551#comment-13834551 ] Andrew Wang commented on HDFS-5430: --- I ran the WebHDFS test locally a few times and it worked, it looks like a problem with the build host. The OEV test we also know will flake since it's changing a binary file. Based on that, will commit shortly based on Colin's +1. Thanks for the review! Support TTL on CacheBasedPathDirectives --- Key: HDFS-5430 URL: https://issues.apache.org/jira/browse/HDFS-5430 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Andrew Wang Priority: Minor Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch, hdfs-5430-3.patch, hdfs-5430-4.patch It would be nice if CacheBasedPathDirectives would support an expiration time, after which they would be automatically removed by the NameNode. This time would probably be in wall-block time for the convenience of system administrators. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[
https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832132#comment-13832132
]
Hadoop QA commented on HDFS-5430:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615681/hdfs-5430-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5563//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5563//console
This message is automatically generated.
Support TTL on CacheBasedPathDirectives
---
Key: HDFS-5430
URL: https://issues.apache.org/jira/browse/HDFS-5430
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch, hdfs-5430-3.patch
It would be nice if CacheBasedPathDirectives would support an expiration
time, after which they would be automatically removed by the NameNode. This
time would probably be in wall-block time for the convenience of system
administrators.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[
https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832141#comment-13832141
]
Colin Patrick McCabe commented on HDFS-5430:
I think it would be nicer to have something like this:
{code}
public class CacheDirectiveInfo {
public static class Expiration {
public static Expiration newRelative(long ms) {
return new Expiration(ms, true);
}
public static Expiration newAbsolute(long ms) {
return new Expiration(ms, false);
}
public static Expiration newAbsolute(Date date) {
return new Expiration(date.toLong(), false);
}
private Expiration(long ms, boolean isRelative) {
this.ms = ms;
this.isRelative = isRelative;
}
public long toAbsolute() { // using local clock
...
}
private final long ms;
private final boolean isRelative;
}
...
}
{code}
That way, we wouldn't have to worry about the awkward situation of having two
fields in {{CacheDirectiveInfo}}, which can't be both used at the same time.
What do you think?
We can continue making this just a long in CacheDirective, since at that point,
it's always absolute. It would be nice to have a constant for NEVER, rather
than hard-coding -1 (Sun style guide discourages magic numbers.)
I also wonder if we can add a {{boolean expired}} or something to
{{CacheDirectiveStats}}, since otherwise the client can't easily figure out if
the directive has expired. After all, the clock on the NN might be different.
Looks good aside from that.
Support TTL on CacheBasedPathDirectives
---
Key: HDFS-5430
URL: https://issues.apache.org/jira/browse/HDFS-5430
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch, hdfs-5430-3.patch
It would be nice if CacheBasedPathDirectives would support an expiration
time, after which they would be automatically removed by the NameNode. This
time would probably be in wall-block time for the convenience of system
administrators.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[
https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829816#comment-13829816
]
Hadoop QA commented on HDFS-5430:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615269/hdfs-5430-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
org.apache.hadoop.cli.TestCacheAdminCLI
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5540//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5540//console
This message is automatically generated.
Support TTL on CacheBasedPathDirectives
---
Key: HDFS-5430
URL: https://issues.apache.org/jira/browse/HDFS-5430
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-5430-1.patch
It would be nice if CacheBasedPathDirectives would support an expiration
time, after which they would be automatically removed by the NameNode. This
time would probably be in wall-block time for the convenience of system
administrators.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[
https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13830513#comment-13830513
]
Hadoop QA commented on HDFS-5430:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615406/hdfs-5430-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-2 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5550//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5550//console
This message is automatically generated.
Support TTL on CacheBasedPathDirectives
---
Key: HDFS-5430
URL: https://issues.apache.org/jira/browse/HDFS-5430
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Priority: Minor
Attachments: hdfs-5430-1.patch, hdfs-5430-2.patch
It would be nice if CacheBasedPathDirectives would support an expiration
time, after which they would be automatically removed by the NameNode. This
time would probably be in wall-block time for the convenience of system
administrators.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5430) Support TTL on CacheBasedPathDirectives
[ https://issues.apache.org/jira/browse/HDFS-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829663#comment-13829663 ] Andrew Wang commented on HDFS-5430: --- Couple further notes, I did a couple missed PATH_BASED renames as well, and changed the edit log loading for addDirective since I think it would potentially allow users to pass in their own directive IDs. Support TTL on CacheBasedPathDirectives --- Key: HDFS-5430 URL: https://issues.apache.org/jira/browse/HDFS-5430 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Andrew Wang Priority: Minor Attachments: hdfs-5430-1.patch It would be nice if CacheBasedPathDirectives would support an expiration time, after which they would be automatically removed by the NameNode. This time would probably be in wall-block time for the convenience of system administrators. -- This message was sent by Atlassian JIRA (v6.1#6144)
