[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700274#comment-14700274 ] Colin Patrick McCabe commented on HDFS-7446: I would like to see this backported to 2.6.1 just because otherwise it will create hassles for people who want to start using inotify. Do you think this is feasible? HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Labels: 2.6.1-candidate Fix For: 2.7.0 Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697725#comment-14697725 ] Ming Ma commented on HDFS-7446: --- For the 2.6.1 effort, the backport is straightforward. But the API has changed compared to 2.6.0. This incompatibility only impacts folks who have been using inotify functionality introduced in 2.6.0. HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Labels: 2.6.1-candidate Fix For: 2.7.0 Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232899#comment-14232899 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-Yarn-trunk #763 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/763/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232912#comment-14232912 ] Hudson commented on HDFS-7446: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #24 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/24/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233040#comment-14233040 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1955 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1955/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233056#comment-14233056 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #24 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/24/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233111#comment-14233111 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1978 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1978/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233126#comment-14233126 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #24 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/24/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232426#comment-14232426 ] Hudson commented on HDFS-7446: -- FAILURE: Integrated in Hadoop-trunk-Commit #6635 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6635/]) HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) (cmccabe: rev 75a326aaff8c92349701d9b3473c3070b8c2be44) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatchList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/inotify.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/InotifyFSEditLogOpTranslator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventBatch.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/inotify/EventsList.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch, HDFS-7446.003.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230859#comment-14230859
]
Hadoop QA commented on HDFS-7446:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12684498/HDFS-7446.002.patch
against trunk revision 0c58890.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1218 javac
compiler warnings (more than the trunk's current 1217 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsTokens
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8885//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8885//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8885//console
This message is automatically generated.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch,
HDFS-7446.003.patch
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231019#comment-14231019
]
Hadoop QA commented on HDFS-7446:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12684534/HDFS-7446.003.patch
against trunk revision a7fba0b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8890//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8890//console
This message is automatically generated.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7446.001.patch, HDFS-7446.002.patch,
HDFS-7446.003.patch
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226755#comment-14226755
]
Colin Patrick McCabe commented on HDFS-7446:
bq. It feels like we have a mismatch between the underlying data and our
objects. The need for the VHS-rewind in getTxidBatchSize is one example, what
we really want there is an iterator of EditEvents, with one EditEvents per txid
(name is just a suggestion).
Basically, the code in {{getTxidBatchSize}} is converting between what we get
back from the NameNode RPC (a bunch of Events, not grouped by txid) to what we
want to return (a batch of events that all have the same txid).
I guess we could do the conversion earlier, in
{{PBHelper#convert(GetEditsFromTxidResponseProto resp)}}. This would mean
pushing the batch concept into {{EventsList}}. I'm not sure there's much
benefit in this, though. {{EventsList}} is used on the {{NameNode}} as well,
and the NN doesn't care about grouping Events of the same txid into batches.
But we'd have to implement all that logic anyway if we went down this route.
(The NN has its own concept of batching... how many Events it sends back in a
single RPC... and of course it will not split a txid across RPCs. But that's
the only batching we get out of it.)
I admit that rewinding the iterator feels awkward. I was looking for a utility
function in {{Iterators}} or something that would help with this, but I didn't
find anything. Java doesn't have the concept of duplicating iterators that C++
does, so you can't just make a copy of the iterator and then push only that
copy forward. I wanted to avoid allocating an array until I knew what size I
wanted. I suppose I could create a {{LinkedList}} rather than an array, and
avoid the rewinding that way. That's not very efficient, though. One slightly
awkward function feels like an acceptable tradeoff for memory efficiency.
bq. The txid could also be moved into EditEvents which would also save some
bytes.
Most batches are small-- I think the median size will be a batch of size 1,
with maybe some outliers at 2, 3, or 4. I don't think there's a lot of memory
savings to be had by moving the txid into {{EditEvents}}.
A bigger issue is that we need (well maybe not need, but the code will be
really, really awkward if we don't) to add txid to the protobuf {{EventProto}}
structure in {{inotify.proto}}. This is because there can be gaps in the
txids returned by {{EventsListProto}}. Currently, all {{EventsListProto}} has
is firstTxid, lastTxid, and a list of Events. But you don't know what the txid
of any of those Events actually is... if firstTxid is 100 and lastTxid is 200,
and you have 50 events, where are they in that sequence? Who knows. And if
you add this to the PB, it's extremely awkward not to add it to the Java
{{Event}} class as well.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7446.001.patch
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225571#comment-14225571
]
Colin Patrick McCabe commented on HDFS-7446:
This patch adds a txid field to all {{Event}} objects. We have to send this
over the wire, since the existing information (start txid and stop txid for a
group of txids we got in an RPC) is not enough. Not every edit log txid maps
to an event.
Another complication is the fact that some txids map to more than one event.
This makes it somewhat difficult for clients to know what they've read up to
when using a one-event-at-a-time interface. This patch solves that by having
the {{DFSInotifyEventInputStream}} return an array of events. In the cases
where a single txid maps to multiple events, we return an array of all those
events. So the client knows that after it has finished processing this batch,
it is done with that transaction id. This interface is marked as unstable, so
changing it is not a problem.
Miscellaneous cleanups: I made all some fields final in the {{Event}}
structures. In cases where I modified a unit test, I replaced assertTrue(1 ==
foo) with assertEquals(1, foo). The latter gives nicer error messages when the
test fails.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[ https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225605#comment-14225605 ] Andrew Wang commented on HDFS-7446: --- Hey Colin, thanks for working on this. You definitely bring up a good point about the txids. Since this is marked as unstable and still quite new, I think it's okay to make sweeping changes to the API. I had just a few high-level review comments, the code itself looks fine: * It feels like we have a mismatch between the underlying data and our objects. The need for the VHS-rewind in getTxidBatchSize is one example, what we really want there is an iterator of EditEvents, with one EditEvents per txid (name is just a suggestion). * The txid could also be moved into EditEvents which would also save some bytes. I'm hoping this isn't too bad to do, since the edit log translator already returns an Event[] per op, and it seems like most of the PB code can be reused. HDFS inotify should have the ability to determine what txid it has read up to - Key: HDFS-7446 URL: https://issues.apache.org/jira/browse/HDFS-7446 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7446.001.patch HDFS inotify should have the ability to determine what txid it has read up to. This will allow users who want to avoid missing any events to record this txid and use it to resume reading events at the spot they left off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7446) HDFS inotify should have the ability to determine what txid it has read up to
[
https://issues.apache.org/jira/browse/HDFS-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225710#comment-14225710
]
Hadoop QA commented on HDFS-7446:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683729/HDFS-7446.001.patch
against trunk revision a655973.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.mapreduce.v2.app.TestCheckpointPreemptionPolicy
org.apache.hadoop.mapreduce.v2.app.TestMRClientService
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8841//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8841//console
This message is automatically generated.
HDFS inotify should have the ability to determine what txid it has read up to
-
Key: HDFS-7446
URL: https://issues.apache.org/jira/browse/HDFS-7446
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7446.001.patch
HDFS inotify should have the ability to determine what txid it has read up
to. This will allow users who want to avoid missing any events to record
this txid and use it to resume reading events at the spot they left off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
