[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936314#comment-14936314 ] TezQA commented on TEZ-2758: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12764360/TEZ-2758-4.patch against master revision 773. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1185//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1185//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Fix For: 0.7.1, 0.6.3, 0.8.1 > > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch, > TEZ-2758-4.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936277#comment-14936277 ] Jeff Zhang commented on TEZ-2758: - Committed to 0.6/0.7/master > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Fix For: 0.6.3 > > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch, > TEZ-2758-4.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935873#comment-14935873 ] TezQA commented on TEZ-2758: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12764200/TEZ-2758-3.patch against master revision 773. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1183//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1183//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935854#comment-14935854 ] TezQA commented on TEZ-2758: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12764200/TEZ-2758-3.patch against master revision 773. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1182//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1182//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935763#comment-14935763 ] Hitesh Shah commented on TEZ-2758: -- Re-triggered precommit. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935758#comment-14935758 ] Hitesh Shah commented on TEZ-2758: -- bq. Can not verify it is closed Because can't get the stream object as it already be removed. What I meant was adding "assertTrue(recoveryService.outputStreamMap.containsKey(dagId));" before sending the first dag finished event. +1. Feel free to make the above addition ( if the check makes sense ) and commit. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934791#comment-14934791 ] Jeff Zhang commented on TEZ-2758: - Comments addressed. bq. Test ( testMultipleDAGFinishedEvent ) should contain atleast one assertTrue check in addition to the checks that stream is closed and removed. Can not verify it is closed Because can't get the stream object as it already be removed. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch, TEZ-2758-3.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933734#comment-14933734 ] Hitesh Shah commented on TEZ-2758: -- Comments: Mostly looks good. bq. "throw new IOException("Summary file already exists, summaryPath:" + summaryPath);" Maybe just create summary stream object in the serviceStart() so that this path is never used? ( this is optional - if not changed, please confirm that it ends up in the RECOVERY_FATAL_OCCURRED_DIR creation path ) bq. throw new IOException("Recovery file already exists, recoveryPath:" + dagFilePath); Should we follow the "RECOVERY_FATAL_OCCURRED_DIR" approach if we encounter this error? i.e. let processing continue but not recover if we hit an unknown situation like this? bq.assertFalse(recoveryService.outputStreamMap.containsKey(dagId)); Test ( testMultipleDAGFinishedEvent ) should contain atleast one assertTrue check in addition to the checks that stream is closed and removed. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933326#comment-14933326 ] Jeff Zhang commented on TEZ-2758: - [~hitesh] Please help review it. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14917357#comment-14917357 ] TezQA commented on TEZ-2758: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12762658/TEZ-2758-2.patch against master revision 406721a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1176//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1176//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch, TEZ-2758-2.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14910111#comment-14910111 ] TezQA commented on TEZ-2758: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12762629/TEZ-2758-1.patch against master revision 406721a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1175//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1175//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905690#comment-14905690 ] Jeff Zhang commented on TEZ-2758: - bq. For the first dag finished event seen, lets write the dag finished to both summary and recovery log but after that for any retrospective failures, only write to summary but not to the main log ( if the outputstream has been closed ). Does that work? Yes, this is what I plan to do. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904694#comment-14904694 ] Hitesh Shah commented on TEZ-2758: -- bq. Right, It is not necessary to write DAGFinishedEvent to recovery file after we write it to summary log. We won't read recovery file as long as summary log says completed. That means the append API could be removed. Will post a patch soon. Lets write the dag finished to both summary and recovery log but after that for any retrospective failures, only write to summary but not to the main log ( if the outputstream has been closed ). Does that work? > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904351#comment-14904351 ] Jeff Zhang commented on TEZ-2758: - bq. Also, my original question was whether we read the main dag file even if summary log says completed? Right, It is not necessary to write DAGFinishedEvent to recovery file after we write it to summary log. We won't read recovery file as long as summary log says completed. That means the append API could be removed. Will post a patch soon. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903577#comment-14903577 ] Hitesh Shah commented on TEZ-2758: -- [~zjffdu] The option to close on a new dag submission seems interesting. Only consideration would be whether a dag finished event for an older dag can show up after a dag submitted? Also, my original question was whether we read the main dag file even if summary log says completed? Assuming we do not read dag data if there is another dag in the summary then we can first close stream when new dag is submitted and also drop all older dag events that show up after the stream has been closed? > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725759#comment-14725759 ] Hitesh Shah commented on TEZ-2758: -- In what scenarios will we read the dag recovery stream file when the summary stream says dag is complete? Will we read it back for completed dags anyway to re-construct the dag data? If there are geniune cases where we will likely need append, maybe we just close this jira as wont fix? > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724983#comment-14724983 ] Jeff Zhang commented on TEZ-2758: - Ideally append should never be used, but there's one special case. DAGFinishedEvent happens multiple times (SUCCEEDED --> ERROR). After DAGFinishedEvent is seen, output stream for recovery file is closed. So if we see the next DAGFinishedEvent, append is needed if we want to record this event. * Should we need to record the next DAGFinsihedEvent if dag has already finished ? (If yes, it would be hard to decide when dag is really finished (when to close the outputstream), maybe consider the dag is finished if the next dag is started) [~hitesh] Any thoughts ? > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2758) Remove append API in RecoveryService after TEZ-1909
[ https://issues.apache.org/jira/browse/TEZ-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724721#comment-14724721 ] TezQA commented on TEZ-2758: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12753435/TEZ-2758-1.patch against master revision 2219c5e. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1051//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/1051//artifact/patchprocess/newPatchFindbugsWarningstez-api.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1051//console This message is automatically generated. > Remove append API in RecoveryService after TEZ-1909 > --- > > Key: TEZ-2758 > URL: https://issues.apache.org/jira/browse/TEZ-2758 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.6.2 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2758-1.patch > > > After TEZ-1909, there would be no case for append recovery file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)