[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062896#comment-15062896 ] Jason Lowe commented on TEZ-3009: - Sample container log showing the problem: {noformat} 2015-12-11 18:53:23,832 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e06_1449209941524_271349_01_002271 2015-12-11 18:53:23,879 [INFO] [main] |task.TezChild|: Shutdown invoked for container container_e06_1449209941524_271349_01_002271 2015-12-11 18:53:23,880 [INFO] [main] |task.TezChild|: Shutting down container container_e06_1449209941524_271349_01_002271 {noformat} There's straightline code between the "Attempting to fetch new task ..." log message and a later log of "Got TaskUpdate for ...". However we don't see the latter log message, so something threw an exception. Unfortunately the code that catches the exception squirrels it into a return result that is subsequently ignored by the main code without logging it. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064616#comment-15064616 ] Siddharth Seth commented on TEZ-3009: - I believe this is fixed in master. Should be a simple log line addition in branch-0.7 ? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064650#comment-15064650 ] Jason Lowe commented on TEZ-3009: - I don't see any indication it's fixed in master. From TezChild.run: {code} try { containerTask = getTaskFuture.get(); } catch (ExecutionException e) { error = true; Throwable cause = e.getCause(); return new ContainerExecutionResult(ContainerExecutionResult.ExitStatus.EXECUTION_FAILURE, cause, "Execution Exception while fetching new work: " + e.getMessage()); {code} So the exception can be silently handled unless the return result is inspected. TezChild.main simply calls the run method and ignores the result. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064899#comment-15064899 ] Siddharth Seth commented on TEZ-3009: - Misinterpreted this. An error from TezTaskRuner.run() is also not always logged - except for the logging which may happen insider of TezTaskRunner itself. That, from looking at some logs in the past, does not always log errors. TezTaskRunner2 in master takes care of this better - but TezChild itself does not. The patch for the fetchWork failing looks good. Log at ERROR level instead ? +1 other than that. Can't comment on the UI aspects of the navigation - I would've thought that if a task is linked to the container, the task logs would point to the container log. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064903#comment-15064903 ] TezQA commented on TEZ-3009: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778579/TEZ-3009.001.patch against master revision 4ed7d1a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1389//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/1389//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1389//console This message is automatically generated. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066831#comment-15066831 ] TezQA commented on TEZ-3009: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch against master revision 4ed7d1a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.test.TestFaultTolerance Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1391//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/1391//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1391//console This message is automatically generated. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171026#comment-15171026 ] Tsuyoshi Ozawa commented on TEZ-3009: - Kicking CI again. [~sseth], the patch looks good to me(non-binding). Could you check it? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189051#comment-15189051 ] TezQA commented on TEZ-3009: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch against master revision dbd763f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.test.TestFaultTolerance Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1555//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1555//console This message is automatically generated. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339049#comment-15339049 ] Tsuyoshi Ozawa commented on TEZ-3009: - [~hitesh] could you check this? Jason's patch LGTM, while my review is non-binding one. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339109#comment-15339109 ] TezQA commented on TEZ-3009: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch against master revision 993a7b5. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.dag.app.rm.TestContainerReuse Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1806//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1806//console This message is automatically generated. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340306#comment-15340306 ] Hitesh Shah commented on TEZ-3009: -- \cc [~sseth] as he was already reviewing this > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450077#comment-15450077 ] Hitesh Shah commented on TEZ-3009: -- +1 > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450260#comment-15450260 ] TezQA commented on TEZ-3009: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch against master revision 5149cc4. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.hadoop.mapred.split.TestGroupedSplits Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/1946//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1946//console This message is automatically generated. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)