[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062896#comment-15062896
]
Jason Lowe commented on TEZ-3009:
-
Sample container log showing the problem:
{noformat}
2015-12-11 18:53:23,832 [INFO] [TezChild] |task.ContainerReporter|: Attempting
to fetch new task for container container_e06_1449209941524_271349_01_002271
2015-12-11 18:53:23,879 [INFO] [main] |task.TezChild|: Shutdown invoked for
container container_e06_1449209941524_271349_01_002271
2015-12-11 18:53:23,880 [INFO] [main] |task.TezChild|: Shutting down container
container_e06_1449209941524_271349_01_002271
{noformat}
There's straightline code between the "Attempting to fetch new task ..." log
message and a later log of "Got TaskUpdate for ...". However we don't see the
latter log message, so something threw an exception. Unfortunately the code
that catches the exception squirrels it into a return result that is
subsequently ignored by the main code without logging it.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064616#comment-15064616 ] Siddharth Seth commented on TEZ-3009: - I believe this is fixed in master. Should be a simple log line addition in branch-0.7 ? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064650#comment-15064650
]
Jason Lowe commented on TEZ-3009:
-
I don't see any indication it's fixed in master. From TezChild.run:
{code}
try {
containerTask = getTaskFuture.get();
} catch (ExecutionException e) {
error = true;
Throwable cause = e.getCause();
return new
ContainerExecutionResult(ContainerExecutionResult.ExitStatus.EXECUTION_FAILURE,
cause, "Execution Exception while fetching new work: " +
e.getMessage());
{code}
So the exception can be silently handled unless the return result is inspected.
TezChild.main simply calls the run method and ignores the result.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064899#comment-15064899 ] Siddharth Seth commented on TEZ-3009: - Misinterpreted this. An error from TezTaskRuner.run() is also not always logged - except for the logging which may happen insider of TezTaskRunner itself. That, from looking at some logs in the past, does not always log errors. TezTaskRunner2 in master takes care of this better - but TezChild itself does not. The patch for the fetchWork failing looks good. Log at ERROR level instead ? +1 other than that. Can't comment on the UI aspects of the navigation - I would've thought that if a task is linked to the container, the task logs would point to the container log. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064903#comment-15064903
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778579/TEZ-3009.001.patch
against master revision 4ed7d1a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1389//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1389//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1389//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066831#comment-15066831
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision 4ed7d1a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1391//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1391//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1391//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171026#comment-15171026 ] Tsuyoshi Ozawa commented on TEZ-3009: - Kicking CI again. [~sseth], the patch looks good to me(non-binding). Could you check it? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189051#comment-15189051
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision dbd763f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1555//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1555//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339049#comment-15339049 ] Tsuyoshi Ozawa commented on TEZ-3009: - [~hitesh] could you check this? Jason's patch LGTM, while my review is non-binding one. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339109#comment-15339109
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1806//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1806//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340306#comment-15340306 ] Hitesh Shah commented on TEZ-3009: -- \cc [~sseth] as he was already reviewing this > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450077#comment-15450077 ] Hitesh Shah commented on TEZ-3009: -- +1 > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450260#comment-15450260
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision 5149cc4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.hadoop.mapred.split.TestGroupedSplits
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1946//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1946//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
