[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876602#comment-14876602
 ] 

Siddharth Seth commented on TEZ-2774:
-

[~bikassaha] - is the addendum patch required in branch-7 as well ?

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, 
> TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876634#comment-14876634
 ] 

Bikas Saha commented on TEZ-2774:
-

I just did.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, 
> TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[TezQA (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804316#comment-14804316
 ] 

TezQA commented on TEZ-2774:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  
http://issues.apache.org/jira/secure/attachment/12757126/TEZ-2774.addendum.2.patch
  against master revision 1a065b9.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1148//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1148//console

This message is automatically generated.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, 
> TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804367#comment-14804367
 ] 

Bikas Saha commented on TEZ-2774:
-

[~hitesh] [~sseth] Can you take a quick look at the findbugs fix addendum 
please.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, 
> TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Hitesh Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804760#comment-14804760
 ] 

Hitesh Shah commented on TEZ-2774:
--

+1

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, 
> TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790974#comment-14790974
 ] 

Siddharth Seth commented on TEZ-2774:
-

Looks fine.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791022#comment-14791022
 ] 

Bikas Saha commented on TEZ-2774:
-

Thanks!
commit 1a065b9d87d84645363d0c65ae021a6a514169a8
Author: Bikas Saha 
Date:   Wed Sep 16 12:50:38 2015 -0700

TEZ-2774. addendum to add a preemption periodic log


> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746076#comment-14746076
 ] 

Bikas Saha commented on TEZ-2774:
-

Sounds good. The log is once per vertex - so it should be fine.
Lets keep the completion log in the vertex for now. This can be a candidate for 
deletion in the next phase. The logs did not grow by much after the .5 patch. I 
am fine with .5 patch going in.
My vote is only for a .7 backport. 0.5 code is different enough that I would 
avoid touching that anymore. Even 0.7 will be painful due to file renames.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746085#comment-14746085
 ] 

Siddharth Seth commented on TEZ-2774:
-

Committing.

On the backports, I think we've been moving most bug fixes all the way to 0.5. 
This would be useful there as well, since there are people who're still using 
that line. Anyone else has thoughts on this ?

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746033#comment-14746033
 ] 

Siddharth Seth commented on TEZ-2774:
-

The classpath will typically be small. It'll be large for MiniClusters. It 
provides a good mechanism to figure out whether multiple jars of the same 
version have been added, so I'd retain that. Once per vertex does not add a lot.

In the most recent patch.
{code}
+// this log helps quickly count the completion count for a vertex.
+// grepping and counting for attempts and handling re-tries is time 
consuming
+LOG.info("Task Completion: " + 
constructCheckTasksForCompletionLog(vertex));
{code}
This log line still seems a little excessive, given all the information 
available via history.

IAC, Will wait for your reply to this, and commit this patch or the next one.

Backporting this to various branches is not going to be trivial - it won't 
apply cleanly to any of the branches. I'll open follow up jiras for the 
backports. I can take care of the 0.7 port. [~bikassaha], [~hitesh], [~zjffdu] 
- would appreciate help with the 0.6/0.5 (potentially very similar).



> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[TezQA (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744305#comment-14744305
 ] 

TezQA commented on TEZ-2774:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12755791/TEZ-2774.5.patch
  against master revision b3ad595.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1132//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1132//console

This message is automatically generated.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744189#comment-14744189
 ] 

Bikas Saha commented on TEZ-2774:
-

Looks like there is a new log that dumps containercontext. thats a huge log of 
which a large part is the entire classpath. fortunately, its once per vertex 
but perhaps the classpath could be debug?

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744040#comment-14744040
 ] 

Bikas Saha commented on TEZ-2774:
-

bq. Minor nit - can this only be disabled for successful attempts?
Done.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch, TEZ-2774.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Hitesh Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742922#comment-14742922
 ] 

Hitesh Shah commented on TEZ-2774:
--

bq. Biggest removal is counter logging in attempt finished event since it 
duplicates counter logging in task finished event. That should save a lot of 
space and make up for some of the logs that I added back. Counter logging 
removal does not reduce simulation logs since they dont have counters.

Minor nit - can this only be disabled for successful attempts? Failed attempts 
might need counters to shed some light on GC issues, etc?




> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[TezQA (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741903#comment-14741903
 ] 

TezQA commented on TEZ-2774:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12755533/TEZ-2774.4.patch
  against master revision b6f15dc.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1120//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1120//console

This message is automatically generated.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, 
> TEZ-2774.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739338#comment-14739338
 ] 

Siddharth Seth commented on TEZ-2774:
-

Some statistics on log sizes, on a 20 node cluster
h6. JoinValidate example with 100, 50, 100 (lhsScan, rhsScan, SortMergeJoin) 
tasks
||Type||TotalLogSize||AM LogSize||SortMergeLogSize per task||
|Current|38MB|4MB|~300KB|
|Reduced|11MB|2.1MB|~65K|

h6. HashJoin example with 100, 100, 200 (lhsScan, rhsScan, HashJoin) tasks
||Type||TotalLogSize||AM LogSize||HashJoinLogSize per task||
|Current|316MB|7.2MB|~1.6MB|
|Reduced|65MB|3.3MB|~330KB|

That's some pretty large log files that we generate at the moment, which makes 
it tougher to read logs as well as hurts performance. Clearly we need adequate 
information in the logs to debug in case of issues. Given this affects everyone 
trying to debug via log files, please go ahead and modify the patch to add back 
/ change whatever is required. While doing this though, running a couple of 
jobs will help, and please try looking for information that is already 
available via some other source, so that we can try keeping the size of the 
logs small. 

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[TezQA (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737886#comment-14737886
 ] 

TezQA commented on TEZ-2774:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12755020/TEZ-2774.3.txt
  against master revision 00508f8.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.dag.app.rm.TestContainerReuse

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1101//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1101//console

This message is automatically generated.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737237#comment-14737237
 ] 

Siddharth Seth commented on TEZ-2774:
-

Scenarios make all the difference. If speculation is the concern, some of these 
messages can be enabled only when speculation is turned on - instead of turning 
them on for all running jobs.
Task completions are already available via the history events.
The task scheduler logs for min-held and prewarm are alredy at INFO level - 
requested by other reviewers as well, for the same reason.
The input spec log is already present in the TaskSpec log.

The intent is to reduce the size of the AM log - which otherwise has a lot of 
duplicated information, and becomes difficult to read. Keeping all log lines 
wholesale because they've been helpful in the past means we'll keep these later 
as well. Instead of keeping all of them, it's better to look at where else this 
information is available, how relevant it is, and whether it can be obtained 
without excessive logging.
The task request to the scheduler - I don't really see how this helps anything. 
Logging the equivalent of the CLC once per vertex should give us enough 
information on what tasks are requesting.
So once again, please look at what is available after the changes go in, and 
suggest potential improvements - with scenarios in mind - rather than trying to 
keep all log lines with duplicated infromation.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[TezQA (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735828#comment-14735828
 ] 

TezQA commented on TEZ-2774:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12754758/TEZ-2774.2.txt
  against master revision c07d6b7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.dag.app.rm.TestContainerReuse

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1091//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1091//console

This message is automatically generated.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

[Bikas Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735829#comment-14735829
 ] 

Bikas Saha commented on TEZ-2774:
-

I will reiterate my initial comments about the logs I have mentioned. Let us 
keep them there at info level. Logging at debug level is not always useful 
because it may be an intermittent issue or too expensive to reproduce. Some of 
the removed logs like which attempt succeeded and which got killed will only be 
relevant during speculation and not logged all the time. Removing the kill log 
is not going to reduce general logging size but will be sorely missed if there 
is a bug in speculation. The task scheduler logs for containers have been 
useful when debugging why min-held or pre-warm containers were not working as 
expected. The input-spec log is used when figuring out how many events should 
have actually come to an input and how many actually did for debugging hung 
jobs. Then correlating it with event routing log from the AM to see where the 
error is. The event routing log from the AM was removed in the initial patch 
while the the next version introduces it back but continues to hide the task 
spec log. This seems arbitrary.

It is hard to explain all the debugging context from previous experience. 
Hence, my suggestion is to accept the comments as is instead of debating them 
subjectively. With time we can remove more logs. But I am quite wary of doing 
such a bulk removal and not having logs to debug when I need them.

> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)