[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983591#comment-14983591 ] Siddharth Seth commented on TEZ-2917: - On the stuff the can be picked up from config - that's almost always the case, except it can be difficult to obtain the config for a job in Tez, unless the equivalent text fields have been written to config. After that, it can be retrieved if ATS is running. I don't mind removing some of the simpler lines like Initializing specific input. (That was 3 log lines before 2774/2775). Awaiting initialization - maybe once and on a timer. Cleaned up task includes information about the exit status - so it's more than a "Cleaning up" state transition line. On the others in the Fetcher, Merger etc - they were explicitly retained in TEZ-2775. I'm not going to +1 the removal since obviously there's others who think they need to be kept. Ability to debug gets priority. > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983531#comment-14983531 ] Sergey Shelukhin commented on TEZ-2917: --- MRInputLegacy deferring initialization - how is this useful. Another thing btw is that whatever developer can figure out by just looking at config file, metadata, etc., and code should not be on info level. I.e. logging configs and other such things. getContext().getDestinationVertexName() + ": " + "outputFormat=" + outputFormatClassName + ", using newmapreduce API=" + useNewApi - that looks like it could just be figured out from the job and config Using oldApi, MRpartitionerClass= - same Waiting for N... - same. It is just logged all the time and it seems extremely situational, when some small piece of code got stuck. Maybe it can log waiting once. Cleaning up task, Initializing task are extremely situational given the surrounding log lines (i.e. running... is logged at info). I dunno, I don't really care either way, this is based on users' complaints that too much obscure stuff that doesn't matter is getting logged. [~hagleitn] any opinion? > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983439#comment-14983439 ] Siddharth Seth commented on TEZ-2917: - Almost all of them. We've been through the same exercise in TEZ-2774 and TEZ-2775. There is value in disabling a lot of these log lines in specific scenarios. However, for regular production runs of potentially long running Tez jobs - leaving them there is useful. Some of the noisier lines were explicitly added back in the previous jiras. > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983419#comment-14983419 ] Sergey Shelukhin commented on TEZ-2917: --- I dunno if it matters how we disable the lines, as long as we do - to debug, you'd still have to reenable them. Many of these lines look like they would only be useful in very narrow debugging scenarios (e.g. ratio calculation details), so I think they belong on debug level. The lines that are useful for general understanding of what is going on are retained (e.g. for task transitions, running and finished, plus unexpected conditions, are info, but stuff like initializing, cleaning up or whatever are debug). Do you have suggestions for which lines to keep at info? > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983267#comment-14983267 ] Siddharth Seth commented on TEZ-2917: - [~sershe] - there have been a couple other jiras which try cleaning up logs. TEZ-2774, TEZ-2775. Some of these changes are even more aggressive than the ones on that jira. Many of the log lines become useful when things break. A lot of them show up once per task. Some of them though show up multiple times per task - and cause far more noise. Wondering if there's some other way to control these noisy lines. One would be to go and move a large chunk of the logging to TRACE, and move such lines to DEBUG. Another would be to use a cusom LOGGER for such lines - so that they can be turned off via logging configuration when required. > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[
https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981689#comment-14981689
]
TezQA commented on TEZ-2917:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769651/TEZ-2917.patch
against master revision 9ca6947.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1265//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1265//console
This message is automatically generated.
> change some logs from info to debug
> ---
>
> Key: TEZ-2917
> URL: https://issues.apache.org/jira/browse/TEZ-2917
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: TEZ-2917.patch
>
>
> I've done a highly unscientific summarization of the logs from some random
> queries, and will now change some log statements that are the most prevalent
> and not extremely useful from info to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981500#comment-14981500 ] Sergey Shelukhin commented on TEZ-2917: --- I ran some queries, grouped logs by first 30 characters with numbers removed, and went thru them manually for those that were present in relatively large numbers. So this is how these were removed. > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981494#comment-14981494 ] Sergey Shelukhin commented on TEZ-2917: --- This is mostly coming from the complaint that logs are hard to read, not from size. > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981477#comment-14981477 ] Hitesh Shah commented on TEZ-2917: -- How much general savings in terms of log size does this provide on a decent-size query? > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2917) change some logs from info to debug
[ https://issues.apache.org/jira/browse/TEZ-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981456#comment-14981456 ] Sergey Shelukhin commented on TEZ-2917: --- [~sseth] can you take a look? > change some logs from info to debug > --- > > Key: TEZ-2917 > URL: https://issues.apache.org/jira/browse/TEZ-2917 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2917.patch > > > I've done a highly unscientific summarization of the logs from some random > queries, and will now change some log statements that are the most prevalent > and not extremely useful from info to debug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
