[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404453#comment-15404453 ] Thejas M Nair commented on HIVE-14394: -- +1 > Reduce excessive INFO level logging > --- > > Key: HIVE-14394 > URL: https://issues.apache.org/jira/browse/HIVE-14394 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14394.2.patch, HIVE-14394.patch > > > We need to cull down on the number of logs we generate in HMS and HS2 that > are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[
https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403361#comment-15403361
]
Hive QA commented on HIVE-14394:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821350/HIVE-14394.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10420 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/729/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/729/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-729/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821350 - PreCommit-HIVE-MASTER-Build
> Reduce excessive INFO level logging
> ---
>
> Key: HIVE-14394
> URL: https://issues.apache.org/jira/browse/HIVE-14394
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-14394.2.patch, HIVE-14394.patch
>
>
> We need to cull down on the number of logs we generate in HMS and HS2 that
> are not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402434#comment-15402434 ] Josh Elser commented on HIVE-14394: --- I was able to build master with your v2 patch locally. LGTM. > Reduce excessive INFO level logging > --- > > Key: HIVE-14394 > URL: https://issues.apache.org/jira/browse/HIVE-14394 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14394.2.patch, HIVE-14394.patch > > > We need to cull down on the number of logs we generate in HMS and HS2 that > are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402297#comment-15402297 ] Sushanth Sowmyan commented on HIVE-14394: - Awesome! :) Thanks, I'll update the pom dep instead of this. > Reduce excessive INFO level logging > --- > > Key: HIVE-14394 > URL: https://issues.apache.org/jira/browse/HIVE-14394 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14394.patch > > > We need to cull down on the number of logs we generate in HMS and HS2 that > are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402097#comment-15402097 ] Josh Elser commented on HIVE-14394: --- Closing the loop on the upstream fix (since most probably won't be watching [~sushanth]'s PR): I just merged in his change (thanks so much for catching and fixing) and released an 0.1.1 of the reporter. It should be available via Maven central now (but not available via http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22dropwizard-metrics-hadoop-metrics2-reporter%22 for a few hours). Sorry for the temporary pain! > Reduce excessive INFO level logging > --- > > Key: HIVE-14394 > URL: https://issues.apache.org/jira/browse/HIVE-14394 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-14394.patch > > > We need to cull down on the number of logs we generate in HMS and HS2 that > are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14394) Reduce excessive INFO level logging
[ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401311#comment-15401311 ] Sushanth Sowmyan commented on HIVE-14394: - The recent event that predicated the filing of this bug is reports that we're filling up disks of the metastore box from recent hive installations, due to the excessive amount of metrics logging we generate. At first, this is not a hive problem, because the primary bit of extra logging is coming from HadoopMetrics2Reporter, which is a new library we depend on to publish our Codahale metrics to the Hadoop Metrics2 subsystem, and is easily fixed by changing the loglevel for that class to DEBUG, rather than retaining it on INFO. However, it becomes our problem for upgrade scenarios where the customer now has a manual change they need to make for smooth operations of their cluster because we upgraded a dependency. Thus, we need to work with the maintainer ([~elserj] of HadoopMetrics2Reporter to get a new version of the library, with info-level logging reduced to debug-level logging. To this end, I have filed a github pull request : https://github.com/joshelser/dropwizard-hadoop-metrics2/pull/1 . However, in the meanwhile, we also need to make a patch available (even if we don't commit it to hive) to solve the problem those installations have. Thus, I'm uploading a temporary patch to move the Metrics2 Reporter to the hive project, with the fix, so that those that need it can use this patch. I will update this patch with a newer patch which bumps up the version number of the dependency with the log-level aspect changed once we have a newer library version with that. > Reduce excessive INFO level logging > --- > > Key: HIVE-14394 > URL: https://issues.apache.org/jira/browse/HIVE-14394 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > > We need to cull down on the number of logs we generate in HMS and HS2 that > are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
