[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811588#comment-15811588 ] Dhiraj Kumar commented on HIVE-15531: - If commons-logging is not used by hive then log4j-jcl is not required. But if commons-logging is being used by Hadoop, a bridge will be required. By default log4j-1.2-api acts as bridge for hadoop logging which had been added in the commit https://github.com/apache/hive/commit/c93d6c77e31e2eb9b40f5167ab3491d44eae351a Also agree that with classpath properly ordered, this issue will not surface at all. I would have liked it with log4j-jcl, removing classpath ordering dependency as well as user lib into classpath (coming higher than hive log4j's lib) I will leave it to you and will close the discussion here. > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version Logger, does not configure it properly since configuration for it is > missing and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811016#comment-15811016 ] Prasanth Jayachandran commented on HIVE-15531: -- We used to have log4j-jcl bridge before when hive used both commons logging and slf4j. We then moved hive completely use slf4j removing the log4j-jcl bridge as it is no longer. This is the commit that moved to slf4j and remove log4j-jcl bridge. https://github.com/apache/hive/commit/a8ea6d82b0b6763f846848afeb5b338ab94e474a Regarding the classpath ordering, we always force loading log4j2 jars first before loading hadoop jars. https://github.com/apache/hive/blob/master/bin/hive#L193 > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version Logger, does not configure it properly since configuration for it is > missing and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[
https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810881#comment-15810881
]
Dhiraj Kumar commented on HIVE-15531:
-
{quote} Does Hive logger still get log4j2, and Hadoop logger log4j? {quote}
It is possible if classpath is messed up. Will depend what extra library end
user has put in their classpath. That is exactly what happened in my case as
explained in earlier comment.
This patch makes sure that even if end users mess up their classpath (just from
log4j perspective), still logging will be setup properly and won't get into
log4j vs log4j2 issue.
I did not look it from the perspective of slf4j.
[~prasanth_j] can you have a look?
> Hive breaks Hadoop commons logging with log4j2
> --
>
> Key: HIVE-15531
> URL: https://issues.apache.org/jira/browse/HIVE-15531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
>Priority: Minor
> Attachments: HIVE-15531.patch
>
>
> Hadoop (2.7), which is using Commons-logging is not compatible with log4j2
> without bridge.
> The bridge is missing in Hive.
> This leads to a problem whereby commons-logging initialises a log4j (1.2)
> version Logger, does not configure it properly since configuration for it is
> missing and sends logging output to stdout (the default).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805856#comment-15805856 ] Sergey Shelukhin commented on HIVE-15531: - Hmm... not sure what the best approach there. Does Hive logger still get log4j2, and Hadoop logger log4j? My assumption was that with slf4j, they'd both use the same. [~prasanth_j] did the slf4j work and might know better. > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version Logger, does not configure it properly since configuration for it is > missing and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[
https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803961#comment-15803961
]
Dhiraj Kumar commented on HIVE-15531:
-
[~sershe] I thought about test cases with below approach
a) Manipulating classpath at runtime to put log4j 1.2 before log4j 2.4
libraries and showcasing the fact that commons-log will pick older version of
logger.
b) Keeping log4j 1.2 in beginning and putting log4j-jcl anywhere in classpath
and showing that commons-logging will pick Log4j 2.4
The problem with this approach I have to manipulate classpath at runtime which
might affect other tests downstream. Moreover, I am not able to find a clean
solution to do it.
Would a trivial test like this suffice?
{code}
Log log = LogFactory.getLog(CommonsLoggingTest.class);
assertEquals(log.getClass(),org.apache.logging.log4j.jcl.Log4jLog.class);
{code}
> Hive breaks Hadoop commons logging with log4j2
> --
>
> Key: HIVE-15531
> URL: https://issues.apache.org/jira/browse/HIVE-15531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
>Priority: Minor
> Attachments: HIVE-15531.patch
>
>
> Hadoop (2.7), which is using Commons-logging is not compatible with log4j2
> without bridge.
> The bridge is missing in Hive.
> This leads to a problem whereby commons-logging initialises a log4j (1.2)
> version Logger, does not configure it properly since configuration for it is
> missing and sends logging output to stdout (the default).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800899#comment-15800899 ] Dhiraj Kumar commented on HIVE-15531: - Related issue [HIVE-11572|https://issues.apache.org/jira/browse/HIVE-11572] > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version Logger, does not configure it properly since configuration for it is > missing and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[
https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798980#comment-15798980
]
Dhiraj Kumar commented on HIVE-15531:
-
[~sershe], I dug the bug deeper and found out that it was our environmental
issue. There were two problem at our end,
1. An UDF jar in near beginning of classpath with log4j (1.2) classes in it. It
was picked by Commons-logger.
2. Another jar, having a basic log4j.properties was picked for properties since
hive configuration directory (top in class path) was missing log4j.properties
and was having log4j2.properties.
So, even without log4j-jcl (current patch), it may work fine. Since
log4j-1.2-api (already part of hive 2.x) has org.apache.log4j.Logger class and
initialises it properly with log4j2 configuration. That works like bridge.
Provided there is no other class in class path before log4j-1.2-api which has
org.apache.log4j.Logger class.
log4j-jcl overcomes classpath issue since commons-logging tries to find
LogFactory and log4j-jcl injects itself there. So there won't be any classpath
issue with it. Code snippet below from org.apache.commons.logging.LogFactory
{code}
// Second, try to find a service by using the JDK1.3 class
// discovery mechanism, which involves putting a file with the name
// of an interface class in the META-INF/services directory, where the
// contents of the file is a single line specifying a concrete class
// that implements the desired interface.
if (factory == null) {
if (isDiagnosticsEnabled()) {
logDiagnostic("[LOOKUP] Looking for a resource file of name ["
+ SERVICE_ID +
"] to define the LogFactory subclass to use...");
}
try {
final InputStream is = getResourceAsStream(contextClassLoader,
SERVICE_ID);
{code}
Since this issue reflects itself based on classpath, I am thinking to inject
log4j 1.2 in classpath with test scope and check that right class has been
loaded by Commons logging.
Would like to know your thoughts.
> Hive breaks Hadoop commons logging with log4j2
> --
>
> Key: HIVE-15531
> URL: https://issues.apache.org/jira/browse/HIVE-15531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
>Priority: Minor
> Attachments: HIVE-15531.patch
>
>
> Hadoop (2.7), which is using Commons-logging is not compatible with log4j2
> without bridge.
> The bridge is missing in Hive.
> This leads to a problem whereby commons-logging initialises a log4j (1.2)
> version, does not configure it properly since configuration for it is missing
> and sends logging output to stdout (the default).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[
https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796261#comment-15796261
]
Hive QA commented on HIVE-15531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12845384/HIVE-15531.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10899 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[case_sensitivity]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_coalesce]
(batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=96)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2768/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2768/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2768/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12845384 - PreCommit-HIVE-Build
> Hive breaks Hadoop commons logging with log4j2
> --
>
> Key: HIVE-15531
> URL: https://issues.apache.org/jira/browse/HIVE-15531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
>Priority: Minor
> Attachments: HIVE-15531.patch
>
>
> Hadoop (2.7), which is using Commons-logging is not compatible with log4j2
> without bridge.
> The bridge is missing in Hive.
> This leads to a problem whereby commons-logging initialises a log4j (1.2)
> version, does not configure it properly since configuration for it is missing
> and sends logging output to stdout (the default).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795975#comment-15795975 ] Sergey Shelukhin commented on HIVE-15531: - [~dhiraj.kumar] can you change it to patch available, so that the tests would run? > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version, does not configure it properly since configuration for it is missing > and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795972#comment-15795972 ] Sergey Shelukhin commented on HIVE-15531: - +1 pending tests > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version, does not configure it properly since configuration for it is missing > and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
