Jason Dere created HIVE-12179:
---------------------------------
Summary: Add option to not add spark-assembly.jar to Hive classpath
Key: HIVE-12179
URL: https://issues.apache.org/jira/browse/HIVE-12179
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Jason Dere
After running the following Hive script:
{noformat}
add jar hdfs:///tmp/junit-4.11.jar;
show tables;
{noformat}
I can see the following lines getting printed to stdout when Hive exits:
{noformat}
WARN: The method class
org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked.
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.
{noformat}
Also seeing the following warnings in stderr:
{noformat}
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{noformat}
It looks like this is due to the addition of the shaded spark-assembly.jar to
the classpath, which contains classes from icl-over-slf4j.jar (which is causing
the stdout messages) and slf4j-log4j12.jar.
Removing spark-assembly.jar from being added to the classpath causes these
messages to go away. It would be good to have a way to specify that Hive not
add spark-assembly.jar to the class path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
