[
https://issues.apache.org/jira/browse/KUDU-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Evans updated KUDU-3157:
-----------------------------
Description:
It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
{{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started being included.
{code}
# for 1.7.0
jar tvf
~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.7.0/kudu-spark2_2.11-1.7.0.jar
| grep slf4j
1203 Wed Jun 29 14:43:04 CDT 2016 META-INF/license/LICENSE.slf4j.txt
#for 1.12.0
jar tvf
~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.12.0/kudu-spark2_2.11-1.12.0.jar
| grep slf4j
0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/
0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/event/
555 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventConstants.class
9842 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventRecodingLogger.class
1450 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/Level.class
432 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/LoggingEvent.class
2615 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/SubstituteLoggingEvent.class
0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/helpers/
3285 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarker.class
1634 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarkerFactory.class
1233 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter$1.class
2618 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter.class
1091 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/FormattingTuple.class
4773 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MarkerIgnoringBase.class
7553 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MessageFormatter.class
823 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NamedLoggerBase.class
3267 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLogger.class
584 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLoggerFactory.class
1249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPMDCAdapter.class
10262 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/SubstituteLogger.class
2199 Mon Dec 16 22:03:32 CST 2019
org/slf4j/helpers/SubstituteLoggerFactory.class
196 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util$1.class
765 Mon Dec 16 22:03:32 CST 2019
org/slf4j/helpers/Util$ClassContextSecurityManager.class
2952 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util.class
180 Mon Dec 16 22:03:32 CST 2019 org/slf4j/ILoggerFactory.class
272 Mon Dec 16 22:03:32 CST 2019 org/slf4j/IMarkerFactory.class
1375 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Logger.class
12575 Mon Dec 16 22:03:32 CST 2019 org/slf4j/LoggerFactory.class
666 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Marker.class
1621 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MarkerFactory.class
177 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$1.class
749 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$MDCCloseable.class
3675 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC.class
0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/spi/
455 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LocationAwareLogger.class
249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LoggerFactoryBinder.class
249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MarkerFactoryBinder.class
529 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MDCAdapter.class
0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/
0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/slf4j-api/
3835 Mon Dec 16 21:59:52 CST 2019 META-INF/maven/org.slf4j/slf4j-api/pom.xml
109 Mon Dec 16 22:00:08 CST 2019
META-INF/maven/org.slf4j/slf4j-api/pom.properties
{code}
These should be removed from the jar, or at least relocated so as not to clash
with Spark vendor dependencies, which can cause {{LinkageError}} in Spark
clusters. Real life example of this problem, seen on a Spark job started on
CDH 6.3 cluster:
{code}
Exception in thread "main" java.lang.LinkageError: loader constraint violation:
when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader)
of the current class, org/slf4j/LoggerFactory, and the class loader (instance
of sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
{code}
was:
It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
{{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started being included.
These should be removed from the jar, or at least relocated so as not to clash
with Spark vendor dependencies, which can cause {{LinkageError}} in Spark
clusters. Real life example of this problem, seen on a Spark job started on
CDH 6.3 cluster:
{code}
Exception in thread "main" java.lang.LinkageError: loader constraint violation:
when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader)
of the current class, org/slf4j/LoggerFactory, and the class loader (instance
of sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
{code}
> kudu-spark2 jar is now including slf4j classes
> ----------------------------------------------
>
> Key: KUDU-3157
> URL: https://issues.apache.org/jira/browse/KUDU-3157
> Project: Kudu
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.12.0
> Reporter: Jeff Evans
> Priority: Major
>
> It seems that sometime between versions {{1.7.0}} and {{1.12.0}} of
> {{org.apache.kudu:kudu-spark2_2.11}}, some SLF4J classes started being
> included.
> {code}
> # for 1.7.0
> jar tvf
> ~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.7.0/kudu-spark2_2.11-1.7.0.jar
> | grep slf4j
> 1203 Wed Jun 29 14:43:04 CDT 2016 META-INF/license/LICENSE.slf4j.txt
> #for 1.12.0
> jar tvf
> ~/.m2/repository/org/apache/kudu/kudu-spark2_2.11/1.12.0/kudu-spark2_2.11-1.12.0.jar
> | grep slf4j
> 0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/
> 0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/event/
> 555 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventConstants.class
> 9842 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/EventRecodingLogger.class
> 1450 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/Level.class
> 432 Mon Dec 16 22:03:32 CST 2019 org/slf4j/event/LoggingEvent.class
> 2615 Mon Dec 16 22:03:32 CST 2019
> org/slf4j/event/SubstituteLoggingEvent.class
> 0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/helpers/
> 3285 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarker.class
> 1634 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMarkerFactory.class
> 1233 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter$1.class
> 2618 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/BasicMDCAdapter.class
> 1091 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/FormattingTuple.class
> 4773 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MarkerIgnoringBase.class
> 7553 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/MessageFormatter.class
> 823 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLogger.class
> 584 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPLoggerFactory.class
> 1249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/NOPMDCAdapter.class
> 10262 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/SubstituteLogger.class
> 2199 Mon Dec 16 22:03:32 CST 2019
> org/slf4j/helpers/SubstituteLoggerFactory.class
> 196 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util$1.class
> 765 Mon Dec 16 22:03:32 CST 2019
> org/slf4j/helpers/Util$ClassContextSecurityManager.class
> 2952 Mon Dec 16 22:03:32 CST 2019 org/slf4j/helpers/Util.class
> 180 Mon Dec 16 22:03:32 CST 2019 org/slf4j/ILoggerFactory.class
> 272 Mon Dec 16 22:03:32 CST 2019 org/slf4j/IMarkerFactory.class
> 1375 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Logger.class
> 12575 Mon Dec 16 22:03:32 CST 2019 org/slf4j/LoggerFactory.class
> 666 Mon Dec 16 22:03:32 CST 2019 org/slf4j/Marker.class
> 1621 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MarkerFactory.class
> 177 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$1.class
> 749 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC$MDCCloseable.class
> 3675 Mon Dec 16 22:03:32 CST 2019 org/slf4j/MDC.class
> 0 Mon Apr 27 14:53:30 CDT 2020 org/slf4j/spi/
> 455 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LocationAwareLogger.class
> 249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/LoggerFactoryBinder.class
> 249 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MarkerFactoryBinder.class
> 529 Mon Dec 16 22:03:32 CST 2019 org/slf4j/spi/MDCAdapter.class
> 0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/
> 0 Mon Apr 27 14:53:30 CDT 2020 META-INF/maven/org.slf4j/slf4j-api/
> 3835 Mon Dec 16 21:59:52 CST 2019 META-INF/maven/org.slf4j/slf4j-api/pom.xml
> 109 Mon Dec 16 22:00:08 CST 2019
> META-INF/maven/org.slf4j/slf4j-api/pom.properties
> {code}
> These should be removed from the jar, or at least relocated so as not to
> clash with Spark vendor dependencies, which can cause {{LinkageError}} in
> Spark clusters. Real life example of this problem, seen on a Spark job
> started on CDH 6.3 cluster:
> {code}
> Exception in thread "main" java.lang.LinkageError: loader constraint
> violation: when resolving method
> "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
> the class loader (instance of
> org/apache/spark/util/ChildFirstURLClassLoader) of the current class,
> org/slf4j/LoggerFactory, and the class loader (instance of
> sun/misc/Launcher$AppClassLoader) for the method\'s defining class,
> org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
> org/slf4j/ILoggerFactory used in the signature
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:423)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
