[
https://issues.apache.org/jira/browse/SPARK-30783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated SPARK-30783:
-
Description:
hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate
classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark can
pick up classes defined in hive instead of its thrift server module, which can
cause hard to debug runtime errors due to class loading order and compilation
errors for applications depend on spark.
If you compare hive-service-rpc 2.3.6's jar
([https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar])
and spark thrift server's jar (e.g.
[https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar),]
you will see that all of classes provided by hive-service-rpc-2.3.6.jar are
covered by spark thrift server's jar. I am attaching the list of jar contents
for your reference.
was:hive-service-rpc 2.3.6 and spark sql's thrift server module have
duplicate classes. Leaving hive-service-rpc 2.3.6 in the class path means that
spark can pick up classes defined in hive instead of its thrift server module,
which can cause hard to debug runtime errors due to class loading order and
compilation errors for applications depend on spark.
> Hive 2.3 profile should exclude hive-service-rpc
>
>
> Key: SPARK-30783
> URL: https://issues.apache.org/jira/browse/SPARK-30783
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Blocker
>
> hive-service-rpc 2.3.6 and spark sql's thrift server module have duplicate
> classes. Leaving hive-service-rpc 2.3.6 in the class path means that spark
> can pick up classes defined in hive instead of its thrift server module,
> which can cause hard to debug runtime errors due to class loading order and
> compilation errors for applications depend on spark.
>
> If you compare hive-service-rpc 2.3.6's jar
> ([https://search.maven.org/remotecontent?filepath=org/apache/hive/hive-service-rpc/2.3.6/hive-service-rpc-2.3.6.jar])
> and spark thrift server's jar (e.g.
> [https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hive-thriftserver_2.12/3.0.0-SNAPSHOT/spark-hive-thriftserver_2.12-3.0.0-20200207.021914-364.jar),]
> you will see that all of classes provided by hive-service-rpc-2.3.6.jar are
> covered by spark thrift server's jar. I am attaching the list of jar contents
> for your reference.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
