Matt Cheah created SPARK-10374:
----------------------------------
Summary: Spark-core 1.5.0-RC2 can create version conflicts with
apps depending on protobuf-2.4
Key: SPARK-10374
URL: https://issues.apache.org/jira/browse/SPARK-10374
Project: Spark
Issue Type: Bug
Affects Versions: 1.5.0
Reporter: Matt Cheah
Priority: Blocker
Fix For: 1.5.0
My Hadoop cluster is running 2.0.0-CDH4.7.0, and I have an application that
depends on the Spark 1.5.0 libraries via Gradle, and Hadoop 2.0.0 libraries.
When I run the driver application, I can hit the following error:
{code}
<redacted other messages>… java.lang.UnsupportedOperationException: This is
supposed to be overridden by subclasses.
at
com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:30108)
at
com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
{code}
This application used to work when pulling in Spark 1.4.1 dependencies, and
thus this is a regression.
I used Gradle’s dependencyInsight task to dig a bit deeper. Against our Spark
1.4.1-backed project, it shows that dependency resolution pulls in Protobuf
2.4.0a from the Hadoop CDH4 modules and Protobuf 2.5.0-spark from the Spark
modules. It appears that Spark used to shade its protobuf dependencies and
hence Spark’s and Hadoop’s protobuf dependencies wouldn’t collide. However when
I ran dependencyInsight again against Spark 1.5 and it looks like protobuf is
no longer shaded from the Spark module.
1.4.1 dependencyInsight:
{code}
com.google.protobuf:protobuf-java:2.4.0a
+--- org.apache.hadoop:hadoop-common:2.0.0-cdh4.6.0
| \--- org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.6.0
| +--- compile
| \--- org.apache.spark:spark-core_2.10:1.4.1
| +--- compile
| +--- org.apache.spark:spark-sql_2.10:1.4.1
| | \--- compile
| \--- org.apache.spark:spark-catalyst_2.10:1.4.1
| \--- org.apache.spark:spark-sql_2.10:1.4.1 (*)
\--- org.apache.hadoop:hadoop-hdfs:2.0.0-cdh4.6.0
\--- org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.6.0 (*)
org.spark-project.protobuf:protobuf-java:2.5.0-spark
\--- org.spark-project.akka:akka-remote_2.10:2.3.4-spark
\--- org.apache.spark:spark-core_2.10:1.4.1
+--- compile
+--- org.apache.spark:spark-sql_2.10:1.4.1
| \--- compile
\--- org.apache.spark:spark-catalyst_2.10:1.4.1
\--- org.apache.spark:spark-sql_2.10:1.4.1 (*)
{code}
1.5.0-rc2 dependencyInsight:
{code}
com.google.protobuf:protobuf-java:2.5.0 (conflict resolution)
\--- com.typesafe.akka:akka-remote_2.10:2.3.11
\--- org.apache.spark:spark-core_2.10:1.5.0-rc2
+--- compile
+--- org.apache.spark:spark-sql_2.10:1.5.0-rc2
| \--- compile
\--- org.apache.spark:spark-catalyst_2.10:1.5.0-rc2
\--- org.apache.spark:spark-sql_2.10:1.5.0-rc2 (*)
com.google.protobuf:protobuf-java:2.4.0a -> 2.5.0
+--- org.apache.hadoop:hadoop-common:2.0.0-cdh4.6.0
| \--- org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.6.0
| +--- compile
| \--- org.apache.spark:spark-core_2.10:1.5.0-rc2
| +--- compile
| +--- org.apache.spark:spark-sql_2.10:1.5.0-rc2
| | \--- compile
| \--- org.apache.spark:spark-catalyst_2.10:1.5.0-rc2
| \--- org.apache.spark:spark-sql_2.10:1.5.0-rc2 (*)
\--- org.apache.hadoop:hadoop-hdfs:2.0.0-cdh4.6.0
\--- org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.6.0 (*)
{code}
Clearly we can't force the version to be one way or the other. If I force
protobuf to use 2.5.0, then invoking Hadoop code from my application will break
as Hadoop 2.0.0 jars are compiled against protobuf-2.4. On the other hand,
forcing protobuf to use version 2.4 breaks spark-core code that is compiled
against protobuf-2.5. Note that protobuf-2.4 and protobuf-2.5 are not binary
compatible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
