[
https://issues.apache.org/jira/browse/MESOS-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976194#comment-13976194
]
Patrick Wendell commented on MESOS-1203:
----------------------------------------
This problem is complicated substantially due to the fact that protobufs are
part of the public API of mesos and not just something used internally. When I
proposed this shading originally I wasn't aware of that wrinkle.
I'm pretty sure the reason why Spark doesn't explicitly declare a protobuf
dependency is because we always get it anyways from sub-projects. It's actually
tricky to hard code a particular version because the Spark build is
parameterized and people build Spark with multiple different Hadoop versions.
So there is no "one version of protobuf" that would work correctly in all
parameterizations of the Spark build, sadly.
In Spark we have three sources of the protobuf dependency, (a) Akka (b)
Mesos and (c) Hadoop. We've shaded the akka version so it never conflicts with
anyone, but the Hadoop and Mesos versions can conflict.
We'd be happy to use a shaded package name in our interactions with the Mesos
API. This is an entirely internal thing within Spark and the code that uses
this is tiny.
> Shade protobuf dependency in Mesos Java library
> -----------------------------------------------
>
> Key: MESOS-1203
> URL: https://issues.apache.org/jira/browse/MESOS-1203
> Project: Mesos
> Issue Type: Improvement
> Components: build
> Reporter: Patrick Wendell
>
> Mesos's Java library uses the protobuf library which is also used by Hadoop.
> Unfortunately the protobuf library does not provide binary compatiblity
> between minor versions (for code compiled against 2.4.1 and 2.5.0 cannot run
> together in a single JVM classlaoder) .
> This makes use of Mesos via it's Java API, something that is required for
> Spark and I'm assuming other frameworks, fundamentally incompatible for
> certain Hadoop versions.
> Mesos could shade this jar using the maven shade plug-in. Take a look at the
> Parquet project for an example of shading:
> https://github.com/Parquet/parquet-format/blob/master/pom.xml#L140
> Without this fix Java users won't be able to use Mesos (< 0.17) with newer
> versions of Hadoop. Or Mesos 0.17+ with older versions of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
