Hyea,
I still have to try it myself (I'm trying to create GCE images with Spark
on Mesos 0.18.0) but I think your change is one of the required ones,
however my gut feeling is that others will be required to have this working.
Actually, in my understanding, this core dump is due to protobuf
incompatibilities between 2.4.1 and 2.5.0. So I guess that we should also
recompile or update or ... these projects:
"org.spark-project.akka" %% "akka-remote" %
"2.2.3-shaded-protobuf" excludeAll(excludeNetty),
"org.spark-project.akka" %% "akka-slf4j" %
"2.2.3-shaded-protobuf" excludeAll(excludeNetty),
because they will come with:
"org.spark-project.protobuf" %% "protobuf-java" %
"2.4.1-shaded"
And maybe, to force to whole thing, add this deps:
"com.google.protobuf" % "protobuf-java" % "2.5.0"
That's only my0.02c in order to try to move things forward until someone
from the Spark team to step in with more clear and strong advices...
kr,
Andy Petrella
Belgium (Liège)
* *********
Data Engineer in *NextLab <http://nextlab.be/> sprl* (owner)
Engaged Citizen Coder for *WAJUG <http://wajug.be/>* (co-founder)
Author of *Learning Play! Framework 2
<http://www.packtpub.com/learning-play-framework-2/book>*
Bio: on visify <https://www.vizify.com/es/52c3feec2163aa0010001eaa>
On Thu, Apr 17, 2014 at 8:29 PM, Steven Cox <[email protected]> wrote:
> So I tried a fix found on the list...
>
> "The issue was due to meos version mismatch as I am using latest mesos
> 0.17.0,
> but spark uses 0.13.0.
> Fixed by updating the SparkBuild.scala to latest version."
>
> I changed this line in SparkBuild.scala
> "org.apache.mesos" % "mesos" % "0.13.0",
> to
> "org.apache.mesos" % "mesos" % "0.18.0",
>
> ...ran make-distribution.sh, repackaged and redeployed the tar.gz to HDFS.
>
> It still core dumps like this:
> https://gist.github.com/stevencox/11002498
>
> In this environment:
> Ubuntu 13.10
> Mesos 0.18.0
> Spark 0.9.1
> JDK 1.7.0_45
> Scala 2.10.1
>
> What am I missing?
>
