Dear list,
I’m here for the first time so I do apologise if I’m misbehaving. I am
currently trying running Scala Spark Notebook on Mesos with Docker in bridge
networking. I would like to launch spark executors using the cluster, the
master is running on. This requires registering a framework from inside of the
container and being able to accept the offers back inside. I found out that the
necessary settings, LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTISE_PORT, were
added in mesos 0.24.0, which seems to be released (at least tagged in git). I
have the cluster running mesos 0.24.0 and this seems to be behaving really good
so far.
However, I’m having a problem with the Spark Notebook docker image.
The way I build mesos itself, I use mesosphere/mesos-deb-packaging:
git clone https://github.com/mesosphere/mesos-deb-packaging.git .
git checkout d7e5b7b5a8a04b11eaee6a1f9a0962ef3e77864a
And I’m applying this patch (mesos 0.24.0):
diff --git a/build_mesos b/build_mesos
index 81561bc..f756ef0 100755
--- a/build_mesos
+++ b/build_mesos
@@ -313,9 +313,10 @@ function deb_ {
--deb-recommends zookeeperd
--deb-recommends zookeeper-bin
-d 'java-runtime-headless'
- -d libcurl3
- -d libsvn1
- -d libsasl2-modules
+ -d libcurl4-nss-dev
+ -d libsasl2-dev
+ -d libapr1-dev
+ -d libsvn-dev
--after-install "$this/$asset_dir/mesos.postinst"
--after-remove "$this/$asset_dir/mesos.postrm" )
rm -f "$this”/pkg.deb
My mesos setup
git clone https://github.com/apache/mesos.git .
git fetch origin
git checkout 0.24.0
Having gcc-4.9.2, oracle-java7-installer and the following installed (ubuntu
12.04):
apt-get install -y autoconf libtool
apt-get install -y python-dev python-boto libcurl4-nss-dev libsasl2-dev
maven libapr1-dev libsvn-dev git-core software-properties-common \
python-software-properties ruby1.9.1 ruby1.9.1-dev \
build-essential libxslt-dev libxml2-dev \
wget zlibc zlib1g zlib1g-dev
gem install fpm --no-ri --no-rdoc
Building mesos with command:
mkdir -p versions/0.24.0
cd versions/0.24.0
git clone https://github.com/apache/mesos.git .
git fetch origin
git checkout 0.24.0
cd ../..
./build_mesos --src-dir versions/0.24.0
Gives me a DEB package which can be successfully installed on a clean
ubuntu:14.04 docker image by installing only these:
apt-get install -y openjdk-7-jre build-essential python-dev python-boto
libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev
I build on ubuntu:12.04. The solution needs to run on 12.04.
This is how I build the notebook, there are 2 images, the first one tagged as
myorg/openjdk-7-jre-trusty, the second generated by sbt, however the most
important commands are listed.
# myorg/openjdk-7-jre-trusty
FROM ubuntu:trusty
MAINTAINER ...
RUN \
apt-get update && \
apt-get install -y openjdk-7-jre && \
rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
The notebook itself:
FROM myorg/openjdk-7-jre-trusty
MAINTAINER …
RUN apt-get update -y && apt-get install -y wget build-essential python-dev
python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev
RUN cd /tmp && wget
https://to.my/location/where/I/store/built/mesos-0.24.0.deb && dpkg -i
mesos-0.24.0.deb
ENV MESOS_JAVA_NATIVE_LIBRARY /usr/lib/libmesos.so
ENV MESOS_LOG_DIR /var/log/mesos
I attempt running this code in Spark Notebook:
# all the hosts used are resolvable, files can be accessed on HDFS:
reset("spark-notebook-tests", lastChanges = (c:SparkConf) => {
c.set("spark.mesos.executor.uri",
"hdfs:///resources/spark-1.4.1-cdh4.tar.gz")
.set("spark.master",
"mesos://zk://zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181/mesos")
.set("spark.local.dir", System.getenv("MESOS_SANDBOX"))
})
Spark notebook launched with this command:
export LIBPROCESS_PORT=10000; export
LIBPROCESS_ADVERTISE_IP=10.XXX.XXX.XXX; export
LIBPROCESS_ADVERTISE_PORT=$PORT1; /opt/docker/bin/spark-notebook
The result should be: at least a trace in logs that the offers can’t be sent
and framework registration attempt. Instead, I receive an error, listed below:
java.lang.UnsatisfiedLinkError: /usr/lib/libmesos-0.24.0.so:
/usr/lib/x86_64-linux-gnu/libcurl-nss.so.4: version `CURL_3' not found
(required by /usr/lib/libmesos-0.24.0.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1880)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:54)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:79)
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2537)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:489)
at $iwC$$iwC$$anon$1.reset(<console>:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:48)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:57)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:59)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:61)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:63)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:65)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:67)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:69)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:71)
at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:73)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:75)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:77)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:79)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:81)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:83)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:85)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:87)
at $iwC$$iwC$$iwC.<init>(<console>:89)
at $iwC$$iwC.<init>(<console>:91)
at $iwC.<init>(<console>:93)
at <init>(<console>:95)
at .<init>(<console>:99)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at notebook.kernel.Repl$$anonfun$3.apply(Repl.scala:173)
at notebook.kernel.Repl$$anonfun$3.apply(Repl.scala:173)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at scala.Console$.withOut(Console.scala:126)
at notebook.kernel.Repl.evaluate(Repl.scala:172)
at
notebook.client.ReplCalculator$$anonfun$10$$anon$1$$anonfun$22.apply(ReplCalculator.scala:348)
at
notebook.client.ReplCalculator$$anonfun$10$$anon$1$$anonfun$22.apply(ReplCalculator.scala:345)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I am not sure if I understand the dependencies wrong or there’s another issue.
I have seen this issue mentioned twice in google cache of IRC channels. No
solution though. What could be possibly wrong here?
Any pointers appreciated.
Kind regards,
Radek Gruchalski
[email protected] (mailto:[email protected])
(mailto:[email protected])
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must
you copy or show it to anyone; please delete/destroy and inform the sender
immediately.
