Update: I revuild my mesos-exector-image ,I download *spark-streaming-kafka_2.10-1.6.0.jar* on *`/linker/jars`*
I change my submit command: dcos spark run \ --submit-args='--jars > local:/linker/jars/spark-streaming-kafka_2.10-1.6.0.jar spark2cassandra.py > 10.140.0.14:2181 wlu_spark2cassandra' --docker-image > adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2 Where I get new stderr output on mesos: I only problem is submit the dependency spark-streaming-kafka_2.10-1.6.0.jar to worker. Thanks. 2016-07-13 18:57 GMT+08:00 Luke Adolph <kenan3...@gmail.com>: > Hi all: > My spark runs on mesos.I write a spark streaming app using python, code > on GitHub <https://github.com/adolphlwq/linkerProcessorSample>. > > The app has dependency " > *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*". > > Spark on mesos has two important concepts: Spark Framework and Spark > exector. > > I set my exector run in docker image.The docker image Dockerfile > <https://github.com/adolphlwq/linkerProcessorSample/blob/master/docker/Dockerfile> > is > below: > > # refer ' >> http://spark.apache.org/docs/latest/running-on-mesos.html#spark-properties' >> on 'spark.mesos.executor.docker.image' section > > FROM ubuntu:14.04 >> WORKDIR /linker >> RUN ln -f -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime >> #download mesos >> RUN echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" > >> /etc/apt/sources.list.d/mesosphere.list && \ >> apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ >> apt-get update && \ >> apt-get -y install mesos=0.28.1-2.0.20.ubuntu1404 openjdk-7-jre >> python-pip git vim curl >> RUN git clone https://github.com/adolphlwq/linkerProcessorSample.git && \ >> pip install -r linkerProcessorSample/docker/requirements.txt >> RUN curl -fL >> http://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz >> | tar xzf - -C /usr/local && \ >> apt-get clean >> ENV MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \ >> JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 \ >> SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6 >> ENV PATH=$JAVA_HOME/bin:$PATH >> WORKDIR $SPARK_HOME > > > When I use below command to submit my app program: > > dcos spark run --submit-args='--packages >> org.apache.spark:spark-streaming-kafka_2.10:1.6.1 \ >> spark2cassandra.py zk topic' \ >> -docker-image=adolphlwq/mesos-for-spark-exector-image:1.6.0.beta > > > The exector docker container run successfully, but it has no package for > *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*. > > The *stderr* om mesos is: > > I0713 09:34:52.715551 18124 logging.cpp:188] INFO level logging started! >> I0713 09:34:52.717797 18124 fetcher.cpp:424] Fetcher Info: >> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/home\/ubuntu\/spark2cassandra.py"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.101tec_zkclient-0.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka_2.10-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.slf4j_slf4j-api-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.spark-project.spark_unused-1.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/net.jpountz.lz4_lz4-1.3.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/log4j_log4j-1.2.17.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.yammer.metrics_metrics-core-2.2.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka-clients-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.xerial.snappy_snappy-java-1.1.2.jar"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4\/frameworks\/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024\/executors\/driver-20160713093451-0015\/runs\/84419372-9482-4c58-8f87-4ba528b6885c"} >> I0713 09:34:52.719846 18124 fetcher.cpp:379] Fetching URI >> '/home/ubuntu/spark2cassandra.py' >> I0713 09:34:52.719866 18124 fetcher.cpp:250] Fetching directly into the >> sandbox directory >> I0713 09:34:52.719925 18124 fetcher.cpp:187] Fetching URI >> '/home/ubuntu/spark2cassandra.py' >> I0713 09:34:52.719945 18124 fetcher.cpp:167] Copying resource with >> command:cp '/home/ubuntu/spark2cassandra.py' >> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py' >> W0713 09:34:52.722587 18124 fetcher.cpp:272] Copying instead of >> extracting resource from URI with 'extract' flag, because it does not seem >> to be an archive: /home/ubuntu/spark2cassandra.py >> I0713 09:34:52.724138 18124 fetcher.cpp:456] Fetched >> '/home/ubuntu/spark2cassandra.py' to >> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py' >> I0713 09:34:52.724148 18124 fetcher.cpp:379] Fetching URI >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar' >> I0713 09:34:52.724153 18124 fetcher.cpp:250] Fetching directly into the >> sandbox directory >> I0713 09:34:52.724162 18124 fetcher.cpp:187] Fetching URI >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar' >> I0713 09:34:52.724171 18124 fetcher.cpp:167] Copying resource with >> command:cp >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar' >> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar' >> cp: cannot stat >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar': >> No such file or directory >> Failed to fetch >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar': >> Failed to copy with command 'cp >> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar' > > > May somebody help me to solve the dependencies problem! > > Thanks. > -- > Thanks & Best Regards > 卢文泉 | Adolph Lu > TEL：+86 15651006559 > Linker Networks(http://www.linkernetworks.com/) > -- Thanks & Best Regards 卢文泉 | Adolph Lu TEL：+86 15651006559 Linker Networks(http://www.linkernetworks.com/)
