You can avoid install Spark on each node by uploading Spark distribution
tarball file to HDFS setting |spark.executor.uri| to the HDFS location.
In this way, Mesos will download and the tarball file before launching
containers. Please refer to this Spark documentation page
<http://spark.apache.org/docs/latest/running-on-mesos.html> for details.
However, using |spark.executor.uri| together with fine-grained mode
(which is the default mode) really kills performance, because Mesos
downloads and extracts the tarball every time a Spark /task/ (not
application) is launched.
On 9/21/14 1:16 AM, John Omernik wrote:
I am running the Thrift server in SparkSQL, and running it on the node
I compiled spark on. When I run it, tasks only work if they landed on
that node, other executors started on nodes I didn't compile spark on
(and thus don't have the compile directory) fail. Should spark be
distributed properly with the executor uri in my spark-defaults for
mesos?
Here is the error on nodes with Lost executors
sh: 1: /opt/mapr/spark/spark-1.1.0-SNAPSHOT/sbin/spark-executor: not found
