Spark on Mesos Issue - Do I need to install Spark on Mesos slaves

Hi, I am trying to submit a Spark job on Mesos using spark-submit from my Mesos-Master machine. My SPARK_HOME = /vol1/spark/spark-1.0.2-bin-hadoop2 I have uploaded the spark-1.0.2-bin-hadoop2.tgz to hdfs so that the mesos slaves can download it to invoke the Mesos Spark backend executor. But

NotSerializableException: org.apache.spark.sql.hive.api.java.JavaHiveContext

Hello All, I am trying to query a Hive table using Spark SQL from my java code,but getting the following error: *Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:

Re: HDFS Server/Client IPC version mismatch while trying to access HDFS files using Spark-0.9.1

deb bijoy.comput...@gmail.com wrote: Any suggestions from anyone? Thanks Bijoy On Tue, Jun 10, 2014 at 11:46 PM, bijoy deb bijoy.comput...@gmail.com wrote: Hi all, I have build Shark-0.9.1 using sbt using the below command: SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 sbt

Re: HDFS Server/Client IPC version mismatch while trying to access HDFS files using Spark-0.9.1

Any suggestions from anyone? Thanks Bijoy On Tue, Jun 10, 2014 at 11:46 PM, bijoy deb bijoy.comput...@gmail.com wrote: Hi all, I have build Shark-0.9.1 using sbt using the below command: *SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 sbt/sbt assembly* My Hadoop cluster is also having version

HDFS Server/Client IPC version mismatch while trying to access HDFS files using Spark-0.9.1

Hi all, I have build Shark-0.9.1 using sbt using the below command: *SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.6.0 sbt/sbt assembly* My Hadoop cluster is also having version 2.0.0-mr1-cdh4.6.0. But when I try to execute the below command from Spark shell,which reads a file from HDFS, I get the IPC

Is Spark-1.0.0 not backward compatible with Shark-0.9.1 ?

Hi, I am trying to run build Shark-0.9.1 from source,with Spark-1.0.0 as its dependency,using sbt package command.But I am getting the below error during build,which is making me think that perhaps Spark-1.0.0 is not compatible with Shark-0.9.1: [info] Compilation completed in 9.046 s

Integration issue between Apache Shark-0.9.1 (with in-house hive-0.11) and pre-existing CDH4.6 HIVE-0.10 server

Hi all, I have installed Apache Shark 0.9.1 on my machine which comes bundled with hive-0.11 version of hive jars.I am trying to integrate this with my pre-existing CDH-4.6 version of the Hive server which is of version 0.10.On pointing HIVE_HOME in spark-env.sh to the cloudera version of the

Re: Integration issue between Apache Shark-0.9.1 (with in-house hive-0.11) and pre-existing CDH4.6 HIVE-0.10 server

-communicate-with-client-version Can you try finding matching jars for your Hadoop cluster? On Wed, May 28, 2014 at 8:47 AM, bijoy deb bijoy.comput...@gmail.comwrote: Hi all, I have installed Apache Shark 0.9.1 on my machine which comes bundled with hive-0.11 version of hive jars.I am trying