Re: Using a different spark jars than the one on the cluster

2015-03-23 Thread Denny Lee 
+1  - I currently am doing what Marcelo is suggesting as I have a CDH 5.2
cluster (with Spark 1.1) and I'm also running Spark 1.3.0+ side-by-side in
my cluster.

On Wed, Mar 18, 2015 at 1:23 PM Marcelo Vanzin van...@cloudera.com wrote:

 Since you're using YARN, you should be able to download a Spark 1.3.0
 tarball from Spark's website and use spark-submit from that
 installation to launch your app against the YARN cluster.

 So effectively you would have 1.2.0 and 1.3.0 side-by-side in your cluster.

 On Wed, Mar 18, 2015 at 11:09 AM, jaykatukuri jkatuk...@apple.com wrote:
  Hi all,
  I am trying to run my job which needs spark-sql_2.11-1.3.0.jar.
  The cluster that I am running on is still on spark-1.2.0.
 
  I tried the following :
 
  spark-submit --class class-name --num-executors 100 --master yarn
  application_jar--jars hdfs:///path/spark-sql_2.11-1.3.0.jar
  hdfs:///input_data
 
  But, this did not work, I get an error that it is not able to find a
  class/method that is in spark-sql_2.11-1.3.0.jar .
 
  org.apache.spark.sql.SQLContext.implicits()Lorg/
 apache/spark/sql/SQLContext$implicits$
 
  The question in general is how do we use a different version of spark
 jars
  (spark-core, spark-sql, spark-ml etc) than the one's running on a
 cluster ?
 
  Thanks,
  Jay
 
 
 
 
 
  --
  View this message in context: http://apache-spark-user-list.
 1001560.n3.nabble.com/Using-a-different-spark-jars-than-the-
 one-on-the-cluster-tp22125.html
  Sent from the Apache Spark User List mailing list archive at Nabble.com.
 
  -
  To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
  For additional commands, e-mail: user-h...@spark.apache.org
 



 --
 Marcelo

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

Re: Using a different spark jars than the one on the cluster

2015-03-18 Thread Marcelo Vanzin 
Since you're using YARN, you should be able to download a Spark 1.3.0
tarball from Spark's website and use spark-submit from that
installation to launch your app against the YARN cluster.

So effectively you would have 1.2.0 and 1.3.0 side-by-side in your cluster.

On Wed, Mar 18, 2015 at 11:09 AM, jaykatukuri jkatuk...@apple.com wrote:
 Hi all,
 I am trying to run my job which needs spark-sql_2.11-1.3.0.jar.
 The cluster that I am running on is still on spark-1.2.0.

 I tried the following :

 spark-submit --class class-name --num-executors 100 --master yarn
 application_jar--jars hdfs:///path/spark-sql_2.11-1.3.0.jar
 hdfs:///input_data

 But, this did not work, I get an error that it is not able to find a
 class/method that is in spark-sql_2.11-1.3.0.jar .

 org.apache.spark.sql.SQLContext.implicits()Lorg/apache/spark/sql/SQLContext$implicits$

 The question in general is how do we use a different version of spark jars
 (spark-core, spark-sql, spark-ml etc) than the one's running on a cluster ?

 Thanks,
 Jay





 --
 View this message in context: 
 http://apache-spark-user-list.1001560.n3.nabble.com/Using-a-different-spark-jars-than-the-one-on-the-cluster-tp22125.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 
Marcelo

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org