Re: Master options Cluster/Client descrepencies.
Have a look at http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211 Thanks Best Regards On Wed, Mar 30, 2016 at 12:09 AM, satyajit vegesna < satyajit.apas...@gmail.com> wrote: > > Hi All, > > I have written a spark program on my dev box , >IDE:Intellij >scala version:2.11.7 >spark verison:1.6.1 > > run fine from IDE, by providing proper input and output paths including > master. > > But when i try to deploy the code in my cluster made of below, > >Spark version:1.6.1 > built from source pkg using scala 2.11 > But when i try spark-shell on cluster i get scala version to be > 2.10.5 > hadoop yarn cluster 2.6.0 > > and with additional options, > > --executor-memory > --total-executor-cores > --deploy-mode cluster/client > --master yarn > > i get Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; > at com.movoto.SparkPost$.main(SparkPost.scala:36) > at com.movoto.SparkPost.main(SparkPost.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > i understand this to be a scala version issue, as i have faced this before. > > Is there something that i have change and try things to get the same > program running on cluster. > > Regards, > Satyajit. > >
Fwd: Master options Cluster/Client descrepencies.
Hi All, I have written a spark program on my dev box , IDE:Intellij scala version:2.11.7 spark verison:1.6.1 run fine from IDE, by providing proper input and output paths including master. But when i try to deploy the code in my cluster made of below, Spark version:1.6.1 built from source pkg using scala 2.11 But when i try spark-shell on cluster i get scala version to be 2.10.5 hadoop yarn cluster 2.6.0 and with additional options, --executor-memory --total-executor-cores --deploy-mode cluster/client --master yarn i get Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at com.movoto.SparkPost$.main(SparkPost.scala:36) at com.movoto.SparkPost.main(SparkPost.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) i understand this to be a scala version issue, as i have faced this before. Is there something that i have change and try things to get the same program running on cluster. Regards, Satyajit.
Master options Cluster/Client descrepencies.
Hi All, I have written a spark program on my dev box , IDE:Intellij scala version:2.11.7 spark verison:1.6.1 run fine from IDE, by providing proper input and output paths including master. But when i try to deploy the code in my cluster made of below, Spark version:1.6.1 built from source pkg using scala 2.11 But when i try spark-shell on cluster i get scala version to be 2.10.5 hadoop yarn cluster 2.6.0 and with additional options, --executor-memory --total-executor-cores --deploy-mode cluster/client --master yarn i get Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at com.movoto.SparkPost$.main(SparkPost.scala:36) at com.movoto.SparkPost.main(SparkPost.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) i understand this to be a scala version issue, as i have faced this before. Is there something that i have change and try things to get the same program running on cluster. Regards, Satyajit.