Did you by any chances left a sparkSession.setMaster("local") lurking in
your code?
Last time i checked, to run on yarn you have to package a 'fat jar'. could
you make sure the spark depedencies in your jar matches the version you are
running on Yarn?
alternatively please share code including how you submit your application
to spark
FYI this is the command i am using to submit a program to spark
spark-submit --master yarn --deploy-mode cluster --class
hth
On Tue, May 8, 2018 at 10:14 AM, SparkUser6
wrote:
> I have a simple program that works fine in the local mode. But I am having
> issues when I try to run the program in yarn-cluster mode. I know usually
> no such method happens when compile and run version mismatch but I made
> sure
> I took the same version.
>
> 205 [main] INFO org.spark_project.jetty.server.ServerConnector -
> Started
> Spark@29539e36{HTTP/1.1}{0.0.0.0:4040}
> 205 [main] INFO org.spark_project.jetty.server.Server - Started @3265ms
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.internal.config.package$.APP_CALLER_
> CONTEXT()Lorg/apache/spark/internal/config/OptionalConfigEntry;
> at org.apache.spark.deploy.yarn.Client.submitApplication(
> Client.scala:163)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(
> YarnClientSchedulerBackend.scala:56)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(
> TaskSchedulerImpl.scala:156)
> at org.apache.spark.SparkContext.(SparkContext.scala:509)
> at
> org.apache.spark.api.java.JavaSparkContext.(
> JavaSparkContext.scala:58)
> at
> com.voicebase.etl.PhoenixToElasticSearch.main(PhoenixToElasticSearch.java:
> 54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
