Did you by any chances left a sparkSession.setMaster("local") lurking in your code?
Last time i checked, to run on yarn you have to package a 'fat jar'. could you make sure the spark depedencies in your jar matches the version you are running on Yarn? alternatively please share code including how you submit your application to spark FYI this is the command i am using to submit a program to spark spark-submit --master yarn --deploy-mode cluster --class <MyTaskClass> <myjar.jar> <input args> hth On Tue, May 8, 2018 at 10:14 AM, SparkUser6 <alchemistsrivast...@gmail.com> wrote: > I have a simple program that works fine in the local mode. But I am having > issues when I try to run the program in yarn-cluster mode. I know usually > no such method happens when compile and run version mismatch but I made > sure > I took the same version. > > 205 [main] INFO org.spark_project.jetty.server.ServerConnector - > Started > Spark@29539e36{HTTP/1.1}{0.0.0.0:4040} > 205 [main] INFO org.spark_project.jetty.server.Server - Started @3265ms > Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.spark.internal.config.package$.APP_CALLER_ > CONTEXT()Lorg/apache/spark/internal/config/OptionalConfigEntry; > at org.apache.spark.deploy.yarn.Client.submitApplication( > Client.scala:163) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start( > YarnClientSchedulerBackend.scala:56) > at > org.apache.spark.scheduler.TaskSchedulerImpl.start( > TaskSchedulerImpl.scala:156) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) > at > org.apache.spark.api.java.JavaSparkContext.<init>( > JavaSparkContext.scala:58) > at > com.voicebase.etl.PhoenixToElasticSearch.main(PhoenixToElasticSearch.java: > 54) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:743) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit( > SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >