i want to use yarn cluster with my current code. if i use conf.set("spark.master","local[*]") inplace of conf.set("spark.master","yarn"), everything is very well. but i try to use yarn in setmaster, my code give an below error.
``` package com.example.pocsparkspring; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.security.UserGroupInformation; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.sql.*; public class poc2 { public static void main(String[] args) { try { System.setProperty("hadoop.home.dir", "D:/hadoop"); String HADOOP_CONF_DIR = "D:/Dev/ws/poc-spark-spring/src/main/resources/"; Configuration configuration1 = new Configuration(); configuration1.addResource(new Path(HADOOP_CONF_DIR + "core-site.xml")); configuration1.addResource(new Path(HADOOP_CONF_DIR + "hdfs-site.xml")); configuration1.addResource(new Path(HADOOP_CONF_DIR + "yarn-site.xml")); configuration1.addResource(new Path(HADOOP_CONF_DIR + "mapred-site.xml")); UserGroupInformation.setConfiguration(configuration1); UserGroupInformation.loginUserFromKeytab("user", HADOOP_CONF_DIR + "mykeytab.keytab"); SparkConf conf = new SparkConf(); conf.set("spark.master","yarn"); conf.set("spark.kerberos.keytab", HADOOP_CONF_DIR + "mykeytab.keytab"); conf.set("spark.kerberos.principal", "user"); conf.set("hadoop.security.authentication", "kerberos"); conf.set("hadoop.security.authorization", "true"); conf.set("dfs.client.use.datanode.hostname", "true"); conf.set("spark.security.credentials.hbase.enabled", "false"); conf.set("spark.security.credentials.hive.enabled", "false"); conf.setAppName("poc"); conf.setJars(new String[]{"target/poc-spark-spring.jar"}); JavaSparkContext sparkContext = new JavaSparkContext(conf); SparkSession sparkSession = SparkSession.builder() .sparkContext(sparkContext.sc()) .getOrCreate(); sparkSession.sparkContext().hadoopConfiguration().addResource(new Path(HADOOP_CONF_DIR + "core-site.xml")); sparkSession.sparkContext().hadoopConfiguration().addResource(new Path(HADOOP_CONF_DIR + "hdfs-site.xml")); sparkSession.sparkContext().hadoopConfiguration().addResource(new Path(HADOOP_CONF_DIR + "yarn-site.xml")); sparkSession.sparkContext().hadoopConfiguration().addResource(new Path(HADOOP_CONF_DIR + "mapred-site.xml")); sparkSession.sparkContext().conf().set("spark.kerberos.keytab", HADOOP_CONF_DIR + "mykeytab.keytab"); sparkSession.sparkContext().conf().set("spark.kerberos.principal", "user"); sparkSession.sparkContext().conf().set("hadoop.security.authentication", "kerberos"); sparkSession.sparkContext().conf().set("hadoop.security.authorization", "true"); sparkSession.sparkContext().conf().set("spark.security.credentials.hbase.enabled", "false"); sparkSession.sparkContext().conf().set("spark.security.credentials.hive.enabled", "false"); Dataset<Row> oracleDF = sparkSession.read() .format("jdbc") .option("fetchsize", 10000) .option("url", "jdbc:oracle:thin:@exampledev :1521:exampledev") .option("query", "select * FROM exampletable") .option("driver", "oracle.jdbc.driver.OracleDriver") .option("user", "user") .option("password", "pass") .load(); oracleDF.show(false); oracleDF.write().mode(SaveMode.Append).format("csv").save("hdfs:///hive/stage_space/example"); } catch (Exception e) { e.printStackTrace(); } } } ``` as i mentioned my code give an error. firstly everything looks good. i control cluster gui, i can see in there my job. My job status changed "accepted" but after a while it gives below error. and i can see same error in gui log. ``` 09:38:45.129 [YARN application state monitor] ERROR org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details. 09:38:45.129 [YARN application state monitor] ERROR org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Diagnostics message: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102) at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110) at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:558) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:277) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:926) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:925) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:925) at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:957) at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) Caused by: java.io.IOException: Failed to connect to user.../ 172.21.242.26:59690 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: user.../172.21.242.26:59690 Caused by: java.net.ConnectException: Connection timed out at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) ``` actually i couldnt sure, can i use yarn with my code like this. Because i couldnt found any example like my code.