Sorry, I have to re-send it again as I did not get the answer. Here's the problem I'm facing, I have a standalone java application which is periodically submit Spark jobs to my yarn cluster, btw I'm not using 'spark-submit' or 'org.apache.spark.launcher' to submit my jobs. These jobs are successful and I can see them on Yarn RM webUI, but when I want to follow the link to the app history on Spark historyserver, I always got 404(application is not found) from Spark historyserver.
My code looks likes as below SparkConf conf = new SparkConf().setAppName("testSpak").setMaster("yarn-client") .setJars(new String[]{IOUtil.getJar(MySparkApp.class)}); conf.set("spark.yarn.historyServer.address", "10.247.44.155:18080"); conf.set("spark.history.fs.logDirectory", " hdfs://myHdfsNameNode:55310/scratch/tie/spark/applicationHistory"); JavaSparkContext sc = new JavaSparkContext(conf); try { ... my application code }finally{ sc.stop(); } Anything I did wrong or missed? Do I need to configure something on Yarn side? Thanks -- --Anfernee