Hi I just observed a litlte weird behavior:
I ran a pyspark job, very simple one. conf = SparkConf() conf.setAppName("Historical Meter Load") conf.set("spark.yarn.queue","root.Applications") conf.set("spark.executor.instances","50") conf.set("spark.executor.memory","10g") conf.set("spark.yarn.executor.memoryOverhead","2048") conf.set("spark.sql.shuffle.partitions",1000) conf.set("spark.executor.cores","4") sc = SparkContext(conf = conf) sqlContext = HiveContext(sc) df = sqlContext.sql("some sql") c = df.count() df.filter(df["RNK"] == 1).saveAsTable("some table").mode("overwrite") sc.stop() running is on CDH 5.7 cluster, Spark 1.6.0. Behavior observed: After few hours of running (definitely over 12H, but not sure exacly when), Yarn reported job as Completed, finished successfully, whereas the job kept running (I can see from Application master link) for 22H. Timing of the job is expected. Behavior of YARN is not. Is it a known issue? Is it a pyspark specific issue or same with scala as well? -- Best Regards, Ayan Guha