Hi, I submit an app on Spark2 cluster using standalone scheduler on client mode. The app saves the results of the processing on the HDFS. There is no error on output logs and the app finished successfully. But the problem is it just create _SUCSSES and empty part-00000 file on the saving directory! I checked the app logic on local mode and it works correctly.
Here are the related parts of the logs: 18/12/17 20:57:10 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 78.0 (TID 350) in 13 ms on localhost (executor driver) (8/9) 18/12/17 20:57:10 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 9 blocks 18/12/17 20:57:10 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 18/12/17 20:57:10 INFO executor.Executor: Finished task 8.0 in stage 78.0 (TID 357). 1779 bytes result sent to driver 18/12/17 20:57:10 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 78.0 (TID 357) in 6 ms on localhost (executor driver) (9/9) 18/12/17 20:57:10 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 78.0, whose tasks have all completed, from pool 18/12/17 20:57:10 INFO scheduler.DAGScheduler: ShuffleMapStage 78 (sortBy at SparkSum.scala:260) finished in 0.016 s 18/12/17 20:57:10 INFO scheduler.DAGScheduler: looking for newly runnable stages 18/12/17 20:57:10 INFO scheduler.DAGScheduler: running: Set() 18/12/17 20:57:10 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 79) 18/12/17 20:57:10 INFO scheduler.DAGScheduler: failed: Set() 18/12/17 20:57:10 INFO scheduler.DAGScheduler: Submitting ResultStage 79 (MapPartitionsRDD[240] at saveAsTextFile at SparkSum.scala:261), which has no missing parents 18/12/17 20:57:10 INFO memory.MemoryStore: Block broadcast_46 stored as values in memory (estimated size 73.0 KB, free 1987.9 MB) 18/12/17 20:57:10 INFO memory.MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 26.6 KB, free 1987.9 MB) 18/12/17 20:57:10 INFO storage.BlockManagerInfo: Added broadcast_46_piece0 in memory on 172.16.20.4:40007 (size: 26.6 KB, free: 2004.4 MB) 18/12/17 20:57:10 INFO spark.SparkContext: Created broadcast 46 from broadcast at DAGScheduler.scala:996 18/12/17 20:57:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 79 (MapPartitionsRDD[240] at saveAsTextFile at SparkSum.scala:261) 18/12/17 20:57:10 INFO scheduler.TaskSchedulerImpl: Adding task set 79.0 with 1 tasks 18/12/17 20:57:10 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 79.0 (TID 358, localhost, executor driver, partition 0, PROCESS_LOCAL, 5812 bytes) 18/12/17 20:57:10 INFO executor.Executor: Running task 0.0 in stage 79.0 (TID 358) 18/12/17 20:57:10 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 9 blocks 18/12/17 20:57:10 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 18/12/17 20:57:10 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 18/12/17 20:57:10 INFO output.FileOutputCommitter: Saved output of task 'attempt_20181217205709_0079_m_000000_358' to hdfs://master:9000/home/hduser/soheil/res/_temporary/0/task_20181217205709_0079_m_000000 18/12/17 20:57:10 INFO mapred.SparkHadoopMapRedUtil: attempt_20181217205709_0079_m_000000_358: Committed 18/12/17 20:57:10 INFO executor.Executor: Finished task 0.0 in stage 79.0 (TID 358). 1722 bytes result sent to driver 18/12/17 20:57:10 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 79.0 (TID 358) in 75 ms on localhost (executor driver) (1/1) 18/12/17 20:57:10 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 79.0, whose tasks have all completed, from pool 18/12/17 20:57:10 INFO scheduler.DAGScheduler: ResultStage 79 (saveAsTextFile at SparkSum.scala:261) finished in 0.076 s 18/12/17 20:57:10 INFO scheduler.DAGScheduler: Job 5 finished: saveAsTextFile at SparkSum.scala:261, took 0.150206 s Stopping ... Elapsed Time: 218 Secs 18/12/17 20:57:10 INFO spark.SparkContext: Invoking stop() from shutdown hook 18/12/17 20:57:10 INFO server.ServerConnector: Stopped ServerConnector@5cc5b667{HTTP/1.1}{0.0.0.0:4040} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5ae76500{/stages/stage/kill,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2fd1731c{/jobs/job/kill,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5ae81e1{/api,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@59fc684e{/,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@46c670a6{/static,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@782a4fff {/executors/threadDump/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@30ed9c6c {/executors/threadDump,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@53499d85{/executors/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@64712be{/executors,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@759fad4{/environment/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6ea1bcdc{/environment,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@26be6ca7{/storage/rdd/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32057e6{/storage/rdd,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@62417a16{/storage/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1b0a7baf{/storage,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@70fab835{/stages/pool/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@22c86919{/stages/pool,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72b16078{/stages/stage/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@21694e53{/stages/stage,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7caa550{/stages/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3b9d6699{/stages,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@47428937{/jobs/job/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2c715e84{/jobs/job,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@d78795{/jobs/json,null,UNAVAILABLE} 18/12/17 20:57:10 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@46074492{/jobs,null,UNAVAILABLE} 18/12/17 20:57:10 INFO ui.SparkUI: Stopped Spark web UI at http://master:4040 18/12/17 20:57:10 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/12/17 20:57:10 INFO memory.MemoryStore: MemoryStore cleared 18/12/17 20:57:10 INFO storage.BlockManager: BlockManager stopped 18/12/17 20:57:10 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/12/17 20:57:10 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/12/17 20:57:10 INFO spark.SparkContext: Successfully stopped SparkContext 18/12/17 20:57:10 INFO util.ShutdownHookManager: Shutdown hook called 18/12/17 20:57:10 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-811c8c87-54f7-45ea-8953-9c29f6438c5e and here is the code of the writing part: for (_ <- 1 to 30) { val contribs = links.join(ranks) .values .flatMap { case (ids, rank) => val size = ids.size ids.map(id => (id, rank / size)) } ranks = contribs.reduceByKey(_ + _).mapValues( 0.15 + 0.85 * _) } ranks .mapValues(truncateAt(_, 5)) .sortBy(row => row._2) .saveAsTextFile(hdfs://master:9000/res) } What can be the problem?