Hi, I have the following problem:
After SparkSession is initialized i create a task: val task = new Runnable { } where i make a REST API, and from it's response i read some data from internet/ES/Hive. This task is running to every 5 second with Akka scheduler: scheduler.schedule( Duration(0, TimeUnit.SECONDS), Duration(5, TimeUnit.SECONDS), task) My problem is when i run the code with yarn-cluster, the SparkSession it's immediately closed after init, and the SparkSession don't wait after the tasks that will be scheduled later. But in yarn-client everything it's ok. SparkSession it's not closed automatically, and the tasks is running to every 5 seconds. What it's the problem when i start Spark Application with yarn-cluster mode? How work submitting when i use yarn-cluster? How to resolve this problem? How to make this to work in yarn-cluster? ps. I use Spark 2.1.1, and akka-actor_2.11 - 2.4.17 Thanks, Cosmin