I’d recommend using a scheduler of some kind to trigger your job each hour, and have the Spark job exit when it completes. Spark is not meant to run in any type of “sleep mode”, unless you want to run a structured streaming job and create a separate process to pull data from Casandra and publish it to your streaming endpoint. That decision really depends more on your use case.
On Tue, Jan 22, 2019 at 11:56 PM Soheil Pourbafrani <soheil.i...@gmail.com> wrote: > Hi, > > I want to submit a job in YARN cluster to read data from Cassandra and > write them in HDFS, every hour, for example. > > Is it possible to make Spark Application sleep in a while true loop and > awake every hour to process data? >
