Hi All, Imagine I have a Production spark streaming kafka (direct connection) subscriber and publisher jobs running which publish and subscriber (receive) data from a kafka topic and I save one day's worth of data using dstream.slice to Cassandra daily table (so I create daily table before running spark streaming job).
My question if all the above code runs in some scheduler like autosys how should I say to spark publisher to stop publishing as it is End of day and to spark subscriber to stop receiving to stop receiving without killing the jobs ? if I kill my autosys scheduler turns red saying the job had failed etc ... Is there a way to stop both subscriber and publisher with out killing or terminating the code. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-turn-off-spark-streaming-gracefully-tp25734.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
