Hello,
I'm trying to kick off a spark streaming job to a stand alone master using
spark submit inside of init.d. This is what I have:
DAEMON="spark-submit --class Streamer --executor-memory 500M
--total-executor-cores 4 /path/to/assembly.jar"
start() {
$DAEMON -p /var/run/my_assembly.pid &
echo "OK" &&
return 0
}
However, will return 0 even if spark_submit fails. Is there a way to run
spark-submit in the background and return 0 only if it successfully starts up?
Or better yet, is there something in spark-submit that will allow me to do
this, perhaps via a command line argument?
Thanks,
Ashic.