hi guys: I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log. Any idea is welcome. [hadoop@master ~]$ crontab -e1.22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log and in run4.sh,it wrote:$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 2. 22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 3.22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log 4.22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log --------------------------------
Thanks&Best regards! San.Luo
