Which Spark release are you using ? Can you pastebin the stack trace of the process running on your machine ?
Thanks > On Nov 20, 2015, at 6:46 PM, Vikram Kone <vikramk...@gmail.com> wrote: > > Hi, > I'm seeing a strange problem. I have a spark cluster in standalone mode. I > submit spark jobs from a remote node as follows from the terminal > > spark-submit --master spark://10.1.40.18:7077 --class com.test.Ping > spark-jobs.jar > > when the app is running , when I press ctrl-C on the console terminal, then > the process is killed and so is the app in the spark master UI. When I go to > spark master ui, i see that this app is in state Killed under Completed > applications, which is what I expected to see. > > Now, I created a shell script as follows to do the same > > #!/bin/bash > spark-submit --master spark://10.1.40.18:7077 --class com.test.Ping > spark-jobs.jar > echo $! > my.pid > > When I execute the shell script from terminal, as follows > > $> bash myscript.sh > > The application is submitted correctly to spark master and I can see it as > one of the running apps in teh spark master ui. But when I kill the process > in my terminal as follows > > $> ps kill $(cat my.pid) > > I see that the process is killed on my machine but the spark appliation is > still running in spark master! It doesn't get killed. > > I noticed one more thing that, when I launch the spark job via shell script > and kill the application from spark master UI by clicking on "kill" next to > the running application, it gets killed in spark ui but I still see the > process running in my machine. > > In both cases, I would expect the remote spark app to be killed and my local > process to be killed. > > Why is this happening? and how can I kill a spark app from the terminal > launced via shell script w.o going to the spark master UI? > > I want to launch the spark app via script and log the pid so i can monitor it > remotely > > thanks for the help >
