Hi,
On Tue, Jan 22, 2019 at 11:30 AM Pola Yao wrote:
> "Thread-1" #19 prio=5 os_prio=0 tid=0x7f9b6828e800 nid=0x77cb waiting on
> condition [0x7f9a123e3000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for
Hi Marcelo,
I have dumped through jstack, and saw the ShutdownHookManager :
'''
"Thread-1" #19 prio=5 os_prio=0 tid=0x7f9b6828e800 nid=0x77cb waiting
on condition [0x7f9a123e3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking
Those are daemon threads and not the cause of the problem. The main
thread is waiting for the "org.apache.hadoop.util.ShutdownHookManager"
thread, but I don't see that one in your list.
On Wed, Jan 16, 2019 at 12:08 PM Pola Yao wrote:
>
> Hi Marcelo,
>
> Thanks for your response.
>
> I have
Hi Marcelo,
Thanks for your response.
I have dumped the threads on the server where I submitted the spark
application:
'''
...
"dispatcher-event-loop-2" #28 daemon prio=5 os_prio=0
tid=0x7f56cee0e000 nid=0x1cb6 waiting on condition [0x7f5699811000]
java.lang.Thread.State: WAITING
If System.exit() doesn't work, you may have a bigger problem
somewhere. Check your threads (using e.g. jstack) to see what's going
on.
On Wed, Jan 16, 2019 at 8:09 AM Pola Yao wrote:
>
> Hi Marcelo,
>
> Thanks for your reply! It made sense to me. However, I've tried many ways to
> exit the
Hi Marcelo,
Thanks for your reply! It made sense to me. However, I've tried many ways
to exit the spark (e.g., System.exit()), but failed. Is there an explicit
way to shutdown all the alive threads in the spark application and then
quit afterwards?
On Tue, Jan 15, 2019 at 2:38 PM Marcelo Vanzin
You should check the active threads in your app. Since your pool uses
non-daemon threads, that will prevent the app from exiting.
spark.stop() should have stopped the Spark jobs in other threads, at
least. But if something is blocking one of those threads, or if
something is creating a non-daemon
I submitted a Spark job through ./spark-submit command, the code was
executed successfully, however, the application got stuck when trying to
quit spark.
My code snippet:
'''
{
val spark = SparkSession.builder.master(...).getOrCreate
val pool = Executors.newFixedThreadPool(3)
implicit val xc =
