[
https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884791#comment-16884791
]
Stavros Kontopoulos edited comment on SPARK-27927 at 7/15/19 12:35 AM:
-----------------------------------------------------------------------
I was able to reproduce it reasily with 2.4.3.
I used this tool [https://github.com/jglick/jkillthread] to kill eventloop and
then the other okhttp thread:
19/07/15 00:12:06 INFO StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
Killing "OkHttp [https://kubernetes.default.svc/]...";
Did not find "OkHttp [https://kubernetes.default.svc/]...";
Killing "dag-scheduler-event-loop"
Killing "OkHttp WebSocket [https://kubernetes.default.svc/]...";
Exception in thread "OkHttp WebSocket [https://kubernetes.default.svc/]...";
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(AbstractQueuedSynchronizer.java:1939)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1103)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Killing "OkHttp WebSocket [https://kubernetes.default.svc/]...";
Exception in thread "OkHttp WebSocket [https://kubernetes.default.svc/]...";
java.lang.IllegalMonitorStateException
Unfortunately I cant the kill the latter as another one is created. Anyway that
means that this is just another case of
https://issues.apache.org/jira/browse/SPARK-27812 .
spark.stop() obviously stops the k8s client and everything finishes as expected.
was (Author: skonto):
I was able to reproduce it reasily with 2.4.3.
I used this tool [https://github.com/jglick/jkillthread] to kill eventloop and
then the other okhttp thread:
19/07/15 00:12:06 INFO StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
Killing "OkHttp https://kubernetes.default.svc/...";
Did not find "OkHttp https://kubernetes.default.svc/...";
Killing "dag-scheduler-event-loop"
Killing "OkHttp WebSocket https://kubernetes.default.svc/...";
Exception in thread "OkHttp WebSocket https://kubernetes.default.svc/...";
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(AbstractQueuedSynchronizer.java:1939)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1103)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Killing "OkHttp WebSocket https://kubernetes.default.svc/...";
Exception in thread "OkHttp WebSocket https://kubernetes.default.svc/...";
java.lang.IllegalMonitorStateException
Unfortunately I cant the kill the latter as another one is created. Anyway that
means that this is just another case of
https://issues.apache.org/jira/browse/SPARK-27812
> driver pod hangs with pyspark 2.4.3 and master on kubenetes
> -----------------------------------------------------------
>
> Key: SPARK-27927
> URL: https://issues.apache.org/jira/browse/SPARK-27927
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, PySpark
> Affects Versions: 3.0.0, 2.4.3
> Environment: k8s 1.11.9
> spark 2.4.3 and master branch.
> Reporter: Edwin Biemond
> Priority: Major
> Attachments: driver_threads.log, executor_threads.log
>
>
> When we run a simple pyspark on spark 2.4.3 or 3.0.0 the driver pods hangs
> and never calls the shutdown hook.
> {code:java}
> #!/usr/bin/env python
> from __future__ import print_function
> import os
> import os.path
> import sys
> # Are we really in Spark?
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.appName('hello_world').getOrCreate()
> print('Our Spark version is {}'.format(spark.version))
> print('Spark context information: {} parallelism={} python version={}'.format(
> str(spark.sparkContext),
> spark.sparkContext.defaultParallelism,
> spark.sparkContext.pythonVer
> ))
> {code}
> When we run this on kubernetes the driver and executer are just hanging. We
> see the output of this python script.
> {noformat}
> bash-4.2# cat stdout.log
> Our Spark version is 2.4.3
> Spark context information: <SparkContext
> master=k8s://https://kubernetes.default.svc:443 appName=hello_world>
> parallelism=2 python version=3.6{noformat}
> What works
> * a simple python with a print works fine on 2.4.3 and 3.0.0
> * same setup on 2.4.0
> * 2.4.3 spark-submit with the above pyspark
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
