[jira] [Updated] (SPARK-32969) Spark Submit process not exiting after session.stop()

2020-09-22 Thread El R (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-32969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

El R updated SPARK-32969:
-
Affects Version/s: (was: 3.0.1)

> Spark Submit process not exiting after session.stop()
> -
>
> Key: SPARK-32969
> URL: https://issues.apache.org/jira/browse/SPARK-32969
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, Spark Submit
>Affects Versions: 2.4.7
>Reporter: El R
>Priority: Critical
>
> Exactly 3 spark submit processes are hanging from the first 3 jobs that were 
> submitted to the standalone cluster using client mode. Example from the 
> client:
> {code:java}
> root 1517 0.3 4.7 8412728 1532876 ? Sl 18:49 0:38 
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
> /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
>  -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
> --conf spark.master=spark://3c520b0c6d6e:7077 --conf 
> spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
>  --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
> --conf spark.fileserver.port=46102 --conf 
> packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
> spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
> spark.replClassServer.port=46104 --conf 
> spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
> spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --conf 
> spark.blockManager.port=46105 --conf spark.dynamicAllocation.enabled=true 
> pyspark-shell 
> root 1746 0.4 3.5 8152640 1132420 ? Sl 18:59 0:36 
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
> /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
>  -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
> --conf spark.master=spark://3c520b0c6d6e:7077 --conf 
> spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
>  --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
> --conf spark.fileserver.port=46102 --conf 
> packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
> spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
> spark.replClassServer.port=46104 --conf 
> spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
> spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --conf 
> spark.blockManager.port=46105 --conf spark.dynamicAllocation.enabled=true 
> pyspark-shell 
> root 2239 65.3 7.8 9743456 2527236 ? Sl 19:10 91:30 
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
> /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
>  -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
> --conf spark.master=spark://3c520b0c6d6e:7077 --conf 
> spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
>  --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
> --conf spark.fileserver.port=46102 --conf 
> packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
> spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
> spark.replClassServer.port=46104 --conf 
> spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
> spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --

[jira] [Created] (SPARK-32969) Spark Submit process not exiting after session.stop()

2020-09-22 Thread El R (Jira) 
El R created SPARK-32969:


 Summary: Spark Submit process not exiting after session.stop()
 Key: SPARK-32969
 URL: https://issues.apache.org/jira/browse/SPARK-32969
 Project: Spark
  Issue Type: Bug
  Components: PySpark, Spark Submit
Affects Versions: 3.0.1, 2.4.7
Reporter: El R


Exactly 3 spark submit processes are hanging from the first 3 jobs that were 
submitted to the standalone cluster using client mode. Example from the client:
{code:java}
root 1517 0.3 4.7 8412728 1532876 ? Sl 18:49 0:38 
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
/usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
 -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
--conf spark.master=spark://3c520b0c6d6e:7077 --conf 
spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
 --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
--conf spark.fileserver.port=46102 --conf 
packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
spark.replClassServer.port=46104 --conf 
spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --conf 
spark.blockManager.port=46105 --conf spark.dynamicAllocation.enabled=true 
pyspark-shell 
root 1746 0.4 3.5 8152640 1132420 ? Sl 18:59 0:36 
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
/usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
 -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
--conf spark.master=spark://3c520b0c6d6e:7077 --conf 
spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
 --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
--conf spark.fileserver.port=46102 --conf 
packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
spark.replClassServer.port=46104 --conf 
spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --conf 
spark.blockManager.port=46105 --conf spark.dynamicAllocation.enabled=true 
pyspark-shell 
root 2239 65.3 7.8 9743456 2527236 ? Sl 19:10 91:30 
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp 
/usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-2.7.7/etc/hadoop/:/usr/local/hadoop-2.7.7/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/common/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/yarn/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.7/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
 -Xmx2g org.apache.spark.deploy.SparkSubmit --conf spark.driver.port=46101 
--conf spark.master=spark://3c520b0c6d6e:7077 --conf 
spark.scheduler.allocation.file=/home/jovyan/work/spark_scheduler_allocation.xml
 --conf spark.app.name=REDACTED --conf spark.driver.bindAddress=3c520b0c6d6e 
--conf spark.fileserver.port=46102 --conf 
packages=org.apache.kudu:kudu-spark2_2.11:1.12.0 --conf 
spark.broadcast.port=46103 --conf spark.driver.host=3c520b0c6d6e --conf 
spark.replClassServer.port=46104 --conf 
spark.executorEnv.AF_ALERTS_STREAM_KEY=ALERTS_STREAM_LIST --conf 
spark.scheduler.mode=FAIR --conf spark.shuffle.service.enabled=True --conf 
spark.blockManager.port=46105 --conf spark.dynamicAllocation.enabled=true 
pyspark-shell
 
{code}
The corresponding jobs are showing as 'completed' in spark UI and have closed 
their sessions & exited according to their logs. No worker resources are being 
consumed by these jobs anymore & subsequent jobs are able to receive