Nick Pritchard created SPARK-20848:
--------------------------------------
Summary: Dangling threads when reading parquet files in local mode
Key: SPARK-20848
URL: https://issues.apache.org/jira/browse/SPARK-20848
Project: Spark
Issue Type: Bug
Components: Input/Output, SQL
Affects Versions: 2.1.1, 2.2.0
Reporter: Nick Pritchard
On each call to {{spark.read.parquet}}, a new ForkJoinPool is created. One of
the threads in the pool is kept in the {{WAITING}} state, and never stopped,
which leads to unbounded growth in number of threads.
This behavior is a regression from v2.1.0.
Reproducible example:
{code}
val spark = SparkSession
.builder()
.appName("test")
.master("local")
.getOrCreate()
while(true) {
spark.read.parquet("/path/to/file")
Thread.sleep(5000)
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
