Dongjoon Hyun created SPARK-29735:
-------------------------------------
Summary: CSVDataSource leaks file system
Key: SPARK-29735
URL: https://issues.apache.org/jira/browse/SPARK-29735
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Dongjoon Hyun
-
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113178/consoleFull
{code}
[info] FileBasedDataSourceSuite:
[info] - Writing empty datasets should not fail - orc (309 milliseconds)
[info] - Writing empty datasets should not fail - parquet (367 milliseconds)
[info] - Writing empty datasets should not fail - csv (171 milliseconds)
[info] - Writing empty datasets should not fail - json (130 milliseconds)
[info] - Writing empty datasets should not fail - text (423 milliseconds)
[info] - SPARK-23072 Write and read back unicode column names - orc (274
milliseconds)
[info] - SPARK-23072 Write and read back unicode column names - parquet (318
milliseconds)
[info] - SPARK-23072 Write and read back unicode column names - csv (358
milliseconds)
[info] - SPARK-23072 Write and read back unicode column names - json (290
milliseconds)
[info] - SPARK-15474 Write and read back non-empty schema with empty dataframe
- orc (327 milliseconds)
[info] - SPARK-15474 Write and read back non-empty schema with empty dataframe
- parquet (334 milliseconds)
[info] - SPARK-23271 empty RDD when saved should write a metadata only file -
orc (273 milliseconds)
[info] - SPARK-23271 empty RDD when saved should write a metadata only file -
parquet (352 milliseconds)
[info] - SPARK-23372 error while writing empty schema files using orc (29
milliseconds)
[info] - SPARK-23372 error while writing empty schema files using parquet (15
milliseconds)
[info] - SPARK-23372 error while writing empty schema files using csv (12
milliseconds)
[info] - SPARK-23372 error while writing empty schema files using json (10
milliseconds)
[info] - SPARK-23372 error while writing empty schema files using text (11
milliseconds)
[info] - SPARK-22146 read files containing special characters using orc (256
milliseconds)
[info] - SPARK-22146 read files containing special characters using parquet
(380 milliseconds)
[info] - SPARK-22146 read files containing special characters using csv (428
milliseconds)
[info] - SPARK-22146 read files containing special characters using json (284
milliseconds)
[info] - SPARK-22146 read files containing special characters using text (254
milliseconds)
[info] - SPARK-23148 read files containing special characters using json with
multiline enabled (557 milliseconds)
[info] - SPARK-23148 read files containing special characters using csv with
multiline enabled (424 milliseconds)
[info] - Enabling/disabling ignoreMissingFiles using orc (1 second, 605
milliseconds)
[info] - Enabling/disabling ignoreMissingFiles using parquet (1 second, 895
milliseconds)
09:26:51.342 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in
stage 94.0 (TID 125, amp-jenkins-worker-04.amp, executor driver): TaskKilled
(Stage cancelled)
[info] - Enabling/disabling ignoreMissingFiles using csv (1 second, 672
milliseconds)
09:26:51.344 WARN org.apache.spark.DebugFilesystem: Leaked filesystem
connection created at:
java.lang.Throwable
at
org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:35)
at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:69)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
at
org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.<init>(HadoopFileLinesReader.scala:65)
at
org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.readFile(CSVDataSource.scala:99)
at
org.apache.spark.sql.execution.datasources.v2.csv.CSVPartitionReaderFactory.buildReader(CSVPartitionReaderFactory.scala:68)
at
org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory.$anonfun$createReader$1(FilePartitionReaderFactory.scala:29)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at
org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.getNextReader(FilePartitionReader.scala:109)
at
org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.next(FilePartitionReader.scala:42)
at
org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.next(FilePartitionReader.scala:95)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:62)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1832)
at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1227)
at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1227)
at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2135)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:455)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:458)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
