[
https://issues.apache.org/jira/browse/SPARK-23399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368121#comment-16368121
]
Dongjoon Hyun edited comment on SPARK-23399 at 2/17/18 7:29 AM:
----------------------------------------------------------------
[~mgaido]. I understand what is your intention, but please see the JIRA issue
title.
It's not about `Fix OrcQuerySuite`. Why do you reopen this issue? Please
proceed to file a new JIRA issue for that.
This JIRA issue handles the designed scope as described in the manual test case
in the PR.
For the reported case, I'll investigate more.
was (Author: dongjoon):
[~mgaido]. I understand what is your intention, but please see the JIRA issue
title.
It's not about `Fix OrcQuerySuite`. Why do you reopen this issue? Please
proceed to file a new JIRA issue for that.
> Register a task completion listener first for OrcColumnarBatchReader
> --------------------------------------------------------------------
>
> Key: SPARK-23399
> URL: https://issues.apache.org/jira/browse/SPARK-23399
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Dongjoon Hyun
> Assignee: Dongjoon Hyun
> Priority: Major
> Fix For: 2.3.1
>
>
> This is related with SPARK-23390.
> Currently, there was a opened file leak for OrcColumnarBatchReader.
> {code}
> [info] - Enabling/disabling ignoreMissingFiles using orc (648 milliseconds)
> 15:55:58.673 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in
> stage 61.0 (TID 85, localhost, executor driver): TaskKilled (Stage cancelled)
> 15:55:58.674 WARN org.apache.spark.DebugFilesystem: Leaked filesystem
> connection created at:
> java.lang.Throwable
> at
> org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:36)
> at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:70)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> at
> org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.open(RecordReaderUtils.java:173)
> at
> org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:254)
> at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:633)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:138)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
