[
https://issues.apache.org/jira/browse/SPARK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951223#comment-16951223
]
L. C. Hsieh commented on SPARK-26570:
-------------------------------------
First from the stacktrace, I was thinking it is possible that your
SerializableFileStatus is too many. Then it is possible to cause OOM when
transforming them back to Status. The PR was created for that.
But from your latest posts, from jmap logs, looks like SerializableFileStatus
do not hold too much memory. Are you sure SerializableFileStatus is the cause
of the OOM?
> Out of memory when InMemoryFileIndex bulkListLeafFiles
> ------------------------------------------------------
>
> Key: SPARK-26570
> URL: https://issues.apache.org/jira/browse/SPARK-26570
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.2
> Reporter: deshanxiao
> Priority: Major
> Attachments: image-2019-10-13-18-41-22-090.png,
> image-2019-10-13-18-45-33-770.png, image-2019-10-14-10-00-27-361.png,
> image-2019-10-14-10-32-17-949.png, image-2019-10-14-10-47-47-684.png,
> image-2019-10-14-10-50-47-567.png, image-2019-10-14-10-51-28-374.png,
> screenshot-1.png
>
>
> The *bulkListLeafFiles* will collect all filestatus in memory for every query
> which may cause the oom of driver. I use the spark 2.3.2 meeting with the
> problem. Maybe the latest one also exists the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
