[
https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350242#comment-17350242
]
Yuming Wang commented on SPARK-35441:
-
Please increase your driver memory. or you should merge small
[
https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350221#comment-17350221
]
Zhang Jianguo commented on SPARK-35441:
---
[~hyukjin.kwon]
20,000,000 files
> InMemoryFileIndex
[
https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348100#comment-17348100
]
Hyukjin Kwon commented on SPARK-35441:
--
and how many files? It's unlikely that the list causes OOM.
[
https://issues.apache.org/jira/browse/SPARK-35441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347404#comment-17347404
]
Yuming Wang commented on SPARK-35441:
-
What is your driver memory?
> InMemoryFileIndex load all
