[
https://issues.apache.org/jira/browse/HUDI-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udit Mehrotra updated HUDI-1529:
--------------------------------
Description:
When testing a large dataset around 1.2TB data and around 20k files, we notice
an issue where the spark driver would always run out of memory, when running
queries with use of metadata table *enabled*. The OOM would happen on any
query, even if it was touching a single partition, and was happening in the
*split generation* phase before any tasks would start executing.
Upon analyzing the heap dump, it was analyzed that input format was generating
*millions of splits for every single file*. Upon further analysis of the code
path, it was found that the root cause was because *metadata enabled* code was
ignoring the *blockSize* when returning *FileStatus* objects and setting it to
*0*. Spark by itself does not set any value for the property:
{code:java}
mapreduce.input.fileinputformat.split.minsize
{code}
As a result *minSize* ends up being 1, and with block size as 0 it cause input
format to *generate splits of size 1 bytes***** because of the logic here:
[https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L417]
This ends up in exponential file split objects being creating, causing driver
to run out of memory.
> Spark-SQL drvier runs out of memory when metadata table is enabled
> ------------------------------------------------------------------
>
> Key: HUDI-1529
> URL: https://issues.apache.org/jira/browse/HUDI-1529
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Performance, Spark Integration
> Reporter: Udit Mehrotra
> Assignee: Udit Mehrotra
> Priority: Major
>
> When testing a large dataset around 1.2TB data and around 20k files, we
> notice an issue where the spark driver would always run out of memory, when
> running queries with use of metadata table *enabled*. The OOM would happen on
> any query, even if it was touching a single partition, and was happening in
> the *split generation* phase before any tasks would start executing.
> Upon analyzing the heap dump, it was analyzed that input format was
> generating *millions of splits for every single file*. Upon further analysis
> of the code path, it was found that the root cause was because *metadata
> enabled* code was ignoring the *blockSize* when returning *FileStatus*
> objects and setting it to *0*. Spark by itself does not set any value for the
> property:
> {code:java}
> mapreduce.input.fileinputformat.split.minsize
> {code}
> As a result *minSize* ends up being 1, and with block size as 0 it cause
> input format to *generate splits of size 1 bytes***** because of the logic
> here:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L417]
> This ends up in exponential file split objects being creating, causing driver
> to run out of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
