[ https://issues.apache.org/jira/browse/HUDI-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan reopened HUDI-3460: --------------------------------------- > Flink TM Memory Optimization > ---------------------------- > > Key: HUDI-3460 > URL: https://issues.apache.org/jira/browse/HUDI-3460 > Project: Apache Hudi > Issue Type: Improvement > Components: flink > Reporter: Bo Cui > Assignee: Bo Cui > Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > > HoodieTableSource uses the MapReduce Memory Config, > [https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81] > but the config value may be the `parseMaximumHeapSizeMB* 0.8`, excessive > memory usage affects flink TM stability > ([https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966]) > so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink > rocksdb default is 64MB). > > and write.merge.max_memory,compaction.max_memory should also be changed to > 64MB -- This message was sent by Atlassian Jira (v8.20.1#820001)