[
https://issues.apache.org/jira/browse/HUDI-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-9271:
--------------------------------------
Priority: Critical (was: Major)
> Enhance 8 to 7 Downgrade handler to avoid bringing entire byte[] to driver
> memory
> ---------------------------------------------------------------------------------
>
> Key: HUDI-9271
> URL: https://issues.apache.org/jira/browse/HUDI-9271
> Project: Apache Hudi
> Issue Type: Improvement
> Components: writer-core
> Reporter: sivabalan narayanan
> Priority: Critical
> Fix For: 1.1.0
>
>
> [https://github.com/apache/hudi/blob/720e7eb9e7107202b67b261fc20772d3a6394b7d/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java#L222]
>
>
> We are reading entire record from LSM timeline as GenericRecord and creating
> equivalent archived metaEntry. But if we have a large commit metadata along
> w/ a large compaction plan, this might potentially cause OOM in driver.
> Not sure if there is a easier way out since the record is represented as
> GenericRecord. but filing a tracking ticket.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
