sivabalan narayanan created HUDI-9271:
-----------------------------------------
Summary: Enhance 8 to 7 Downgrade handler to avoid bringing entire
byte[] to driver memory
Key: HUDI-9271
URL: https://issues.apache.org/jira/browse/HUDI-9271
Project: Apache Hudi
Issue Type: Improvement
Components: writer-core
Reporter: sivabalan narayanan
[https://github.com/apache/hudi/blob/720e7eb9e7107202b67b261fc20772d3a6394b7d/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java#L222]
We are reading entire record from LSM timeline as GenericRecord and creating
equivalent archived metaEntry. But if we have a large commit metadata along w/
a large compaction plan, this might potentially cause OOM in driver.
Not sure if there is a easier way out since the record is represented as
GenericRecord. but filing a tracking ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
