[
https://issues.apache.org/jira/browse/HUDI-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuwei Xiao updated HUDI-4753:
-----------------------------
Description:
In current log writing, the avgRecordSize is taken from the first incoming log
record, which may not be accurate, especially in the metadata table case.
In metadata table writing, the first log record is always
`_{_}all_partition{_}_`, whose size may be much larger than a normal partition
record.
This issue will cause performance issue in log writing and compaction, as we
write too many log blocks and spill unnecessary records to disk.
was:
In current log writing, the avgRecordSize is taken from the first incoming log
record, which may not be accurate, especially in metadata table case.
In metadata table writing, the first log record is always `__all_partition__`,
which may be much larger than a normal partition record.
The issue will case performance issue in log writing and compaction, as we need
to write too many log blocks and spill unnecessary record to disk.
> More accurate evaluation of log record during log writing or compaction
> -----------------------------------------------------------------------
>
> Key: HUDI-4753
> URL: https://issues.apache.org/jira/browse/HUDI-4753
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Yuwei Xiao
> Priority: Major
>
> In current log writing, the avgRecordSize is taken from the first incoming
> log record, which may not be accurate, especially in the metadata table case.
>
> In metadata table writing, the first log record is always
> `_{_}all_partition{_}_`, whose size may be much larger than a normal
> partition record.
>
> This issue will cause performance issue in log writing and compaction, as we
> write too many log blocks and spill unnecessary records to disk.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
