cshuo opened a new issue, #19629: URL: https://github.com/apache/hudi/issues/19629
### Task Description ### Describe the problem After switching the Flink MOR writer from inline log files to native Parquet log files, we observed a significant reduction in the size of each log file: - Inline log file: approximately 245 MB - Native Parquet log file: approximately 132 MB The relevant default configurations are: ```text write.batch.size = 256 MB write.log.max.size = 1024 MB write.parquet.max.file.size = 120 MB ``` Inline log files are mainly bounded by the Flink mini-batch size in this workload, so each file is approximately 245 MB after serialization. Native Parquet log files, however, are rolled according to the underlying Parquet writer's canWrite() result. The writer uses write.parquet.max.file.size instead of write.log.max.size. With the default 120 MB target and the additional compression-ratio estimation, the resulting files are approximately 132 MB. As a result, write.log.max.size does not effectively control the size of native Parquet log files, even though these files are logically Hudi log files. ### Why this is a problem This behavior produces significantly more and smaller log files after enabling native logs. For the same amount of incoming data, reducing the average log file size from approximately 245 MB to 132 MB can nearly double the number of log files. A larger number of native log files may have several negative effects: 1. Lower write efficiency Native log files are write-once files. Rolling a file requires closing the current Parquet writer, writing its footer, creating a new file, and initializing another writer. More frequent rolling therefore introduces additional file creation, close, metadata, and filesystem operations. 2. More filesystem metadata operations A higher file count increases NameNode or object-store metadata pressure, including file creation, listing, status checks, and cleanup. 3. Higher commit metadata overhead More physical log files need to be tracked in write statistics, commit metadata, markers, rollback metadata, and cleaning operations. 4. Potential read and compaction overhead Readers and compaction tasks need to open and process more Parquet files for the same file slice. Although native Parquet logs improve scan efficiency, an unnecessarily high file count may offset part of that benefit. 5. Unexpected configuration semantics Users reasonably expect write.log.max.size to control both inline and native log file sizes. Using write.parquet.max.file.size for native logs is surprising and couples the native-log file size with the base Parquet file size. Increasing write.parquet.max.file.size is not an ideal workaround because it also changes the target size of base Parquet files. ### Expected behavior Native log files should have a size policy independent of base files. Possible solutions include: Honor write.log.max.size when creating native log file writers; The native Parquet writer should use this value as its maximum/target file size while retaining the existing Parquet row-group and page-size settings. This would allow users to control native log file granularity without changing the size of base Parquet files. ### Task Type Performance optimization ### Related Issues **Parent feature issue:** (if applicable ) **Related issues:** NOTE: Use `Relationships` button to add parent/blocking issues after issue is created. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
