danny0405 commented on issue #19629: URL: https://github.com/apache/hudi/issues/19629#issuecomment-5289055894
Thanks for raising this. I agree that the configuration semantics are surprising: native Parquet logs are logically log files, but their rollover is currently controlled by the underlying Parquet writer's `canWrite()`, which uses `write.parquet.max.file.size`; `write.log.max.size` is therefore ineffective for native-log rollover. However, I would separate that configuration issue from the performance claim. A ~120–132 MiB Parquet file is not normally considered a small file in Hudi. In fact, 120 MiB is Hudi's default Parquet target and is intended to be a generally read/write-friendly size. The observed ~132 MiB also follows from the 120 MiB target plus the writer's default 10% compression-sizing allowance. The comparison with a ~245 MiB inline log is not completely apples-to-apples. The inline file is an appendable Hudi log container bounded by the Flink mini-batch, while the native file is compressed, columnar, and write-once. Dividing 245 by 132 does not by itself prove that the same workload creates 1.85x as many files; we should compare actual file counts and records per file, together with write throughput, file-open/list operations, read latency, and compaction time. I would also avoid automatically applying the existing 1 GiB `write.log.max.size` default to native Parquet logs. With the current sizing allowance, that could change the default physical size from ~132 MiB to roughly 1.1 GiB, which may reduce metadata operations but can also reduce read parallelism and increase retry/recovery granularity. So I think the independent-sizing requirement is valid, but a dedicated option such as `write.native.log.max.file.size`, defaulting/falling back to the current Parquet target for compatibility, would be safer than directly reusing `write.log.max.size`. The performance motivation should ideally be supported with workload measurements rather than treating ~120 MiB Parquet files as inherently unfriendly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
