GitHub user cshuo added a comment to the discussion: Use ZSTD as the default compression codec for Parquet files
> On scope: hoodie.parquet.compression.codec is a single knob shared by base > files and native Parquet log blocks (HoodieParquetDataBlock, > HoodieFlinkParquetDataBlock, RowDataInlineLogWriteHandle and > HoodieInlineLogAppendHandle all pass > writeConfig.getParquetCompressionCodec()). So this means either a new > log-file-specific config or flipping the shared default for base files too - > different changes, and the base-file case is the one > https://github.com/apache/hudi/issues/13568 and > https://github.com/apache/hudi/issues/14938 already argued with numbers. I would prefer introducing a separate compression configuration for native log files. Log files are short-lived intermediate files and will eventually be compacted into base files, so write throughput and CPU efficiency are more important than maximizing compression ratio. Even if both log files and base files ultimately use ZSTD, we may want to use a lower ZSTD level for log files to favor faster compression, while keeping a different level for long-lived base files. GitHub link: https://github.com/apache/hudi/discussions/19615#discussioncomment-17999989 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
