KranzL opened a new pull request, #1993: URL: https://github.com/apache/iceberg-go/pull/1993
Fixes #1982. The compaction read+write pipeline had fixed memory characteristics: the scanner decodes read.parquet.batch-size rows per Arrow batch (131072 by default) with no per-scan override, and each rolling data writer buffers up to 64 record batches in its input channel. On wide rows the product of those two numbers dominates a compaction worker's peak memory. - WithArrowBatchSize scan option caps the rows decoded per Arrow batch for one scan, overriding the table's read.parquet.batch-size without mutating caller-supplied scan options. - The rolling data writers' record channel capacity (default 64 batches) is now configurable via the WithRecordBatchBufferSize write option, threaded through recordWritingArgs to the writer factory. Applies to both the fanout and the clustered write paths, which share the rolling writer. - WithParquetRowGroupLimit write option overrides the table's write.parquet.row-group-limit per write, bounding rows buffered per row group before each flush. - Compaction group options forward all three into ExecuteCompactionGroup: WithCompactionReadBatchSize, WithCompactionRecordBatchBufferSize, and WithCompactionParquetRowGroupLimit. Defaults stay exactly as they are today. Tests: table/write_read_tuning_test.go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
