fallintoplace opened a new pull request, #1770: URL: https://github.com/apache/iceberg-go/pull/1770
## What - update partition field stats as manifest entries are written - stop retaining every entry's partition map until finalization - keep summary errors reported from `ToManifestFile`, like before - add coverage for null, NaN, binary bounds, and failed entry encoding - add writer and finalization benchmarks ## Why `ManifestWriter` currently keeps one partition map per entry. `ToManifestFile` then walks all of them again to build partition summaries. This makes retained state and finalization work grow with the number of manifest entries. The Java writer already keeps one `PartitionSummary` accumulator and updates it for each entry. The summary values are unchanged. They still follow the spec rules for nulls, NaNs, and lower/upper bounds. ## Benchmark ```text go test . -run '^$' -bench '^BenchmarkManifestWriterPartitionSummaryFinalization$' -benchmem -benchtime=1s -count=5 -cpu=1 ``` Median results on an Apple M1 Pro: | Entries | main | this PR | main B/op | this PR B/op | main allocs/op | this PR allocs/op | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 100 | 31.9 us | 0.92 us | 7,152 | 568 | 627 | 19 | | 1,000 | 265.6 us | 0.71 us | 57,556 | 572 | 6,028 | 20 | | 10,000 | 3.31 ms | 0.66 us | 561,556 | 572 | 60,028 | 20 | This benchmark isolates `ToManifestFile` after entries have already been written. End-to-end manifest writing is still mostly Avro encoding. At 10,000 entries, the end-to-end benchmark also reduced allocations from about 23.02 MB/op to 22.71 MB/op. ## Testing - `go test ./...` - `go test -race . -run '^(TestManifestWriterUpdatesPartitionSummariesIncrementally|TestConstructPartitionSummariesWithDroppedSource|TestManifests)$' -count=1` - `go vet ./...` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
