fallintoplace opened a new pull request, #2005: URL: https://github.com/apache/iceberg-go/pull/2005
## What changed - **Filters inherited manifests concurrently** during overwrite and delete commits. - **Preserves manifest order** by collecting each worker result at its original index. - **Uses the existing concurrency setting** for the filtering and rewrite stage. - **Cleans up replacement manifests** if a worker fails or the operation is cancelled. - Adds a focused benchmark for remote-like metadata latency. ## Why Overwrite commits used to process inherited manifests one at a time. On object storage, each manifest read and replacement write adds latency. ## Benchmark Command: `go test ./table -run '^$' -bench 'BenchmarkOverwriteExistingManifests' -benchtime=5x -count=3`\n\nMachine: Apple M1 Pro, darwin/arm64\n\nWorkload: 32 touched manifests, with a 1 ms delay on each manifest open/create.\n\n| Workers | Time |\n| --- | --- |\n| 1 | 105.8 to 109.6 ms/op |\n| 4 | 31.8 to 35.8 ms/op |\n| 8 | 20.8 to 21.6 ms/op |\n\n## Tests\n\n- `go test ./table -count=1`\n- `go test -race ./table -count=1`\n- `go vet ./table` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
