dramaticlly commented on PR #15426: URL: https://github.com/apache/iceberg/pull/15426#issuecomment-3948664727
> Could you elaborate more on the previous behavior, and how you are changing that in this PR? Happy to! Before this change when we scan the manifest entries using ManifestGroup, we will scan manifest sequentially using `CloseableIterable.concat` even when executorService is provided. After this change, we now can benefit from using threadpool to scan the manifests if ManifestGroup is created with executorService provided in planWith(). One caveat is that we also need to make defensive copy of entries in parallel scanning as entries are reused across all iterations within a manifest given the reuseContainers in ManifestReader. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
