Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22980 )
Change subject: IMPALA-14075: Parallelize ExpireSnapshots.executeDeleteWith() ...................................................................... Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/22980/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22980/2//COMMIT_MSG@14 PS2, Line 14: org.apache.iceberg.util.ThreadPools.getWorkerPool() The class ThreadPools is in the "core" package of Iceberg which means we should use it cautiously, as even minor Iceberg versions can introduce breaking changes to these classes/methods. The Iceberg project follows semantic versioning rules only for the iceberg-api package: "From 1.0.0 forward, the project will follow semver in the public API module, iceberg-api." https://iceberg.apache.org/releases/#110-release That said, the functionality we rely on is quite simple, so it should be easy to rewrite if Iceberg breaks it. I have more concerns about the usage of getWorkerPool(), which is also used during table loading: "The size of the worker pool limits the number of tasks concurrently reading manifests in the base table implementation across all concurrent planning operations." This means a long-running EXPIRE SNAPSHOTS can make Iceberg table loadings slower which can also cause higher lags in the event processor. I'd suggest using a new ThreadPool dedicated for the EXPIRE SNAPSHOTS statement, or one that is shared between all EXPIRE SNAPSHOTS statement. -- To view, visit http://gerrit.cloudera.org:8080/22980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dcbf1e406e1732ef8829eb0cd627d932291d485 Gerrit-Change-Number: 22980 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 05 Jun 2025 14:22:55 +0000 Gerrit-HasComments: Yes
