RussellSpitzer commented on code in PR #6378:
URL: https://github.com/apache/iceberg/pull/6378#discussion_r1048728860
##########
core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java:
##########
@@ -225,25 +225,40 @@ public void close() {
LOG.info("Closing commit service for {}", table);
committerService.shutdown();
+ boolean timeout = false;
+ int waitTime;
try {
// All rewrites have completed and all new files have been created, we
are now waiting for
// the commit
- // pool to finish doing it's commits to Iceberg State. In the case of
partial progress this
+ // pool to finish doing its commits to Iceberg State. In the case of
partial progress this
// should
// have been occurring simultaneously with rewrites, if not there
should be only a single
// commit operation.
- // In either case this should take much less than 10 minutes to
actually complete.
- if (!committerService.awaitTermination(10, TimeUnit.MINUTES)) {
+ // We will wait 10 minutes plus 5 more minutes for each commit left to
perform due to the
+ // time required for writing manifests
+ waitTime = 10 + (completedRewrites.size() / rewritesPerCommit) * 5;
Review Comment:
I was just thinking that 5 minutes is plenty of time to write a manifest
file. In our use case the biggest problem is the amount of time it takes to
write the manifests. Rewriting the Json portion is fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
