RussellSpitzer commented on code in PR #6907:
URL: https://github.com/apache/iceberg/pull/6907#discussion_r1115000990
##########
api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java:
##########
@@ -76,7 +76,7 @@
*/
String MAX_CONCURRENT_FILE_GROUP_REWRITES =
"max-concurrent-file-group-rewrites";
- int MAX_CONCURRENT_FILE_GROUP_REWRITES_DEFAULT = 1;
+ int MAX_CONCURRENT_FILE_GROUP_REWRITES_DEFAULT = 5;
Review Comment:
I like 5 but what I'd really like us to do is check the size of our "file
groups" then use that to determine what to set max concurrent groups too. For
example if we see all file groups are less than say target file size, we set
Max Conncurrent = Num Cores
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
