mxm commented on code in PR #15566:
URL: https://github.com/apache/iceberg/pull/15566#discussion_r2910298870
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:
##########
@@ -682,6 +698,14 @@ IcebergSink build() {
FlinkMaintenanceConfig flinkMaintenanceConfig =
new FlinkMaintenanceConfig(table, writeOptions, readableConfig);
+ // If the "compactMode" option is configured,
+ // add it as a separate maintenance task
+ if (flinkWriteConf.compactMode()) {
+ RewriteDataFilesConfig rewriteDataFilesConfig =
+ flinkMaintenanceConfig.createRewriteDataFilesConfig();
+
maintenanceTasks.add(RewriteDataFiles.builder().config(rewriteDataFilesConfig));
+ }
Review Comment:
Fair, point taken.
>Out of curiosity, is the scenario we’re trying to solve here to avoid
having to maintain a separate maintenance job per table Or what scenario is
this intended to address?
We're trying to make all table maintenance available post-commit. This is
way simpler than having multiple jobs and avoids conflicts with the writer.
I'm going to change the implementation:
- I'll add a similar configuration as for RewriteDataFiles ("compactMode")
- I'll remove the generic `maintenance(..)` builder method.
- Change the code to only allow one maintenance for each type. Builder will
have precedence over the property-based configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
