[GitHub] [hudi] SteNicholas commented on a diff in pull request #9211: [HUDI-6540] Support failed writes clean policy for Flink

2023-07-30 Thread via GitHub 


SteNicholas commented on code in PR #9211:
URL: https://github.com/apache/hudi/pull/9211#discussion_r1278770434


##
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java:
##
@@ -114,10 +116,28 @@ public void testCheckpointFails() throws Exception {
   }
 
   @Test
-  public void testSubtaskFails() throws Exception {
+  public void testSubtaskFailsWithEagerFailedWritesCleanPolicy() throws 
Exception {
+testSubtaskFails()
+// the last checkpoint instant was rolled back by subTaskFails(0, 2)
+// with EAGER cleaning strategy
+.assertNoEvent()

Review Comment:
   @danny0405, I have added a simple test for failed writes rollback. PTAL. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] SteNicholas commented on a diff in pull request #9211: [HUDI-6540] Support failed writes clean policy for Flink

2023-07-26 Thread via GitHub 


SteNicholas commented on code in PR #9211:
URL: https://github.com/apache/hudi/pull/9211#discussion_r1274428725


##
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java:
##
@@ -114,10 +116,28 @@ public void testCheckpointFails() throws Exception {
   }
 
   @Test
-  public void testSubtaskFails() throws Exception {
+  public void testSubtaskFailsWithEagerFailedWritesCleanPolicy() throws 
Exception {
+testSubtaskFails()
+// the last checkpoint instant was rolled back by subTaskFails(0, 2)
+// with EAGER cleaning strategy
+.assertNoEvent()

Review Comment:
   @danny0405, I only modified the name of test method and extracted part into 
a common method, not modify the existing one. PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] SteNicholas commented on a diff in pull request #9211: [HUDI-6540] Support failed writes clean policy for Flink

2023-07-25 Thread via GitHub 


SteNicholas commented on code in PR #9211:
URL: https://github.com/apache/hudi/pull/9211#discussion_r1273233193


##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -95,6 +95,9 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context 
context) {
 DataStream pipeline = Pipelines.append(conf, rowType, 
dataStream, context.isBounded());
 if (OptionsResolver.needsAsyncClustering(conf)) {
   return Pipelines.cluster(conf, rowType, pipeline);
+} else if (OptionsResolver.isLazyFailedWritesCleanPolicy(conf)) {

Review Comment:
   @stream2000, thanks for the reminder. I have modified `HoodieFlinkStreamer`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org