Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
danny0405 merged PR #10388: URL: https://github.com/apache/hudi/pull/10388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1437409285
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
@danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1435470539
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
COW + INSERT is right.
The scene we're talking about now is mor + insert.
```
public static boolean isAppendMode(Configuration conf) {
// 1. inline clustering is supported for COW table;
// 2. async clustering is supported for both COW and MOR table
return isInsertOperation(conf) && ((isCowTable(conf) &&
!conf.getBoolean(FlinkOptions.INSERT_CLUSTER)) || isMorTable(conf));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1435470539
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
COW + INSERT is right.
The scene we're talking about now is mor + insert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1435470539
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
OptionsResolver.isMorTable(conf) = true
conf.getBoolean(FlinkOptions.COMPACTION_SCHEDULE_ENABLED) = true
Why is false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1435470539
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
OptionsResolver.isMorTable(conf) = true
conf.getBoolean(FlinkOptions.COMPACTION_SCHEDULE_ENABLED) = ture
Why is false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
danny0405 commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1435444510
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
For append mode, we mean COW + INSERT, that means `needsScheduleCompaction`
returns false, and there will never schdule any compaction plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1434758685
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
> Did you check the compaction scheduling condition:
>
> ```java
> public static boolean needsScheduleCompaction(Configuration conf) {
> return OptionsResolver.isMorTable(conf)
> && conf.getBoolean(FlinkOptions.COMPACTION_SCHEDULE_ENABLED);
> }
> ```
Do you mean to add checks here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1434755755
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
Hi @danny0405 , if the table type is mor and the compaction.schedule.enabled
is true and the write.operation is append mode, the needsScheduleCompaction
returns true. But the compaction is useless
```
public static boolean isAppendMode(Configuration conf) {
// 1. inline clustering is supported for COW table;
// 2. async clustering is supported for both COW and MOR table
return isInsertOperation(conf) && ((isCowTable(conf) &&
!conf.getBoolean(FlinkOptions.INSERT_CLUSTER)) || isMorTable(conf));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hehuiyuan commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1434755755
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
Hi @danny0405 , if the table type is mor and the compaction.schedule.enabled
is true and the write.operation is append mode, the needsScheduleCompaction
returns true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hudi-bot commented on PR #10388: URL: https://github.com/apache/hudi/pull/10388#issuecomment-1867282274 ## CI report: * 7b77508cefed3712435111c638c3742d9055f9da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21666) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hudi-bot commented on PR #10388: URL: https://github.com/apache/hudi/pull/10388#issuecomment-1867150619 ## CI report: * 6e42a97610e13b490d2ebc9fba1e3d53528147b7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21643) * 7b77508cefed3712435111c638c3742d9055f9da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21666) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
danny0405 commented on code in PR #10388:
URL: https://github.com/apache/hudi/pull/10388#discussion_r1434655686
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##
@@ -96,6 +96,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
// Append mode
if (OptionsResolver.isAppendMode(conf)) {
+// close compaction for append mode
+conf.set(FlinkOptions.COMPACTION_SCHEDULE_ENABLED, false);
Review Comment:
Did you check the compaction scheduling condition:
```java
public static boolean needsScheduleCompaction(Configuration conf) {
return OptionsResolver.isMorTable(conf)
&& conf.getBoolean(FlinkOptions.COMPACTION_SCHEDULE_ENABLED);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7249][Close compaction when using append mode [hudi]
hudi-bot commented on PR #10388: URL: https://github.com/apache/hudi/pull/10388#issuecomment-1867146134 ## CI report: * 6e42a97610e13b490d2ebc9fba1e3d53528147b7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21643) * 7b77508cefed3712435111c638c3742d9055f9da UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
