Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2141196482 There are many conflicts between my local code and Remote. This is my mistake. To make the PR record beautiful, I opened a new PR: https://github.com/apache/hudi/pull/11370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley closed pull request #11359: [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… URL: https://github.com/apache/hudi/pull/11359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2141151906 ## CI report: * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * c8b14bd35eb233306750d8b31780d3da8ba2547d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24157) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2141146409 ## CI report: * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24146) * 0bc90bdc0865275eb0e3650a5bc82c3b3d65d11f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24156) * c8b14bd35eb233306750d8b31780d3da8ba2547d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2141115466 ## CI report: * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24146) * 0bc90bdc0865275eb0e3650a5bc82c3b3d65d11f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24156) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2141109747 ## CI report: * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24146) * 0bc90bdc0865275eb0e3650a5bc82c3b3d65d11f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2140031009 ## CI report: * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24146) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2139791741 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24122) * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24146) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2139638319 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24122) * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN * 9ce101ca9d0c194af5b31b533c83fb21549ca8d3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley commented on code in PR #11359:
URL: https://github.com/apache/hudi/pull/11359#discussion_r1620794903
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##
@@ -370,7 +370,7 @@ public static ConflictResolutionStrategy
getConflictResolutionStrategy(Configura
* Returns whether to commit even when current batch has no data, for flink
defaults false
*/
public static boolean allowCommitOnEmptyBatch(Configuration conf) {
-return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), false);
+return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(),
HoodieWriteConfig.ALLOW_EMPTY_COMMIT.defaultValue());
Review Comment:
After correcting the default return value of
OptionsResolver#allowCommitOnEmptyBatch to "true",
StreamWriteOperatorCoordinator will submit an empty Commit Or DeltaCommit (when
Checkpoint is completed), so when the program queries the latest commit, the
commit is empty, so the returned result is also empty, and the unit test fails
in the end
Modification plan:
When creating a Hudi table, set hoodie.allow.empty.commit = false
Other solutions:
We can modify the default value of the "hoodie.allow.empty.commit", but I
personally think it is not good enough, the reason:
Modify "hoodie.allow.empty.commit" default value to "false", but the default
value of the official document and code is "true", and it is very important to
submit an empty commit by default in Flink, which can track the entire life
cycle. Therefore, do not adopt this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley commented on code in PR #11359:
URL: https://github.com/apache/hudi/pull/11359#discussion_r1620734871
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##
@@ -370,7 +370,7 @@ public static ConflictResolutionStrategy
getConflictResolutionStrategy(Configura
* Returns whether to commit even when current batch has no data, for flink
defaults false
*/
public static boolean allowCommitOnEmptyBatch(Configuration conf) {
-return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), false);
+return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(),
HoodieWriteConfig.ALLOW_EMPTY_COMMIT.defaultValue());
Review Comment:
The original TestHoodieFlinkQuickstart can run successfully because:
After correcting the default return value of
OptionsResolver#allowCommitOnEmptyBatch to "true",
StreamWriteOperatorCoordinator will submit an empty Commit Or DeltaCommit (when
Checkpoint is completed), so when the program queries the latest commit, the
commit is empty, and the returned result is also empty, and the unit test fails.
Modification plan:
When creating a Hudi table, set hoodie.allow.empty.commit = false
Other solutions:
You can modify the default value, but I personally think it is not good
enough. The reason is:
Modify hoodie.allow.empty.commit to false, but the default value of the
official document and code is true, and it is very important to submit an empty
commit by default in Flink, which can track the entire life cycle. Therefore,
do not adopt this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2139622234 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24122) * 4b149d9085498be66c6426b0c3fde90ddf382cec UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley commented on code in PR #11359:
URL: https://github.com/apache/hudi/pull/11359#discussion_r1620734871
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##
@@ -370,7 +370,7 @@ public static ConflictResolutionStrategy
getConflictResolutionStrategy(Configura
* Returns whether to commit even when current batch has no data, for flink
defaults false
*/
public static boolean allowCommitOnEmptyBatch(Configuration conf) {
-return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), false);
+return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(),
HoodieWriteConfig.ALLOW_EMPTY_COMMIT.defaultValue());
Review Comment:
The original TestHoodieFlinkQuickstart can run successfully because:
After correcting the default return value of
OptionsResolver#allowCommitOnEmptyBatch to "true",
StreamWriteOperatorCoordinator will submit an empty Commit Or DeltaCommit (when
Checkpoint is completed), so when the program queries the latest commit, the
commit is empty, and the returned result is also empty, and the unit test fails.
Modification plan:
When creating a Hudi table, set hoodie.allow.empty.commit = false
Other solutions:
You can modify the default value, but I personally think it is not good
enough. The reason is:
Modify hoodie.allow.empty.commit to false, but the default value of the
official document and code is true, and it is very important to submit an empty
commit by default in Flink, which can track the entire life cycle. Therefore,
do not adopt this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley commented on code in PR #11359:
URL: https://github.com/apache/hudi/pull/11359#discussion_r161976
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##
@@ -370,7 +370,7 @@ public static ConflictResolutionStrategy
getConflictResolutionStrategy(Configura
* Returns whether to commit even when current batch has no data, for flink
defaults false
*/
public static boolean allowCommitOnEmptyBatch(Configuration conf) {
-return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), false);
+return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(),
HoodieWriteConfig.ALLOW_EMPTY_COMMIT.defaultValue());
Review Comment:
let me check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
danny0405 commented on code in PR #11359:
URL: https://github.com/apache/hudi/pull/11359#discussion_r1619518356
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##
@@ -370,7 +370,7 @@ public static ConflictResolutionStrategy
getConflictResolutionStrategy(Configura
* Returns whether to commit even when current batch has no data, for flink
defaults false
*/
public static boolean allowCommitOnEmptyBatch(Configuration conf) {
-return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), false);
+return conf.getBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(),
HoodieWriteConfig.ALLOW_EMPTY_COMMIT.defaultValue());
Review Comment:
Hmm, there are some flink test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2137740517 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24122) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2137504428 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24122) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
hudi-bot commented on PR #11359: URL: https://github.com/apache/hudi/pull/11359#issuecomment-2137484574 ## CI report: * c8bf966468abfcab8121f7ba7a63f8098bbf965a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7810] Fix OptionsResolver#allowCommitOnEmptyBatch default value… [hudi]
usberkeley opened a new pull request, #11359: URL: https://github.com/apache/hudi/pull/11359 ### Change Logs OptionsResolver#allowCommitOnEmptyBatch has a hardcoded default value of false, while ALLOW_EMPTY_COMMIT (hoodie.allow.empty.commit) defaults to true, this function returns the wrong default value ### Impact none ### Risk level (write none, low medium or high below) none ### Documentation Update none ### Contributor's checklist - [1] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [1] Change Logs and Impact were stated clearly - [1] Adequate tests were added if applicable - [1] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
