Re: [PR] [HUDI-7438][Test][DNM] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10743: URL: https://github.com/apache/hudi/pull/10743#issuecomment-1962288861 ## CI report: * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22600) * 3d09199fa55f54af95d9d2535e851bb9d03c8815 UNKNOWN * e3182b43f7ae946a1f2a2da7cc8af2e8099bc178 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438][DNM] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962288844 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch HUDI-7438-fix-issue-comment-processing updated (e3182b43f7a -> 69086bc3a84)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch HUDI-7438-fix-issue-comment-processing in repository https://gitbox.apache.org/repos/asf/hudi.git discard e3182b43f7a [HUDI-7438] Fix Azure CI report check with new issue comments add 69086bc3a84 [HUDI-7438] Fix Azure CI report check with new issue comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e3182b43f7a) \ N -- N -- N refs/heads/HUDI-7438-fix-issue-comment-processing (69086bc3a84) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/labeler.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Re: [PR] [MINOR][TESTING] Test PR [hudi]
yihua closed pull request #10737: [MINOR][TESTING] Test PR URL: https://github.com/apache/hudi/pull/10737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962287364 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22587) * 707fc464da051e02301c730b5b5402bbe3bf3a05 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22601) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438][Test] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10743: URL: https://github.com/apache/hudi/pull/10743#issuecomment-1962287404 ## CI report: * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22600) * 3d09199fa55f54af95d9d2535e851bb9d03c8815 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch HUDI-7438-fix-issue-comment-processing updated (3d09199fa55 -> e3182b43f7a)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch HUDI-7438-fix-issue-comment-processing in repository https://gitbox.apache.org/repos/asf/hudi.git discard 3d09199fa55 [HUDI-7438] Fix Azure CI report check with new issue comments add e3182b43f7a [HUDI-7438] Fix Azure CI report check with new issue comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (3d09199fa55) \ N -- N -- N refs/heads/HUDI-7438-fix-issue-comment-processing (e3182b43f7a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/labeler.yml | 58 --- 1 file changed, 43 insertions(+), 15 deletions(-)
Re: [PR] [HUDI-7438][Test] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10743: URL: https://github.com/apache/hudi/pull/10743#issuecomment-1962286022 ## CI report: * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22600) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962286011 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 1734e2500da3e94e7bf3bd2740f9eed513e4b566 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22597) * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962285984 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22587) * 707fc464da051e02301c730b5b5402bbe3bf3a05 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch HUDI-7438-fix-issue-comment-processing updated (98efc813fec -> 3d09199fa55)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch HUDI-7438-fix-issue-comment-processing in repository https://gitbox.apache.org/repos/asf/hudi.git discard 98efc813fec [HUDI-7438] Fix Azure CI report check with new issue comments add 3d09199fa55 [HUDI-7438] Fix Azure CI report check with new issue comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (98efc813fec) \ N -- N -- N refs/heads/HUDI-7438-fix-issue-comment-processing (3d09199fa55) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/create_azure_ci_check.yml | 1 + 1 file changed, 1 insertion(+)
(hudi) branch HUDI-7438-fix-issue-comment-processing updated (0b1be7eef7f -> 98efc813fec)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch HUDI-7438-fix-issue-comment-processing in repository https://gitbox.apache.org/repos/asf/hudi.git omit 0b1be7eef7f [HUDI-7438] Fix Azure CI report check with new issue comments add 98efc813fec [HUDI-7438] Fix Azure CI report check with new issue comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (0b1be7eef7f) \ N -- N -- N refs/heads/HUDI-7438-fix-issue-comment-processing (98efc813fec) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/create_azure_ci_check.yml | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-)
Re: [PR] [HUDI-7438][Test] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10743: URL: https://github.com/apache/hudi/pull/10743#issuecomment-1962277214 ## CI report: * 0b1be7eef7f597cc3cb8899160700b38601a5c4d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Add permissions to the PR size labeler [hudi]
hudi-bot commented on PR #10478: URL: https://github.com/apache/hudi/pull/10478#issuecomment-1962277121 ## CI report: * 3acf3f7f5de88cc1c770644a3a04de93742a1fd9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21908) * 39975ef29a21ea187185722140bc3a529b16ff6a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22599) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Add permissions to the PR size labeler [hudi]
hudi-bot commented on PR #10478: URL: https://github.com/apache/hudi/pull/10478#issuecomment-1962275778 ## CI report: * 3acf3f7f5de88cc1c770644a3a04de93742a1fd9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21908) * 39975ef29a21ea187185722140bc3a529b16ff6a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7438][Test] Fix Azure CI report check with new issue comments [hudi]
yihua opened a new pull request, #10743: URL: https://github.com/apache/hudi/pull/10743 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch HUDI-7438-fix-issue-comment-processing created (now 0b1be7eef7f)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch HUDI-7438-fix-issue-comment-processing in repository https://gitbox.apache.org/repos/asf/hudi.git at 0b1be7eef7f [HUDI-7438] Fix Azure CI report check with new issue comments No new revisions were added by this update.
(hudi) branch fix-size-labeler deleted (was 0b90ccf97e8)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch fix-size-labeler in repository https://gitbox.apache.org/repos/asf/hudi.git was 0b90ccf97e8 Add permissions to the PR size labeler The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] [MINOR][Test] Add permissions to the PR size labeler [hudi]
yihua closed pull request #10742: [MINOR][Test] Add permissions to the PR size labeler URL: https://github.com/apache/hudi/pull/10742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch fix-size-labeler updated (39975ef29a2 -> 0b90ccf97e8)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch fix-size-labeler in repository https://gitbox.apache.org/repos/asf/hudi.git omit 39975ef29a2 Add permissions to the PR size labeler add 0b90ccf97e8 Add permissions to the PR size labeler This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (39975ef29a2) \ N -- N -- N refs/heads/fix-size-labeler (0b90ccf97e8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/labeler.yml | 6 ++ 1 file changed, 6 insertions(+)
[PR] [MINOR][Test] Add permissions to the PR size labeler [hudi]
yihua opened a new pull request, #10742: URL: https://github.com/apache/hudi/pull/10742 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch fix-size-labeler created (now 39975ef29a2)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch fix-size-labeler in repository https://gitbox.apache.org/repos/asf/hudi.git at 39975ef29a2 Add permissions to the PR size labeler No new revisions were added by this update.
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962266190 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) * 1734e2500da3e94e7bf3bd2740f9eed513e4b566 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22597) * 0b1be7eef7f597cc3cb8899160700b38601a5c4d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22598) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962264492 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) * 1734e2500da3e94e7bf3bd2740f9eed513e4b566 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22597) * 0b1be7eef7f597cc3cb8899160700b38601a5c4d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962263014 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) * 1734e2500da3e94e7bf3bd2740f9eed513e4b566 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine: array When there are Top level variables , Struct and Array[struct] (no complex datatype within array[struct]) [hudi]
Jonathanrodrigr12 commented on issue #7717: URL: https://github.com/apache/hudi/issues/7717#issuecomment-1962262653 Hi, i have the same problem but i am use the HoodieMultiTableStreamer **Description** I have a lot parquet files, all of them have this struct  but the first time when i run the job in emr serverless the data is saved, but int the second attemp i have this error **Expected behavior** The second write succeeds. **Environment Description** Hudi hudi-utilities-bundle_2.12-0.14.0-amzn-0.jar Spark version : 3.4.1 EMR: 6.15.0 Stack Trace `org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0 at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:342) at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleInsertPartition(BaseSparkCommitActionExecutor.java:348) at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:259) at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102) at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:905) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:905) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:377) at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1552) at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1462) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1526) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1349) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:375) at org.apache.spark.rdd.RDD.iterator(RDD.scala:326) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:563) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1541) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:566) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: value at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149) at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:387) at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:369) at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:335) ... 30 more Caused by: org.apache.avro.SchemaParseException: Can't redefine: value at org.apache.avro.Schema$Names.put(Schema.java:1586) at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:844) at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1011) at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278) at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039) at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023) at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1173) at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278) at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039) at
Re: [PR] [HUDI-4444] Refactor DataSourceInternalWriterHelper [hudi]
wombatu-kun commented on code in PR #10715: URL: https://github.com/apache/hudi/pull/10715#discussion_r1501346077 ## hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java: ## @@ -66,13 +66,11 @@ public DataSourceInternalWriterHelper(String instantTime, HoodieWriteConfig writ this.extraMetadata = extraMetadata; this.writeClient = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(new JavaSparkContext(sparkSession.sparkContext())), writeConfig); this.writeClient.setOperationType(operationType); -this.writeClient.startCommitWithTime(instantTime); this.writeClient.initTable(operationType, Option.of(instantTime)); Review Comment: oh, yes, may be you are right, but all tests pass succesfully. Can you give me some advice to make this PR correct? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
wombatu-kun commented on code in PR #10728:
URL: https://github.com/apache/hudi/pull/10728#discussion_r1501343613
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -562,7 +562,7 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty
MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE = ConfigProperty
.key("hoodie.merge.allow.duplicate.on.inserts")
- .defaultValue("false")
+ .defaultValue("true")
.markAdvanced()
Review Comment:
i think there is, as people writes tasks and issues:
https://github.com/apache/hudi/issues/8451
https://issues.apache.org/jira/browse/HUDI-6089
https://issues.apache.org/jira/browse/HUDI-6346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
wombatu-kun commented on code in PR #10728:
URL: https://github.com/apache/hudi/pull/10728#discussion_r1501343700
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -562,7 +562,7 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty
MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE = ConfigProperty
.key("hoodie.merge.allow.duplicate.on.inserts")
- .defaultValue("false")
+ .defaultValue("true")
.markAdvanced()
Review Comment:
i think there is, as people create tasks and issues:
https://github.com/apache/hudi/issues/8451
https://issues.apache.org/jira/browse/HUDI-6089
https://issues.apache.org/jira/browse/HUDI-6346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
wombatu-kun commented on code in PR #10728:
URL: https://github.com/apache/hudi/pull/10728#discussion_r1501343613
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -562,7 +562,7 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty
MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE = ConfigProperty
.key("hoodie.merge.allow.duplicate.on.inserts")
- .defaultValue("false")
+ .defaultValue("true")
.markAdvanced()
Review Comment:
i think there is, as people writes tasks and issues:
https://github.com/apache/hudi/issues/8451
https://issues.apache.org/jira/browse/HUDI-6089
https://issues.apache.org/jira/browse/HUDI-6346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (HUDI-7433) Fix a bug in the HoodieBaseListData.isEmpty() empty-check logic
[ https://issues.apache.org/jira/browse/HUDI-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7433. Resolution: Fixed Fixed via master branch: 22e2063261ceded17a12d5443ca58910bd6a471b > Fix a bug in the HoodieBaseListData.isEmpty() empty-check logic > --- > > Key: HUDI-7433 > URL: https://issues.apache.org/jira/browse/HUDI-7433 > Project: Apache Hudi > Issue Type: Bug >Reporter: bradley >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0, 0.14.1 > > Original Estimate: 12h > Remaining Estimate: 12h > > Fix a bug in the HoodieBaseListData.isEmpty() empty-check logic. PR: > [https://github.com/apache/hudi/pull/10722|http://example.com] -- This message was sent by Atlassian Jira (v8.20.10#820010)
(hudi) branch master updated (b8b6917f8b0 -> 22e2063261c)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from b8b6917f8b0 [HUDI-7440] Verify field exist in schema before fetching the value (#10733) add 22e2063261c [HUDI-7433] Fix a bug in the HoodieBaseListData.isEmpty() empty-check logic (#10722) (#10722) No new revisions were added by this update. Summary of changes: .../hudi/common/data/HoodieBaseListData.java | 2 +- .../hudi/common/data/TestHoodieListData.java | 22 ++ 2 files changed, 23 insertions(+), 1 deletion(-)
Re: [PR] Fix a bug in the HoodieBaseListData.isEmpty() empty-check logic [hudi]
danny0405 merged PR #10722: URL: https://github.com/apache/hudi/pull/10722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Bugs about the hudi table created by hive catalog and wrong results when querying RO table [hudi]
danny0405 commented on issue #10735: URL: https://github.com/apache/hudi/issues/10735#issuecomment-1962251473 We should not use Hive catalog, that's why we introduce a `HoodieHiveCatalog` where we do many tasks for `createTable`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (HUDI-7440) Verify field exist in schema before fetching the value
[ https://issues.apache.org/jira/browse/HUDI-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7440. Resolution: Fixed Fixed via master branch: b8b6917f8b0ba0d8b3b3034a275aa1f0947be954 > Verify field exist in schema before fetching the value > -- > > Key: HUDI-7440 > URL: https://issues.apache.org/jira/browse/HUDI-7440 > Project: Apache Hudi > Issue Type: Improvement > Components: code-quality >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.14.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7440) Verify field exist in schema before fetching the value
[ https://issues.apache.org/jira/browse/HUDI-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7440: - Labels: pull-request-available (was: ) > Verify field exist in schema before fetching the value > -- > > Key: HUDI-7440 > URL: https://issues.apache.org/jira/browse/HUDI-7440 > Project: Apache Hudi > Issue Type: Improvement > Components: code-quality >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.14.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
(hudi) branch master updated (cddd7d416a5 -> b8b6917f8b0)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from cddd7d416a5 [HUDI-7275] Separate use of HoodieTimelineTimeZone.UTC and LOCAL in tests to prevent infinite loops (#10738) add b8b6917f8b0 [HUDI-7440] Verify field exist in schema before fetching the value (#10733) No new revisions were added by this update. Summary of changes: .../hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-)
Re: [PR] [HUDI-7440] Verify field exist in schema before fetching the value [hudi]
danny0405 merged PR #10733: URL: https://github.com/apache/hudi/pull/10733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-7440) Verify field exist in schema before fetching the value
Danny Chen created HUDI-7440: Summary: Verify field exist in schema before fetching the value Key: HUDI-7440 URL: https://issues.apache.org/jira/browse/HUDI-7440 Project: Apache Hudi Issue Type: Improvement Components: code-quality Reporter: Danny Chen Fix For: 0.14.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
danny0405 commented on code in PR #10728:
URL: https://github.com/apache/hudi/pull/10728#discussion_r1501342711
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##
@@ -562,7 +562,7 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty
MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE = ConfigProperty
.key("hoodie.merge.allow.duplicate.on.inserts")
- .defaultValue("false")
+ .defaultValue("true")
.markAdvanced()
Review Comment:
Is there any real use case to illustrate this switch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (HUDI-7275) org.apache.hudi.TestHoodieSparkSqlWriter#testInsertDatasetWithTimelineTimezoneUTC causes issues with following tests
[
https://issues.apache.org/jira/browse/HUDI-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed HUDI-7275.
Resolution: Fixed
Fixed via master branch: cddd7d416a5db31de879790a80a33bb86cf02cbc
> org.apache.hudi.TestHoodieSparkSqlWriter#testInsertDatasetWithTimelineTimezoneUTC
> causes issues with following tests
>
>
> Key: HUDI-7275
> URL: https://issues.apache.org/jira/browse/HUDI-7275
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Jonathan Vexler
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.2
>
>
> When the next test runs, it gets stuck in an infinite loop and the output is
> {code:java}
> 60331 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - Released the connection of the timeGenerator lock
> 60331 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - LockProvider for TimeGenerator:
> org.apache.hudi.client.transaction.lock.InProcessLockProvider
> 60331 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ACQUIRING
> 60331 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state ACQUIRED
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state RELEASING
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state RELEASED
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ALREADY_RELEASED
> 60333 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - Released the connection of the timeGenerator lock
> 60333 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - LockProvider for TimeGenerator:
> org.apache.hudi.client.transaction.lock.InProcessLockProvider
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ACQUIRING
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state ACQUIRED
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state RELEASING
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state RELEASED
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
>
[jira] [Updated] (HUDI-7275) org.apache.hudi.TestHoodieSparkSqlWriter#testInsertDatasetWithTimelineTimezoneUTC causes issues with following tests
[
https://issues.apache.org/jira/browse/HUDI-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated HUDI-7275:
-
Fix Version/s: 0.14.2
> org.apache.hudi.TestHoodieSparkSqlWriter#testInsertDatasetWithTimelineTimezoneUTC
> causes issues with following tests
>
>
> Key: HUDI-7275
> URL: https://issues.apache.org/jira/browse/HUDI-7275
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Jonathan Vexler
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.2
>
>
> When the next test runs, it gets stuck in an infinite loop and the output is
> {code:java}
> 60331 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - Released the connection of the timeGenerator lock
> 60331 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - LockProvider for TimeGenerator:
> org.apache.hudi.client.transaction.lock.InProcessLockProvider
> 60331 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ACQUIRING
> 60331 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state ACQUIRED
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state RELEASING
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state RELEASED
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ALREADY_RELEASED
> 60333 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - Released the connection of the timeGenerator lock
> 60333 [main] INFO org.apache.hudi.common.table.timeline.TimeGeneratorBase []
> - LockProvider for TimeGenerator:
> org.apache.hudi.client.transaction.lock.InProcessLockProvider
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state ACQUIRING
> 60333 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state ACQUIRED
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 1,
> Read locks = 0], Thread main, In-process lock state RELEASING
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread main, In-process lock state RELEASED
> 60334 [main] INFO
> org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path
> /var/folders/d0/l7mfhzl1661byhh3mbyg5fv0gn/T/hoodie_test_path7599985521109702031_1,
> Lock Instance
> java.util.concurrent.locks.ReentrantReadWriteLock@5d045508[Write locks = 0,
> Read locks = 0], Thread
(hudi) branch master updated (6f74c7f6ec6 -> cddd7d416a5)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 6f74c7f6ec6 [HUDI-7438] Add GitHub action to check Azure CI report (#10731) add cddd7d416a5 [HUDI-7275] Separate use of HoodieTimelineTimeZone.UTC and LOCAL in tests to prevent infinite loops (#10738) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 59 ++- .../apache/hudi/TestHoodieSparkSqlWriterUtc.scala | 85 ++ 2 files changed, 91 insertions(+), 53 deletions(-) create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriterUtc.scala
Re: [PR] [HUDI-7275] Separate use of HoodieTimelineTimeZone.UTC and LOCAL in tests to prevent infinite loops [hudi]
danny0405 merged PR #10738: URL: https://github.com/apache/hudi/pull/10738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-4444] Refactor DataSourceInternalWriterHelper [hudi]
danny0405 commented on code in PR #10715: URL: https://github.com/apache/hudi/pull/10715#discussion_r1501342066 ## hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java: ## @@ -66,13 +66,11 @@ public DataSourceInternalWriterHelper(String instantTime, HoodieWriteConfig writ this.extraMetadata = extraMetadata; this.writeClient = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(new JavaSparkContext(sparkSession.sparkContext())), writeConfig); this.writeClient.setOperationType(operationType); -this.writeClient.startCommitWithTime(instantTime); this.writeClient.initTable(operationType, Option.of(instantTime)); Review Comment: The `initTable` has a param `Option instantTime` which is used for startting the trasanction conflict process, not sure what happens if the metadata file does not exist there at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch asf-site updated: [HUDI-6089][DOCS] update default value of hoodie.merge.allow.duplicate.on.inserts to true (#10739)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 3235c366f84 [HUDI-6089][DOCS] update default value of hoodie.merge.allow.duplicate.on.inserts to true (#10739) 3235c366f84 is described below commit 3235c366f846adf1498b65efaf469a65a13b035a Author: wombatu-kun AuthorDate: Sat Feb 24 11:25:20 2024 +0700 [HUDI-6089][DOCS] update default value of hoodie.merge.allow.duplicate.on.inserts to true (#10739) Co-authored-by: Vova Kolmakov --- website/docs/configurations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/configurations.md b/website/docs/configurations.md index 18c3581e305..e52f0a52a75 100644 --- a/website/docs/configurations.md +++ b/website/docs/configurations.md @@ -831,7 +831,7 @@ Configurations that control write behavior on Hudi tables. These can be directly | [hoodie.markers.delete.parallelism](#hoodiemarkersdeleteparallelism) | 100 | Determines the parallelism for deleting marker files, which are used to track all files (valid or invalid/partial) written during a write operation. Increase this value if delays are observed, with large batch writes.`Config Param: MARKERS_DELETE_PARALLELISM_VALUE` [...] | [hoodie.markers.timeline_server_based.batch.interval_ms](#hoodiemarkerstimeline_server_basedbatchinterval_ms) | 50 | The batch interval in milliseconds for marker creation batch processing`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS``Since Version: 0.9.0` [...] | [hoodie.markers.timeline_server_based.batch.num_threads](#hoodiemarkerstimeline_server_basedbatchnum_threads) | 20 | Number of threads to use for batch processing marker creation requests at the timeline server`Config Param: MARKERS_TIMELINE_SERVER_BASED_BATCH_NUM_THREADS``Since Version: 0.9.0` [...] -| [hoodie.merge.allow.duplicate.on.inserts](#hoodiemergeallowduplicateoninserts) | false | When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.`Config Param: MERGE_ALLOW_DUPLICATE_ON [...] +| [hoodie.merge.allow.duplicate.on.inserts](#hoodiemergeallowduplicateoninserts) | true | When enabled, we allow duplicate keys even if inserts are routed to merge with an existing file (for ensuring file sizing). This is only relevant for insert operation, since upsert, delete operations will ensure unique key constraints are maintained.`Config Param: MERGE_ALLOW_DUPLICATE_ON [...] | [hoodie.merge.data.validation.enabled](#hoodiemergedatavalidationenabled) | false | When enabled, data validation checks are performed during merges to ensure expected number of records after merge operation.`Config Param: MERGE_DATA_VALIDATION_CHECK_ENABLE` [...] | [hoodie.merge.small.file.group.candidates.limit](#hoodiemergesmallfilegroupcandidateslimit) | 1 | Limits number of file groups, whose base file satisfies small-file limit, to consider for appending records during upsert operation. Only applicable to MOR tables`Config Param: MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT` [...] | [hoodie.release.resource.on.completion.enable](#hoodiereleaseresourceoncompletionenable) | true | Control to enable release all persist rdds when the spark job finish.`Config Param: RELEASE_RESOURCE_ENABLE``Since Version: 0.11.0`
Re: [PR] [HUDI-6089][DOCS] update default value of hoodie.merge.allow.duplicate.on.inserts to true [hudi]
danny0405 merged PR #10739: URL: https://github.com/apache/hudi/pull/10739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Hive SYNC TOOL on EMR failed, Exception in thread main java.ang.NoClassDefFoundError: com/fasterxml/... [hudi]
danny0405 commented on issue #10741: URL: https://github.com/apache/hudi/issues/10741#issuecomment-1962247269 Looks like a jackson jar conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962220086 ## CI report: * 82ab33600666ccd65fd4f963277e71ff2b8c7726 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22594) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962203426 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962200894 ## CI report: * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * b6a1c7b7b8ba77121c972e80bf602de239fa9138 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22592) * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962200833 ## CI report: * 892125e6b08cf7629cc4f9a586809f093673d0b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22590) * 82ab33600666ccd65fd4f963277e71ff2b8c7726 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22594) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962178951 ## CI report: * 892125e6b08cf7629cc4f9a586809f093673d0b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22590) * 82ab33600666ccd65fd4f963277e71ff2b8c7726 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962172458 ## CI report: * 892125e6b08cf7629cc4f9a586809f093673d0b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22590) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch asf-site updated: updated delete to mention duplicates- and did some writing cleanup (#10659)
This is an automated email from the ASF dual-hosted git repository. bhavanisudha pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new eb6a998fd85 updated delete to mention duplicates- and did some writing cleanup (#10659) eb6a998fd85 is described below commit eb6a998fd85deaf7fef551f74ea70b0f08cffe22 Author: nadine farah AuthorDate: Fri Feb 23 16:16:27 2024 -0800 updated delete to mention duplicates- and did some writing cleanup (#10659) --- website/docs/write_operations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/write_operations.md b/website/docs/write_operations.md index 90b87499fe0..3146db05802 100644 --- a/website/docs/write_operations.md +++ b/website/docs/write_operations.md @@ -29,7 +29,7 @@ of initial load. However, this just does a best-effort job at sizing files vs gu Hudi supports implementing two types of deletes on data stored in Hudi tables, by enabling the user to specify a different record payload implementation. - **Soft Deletes** : Retain the record key and just null out the values for all the other fields. This can be achieved by ensuring the appropriate fields are nullable in the table schema and simply upserting the table after setting these fields to null. -- **Hard Deletes** : A stronger form of deletion is to physically remove any trace of the record from the table. This can be achieved in 3 different ways. +- **Hard Deletes** : This method entails completely eradicating all evidence of a record from the table, including any duplicates. There are three distinct approaches to accomplish this: - Using DataSource, set `OPERATION_OPT_KEY` to `DELETE_OPERATION_OPT_VAL`. This will remove all the records in the DataSet being submitted. - Using DataSource, set `PAYLOAD_CLASS_OPT_KEY` to `"org.apache.hudi.EmptyHoodieRecordPayload"`. This will remove all the records in the DataSet being submitted. - Using DataSource or Hudi Streamer, add a column named `_hoodie_is_deleted` to DataSet. The value of this column must be set to `true` for all the records to be deleted and either `false` or left null for any records which are to be upserted.
Re: [PR] updated delete to mention duplicates- and did some writing cleanup [hudi]
bhasudha merged PR #10659: URL: https://github.com/apache/hudi/pull/10659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Can process parquet file if using upsert or bulk_insert but cannot process parquet file if using insert [hudi]
soumilshah1995 commented on issue #10725: URL: https://github.com/apache/hudi/issues/10725#issuecomment-1962150892 Roger that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962138653 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 75e18c04a204774cbe16709edd462fd046f44335 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22589) * bdfb572588500ba933ff3c10596502a50b0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22591) * b6a1c7b7b8ba77121c972e80bf602de239fa9138 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22592) * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22593) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962134737 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 75e18c04a204774cbe16709edd462fd046f44335 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22589) * bdfb572588500ba933ff3c10596502a50b0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22591) * b6a1c7b7b8ba77121c972e80bf602de239fa9138 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22592) * 392c0624e5b0e9ab8883781d0e7ef4c11dc87319 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962130304 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 75e18c04a204774cbe16709edd462fd046f44335 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22589) * bdfb572588500ba933ff3c10596502a50b0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22591) * b6a1c7b7b8ba77121c972e80bf602de239fa9138 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962093422 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 75e18c04a204774cbe16709edd462fd046f44335 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22589) * bdfb572588500ba933ff3c10596502a50b0f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962093263 ## CI report: * bdf483d0c96502fc888e7dc7f2fe087f7643ecb6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22585) * 892125e6b08cf7629cc4f9a586809f093673d0b4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22590) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962087323 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN * 75e18c04a204774cbe16709edd462fd046f44335 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1962087189 ## CI report: * bdf483d0c96502fc888e7dc7f2fe087f7643ecb6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22585) * 892125e6b08cf7629cc4f9a586809f093673d0b4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962081364 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) * 040e2c89e131b994c2a0b7875e512ab992b3c547 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Spark Write into MoR type hudi table small parquets issue + Athena Internal Error [hudi]
huliwuli commented on issue #10716: URL: https://github.com/apache/hudi/issues/10716#issuecomment-1962059629 **Regarding Athena Issue:** Due to the small size of parquets, I implemented clustering (inline) with max commits =1 for test. Athena Raises Error: Generic_INTERNAL_ERROR: Can not read value at 0 in block -1 in S3/XX/XXX//X/date=20XX-XX-XX.parquet I checked commits, hudi-cli shows two commits one is before the clustering, another one is created after the clustering, but when querying data from _rt table it still has the old commit time. I think the Hudi table did not sync with Hive in this case -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
yihua commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1962035139 test comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1962009037 ## CI report: * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22587) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [SUPPORT] Hive SYNC TOOL on EMR failed, Exception in thread main java.ang.NoClassDefFoundError: com/fasterxml/... [hudi]
huliwuli opened a new issue, #10741: URL: https://github.com/apache/hudi/issues/10741 Tips before filing an issue Describe the problem you faced Did Async Clustering on EMR 6.14 and Hive on Athena did not sync the latest commit after clustering? I want to use the hive sync tool to sync it. When using ``` cd /usr/lib/hudi/bin ./run_sync_tool.sh --base-path s3: --database --table --partitioned-by ``` I got the error caused by java.lang.ClassNotFoundException: com.fasterxml.jackson,datatype.jsr310.JavaTimeModule. Also, I noticed AWS documentation includes use-jdbc false  so I did ``` cd /usr/lib/hudi/bin ./run_sync_tool.sh --base-path s3: --database --table --partitioned-by --sync-mode hms --use-jdbc false --sync-tool-classes org.apache.hudi.hive.MultiPartKeysValueExtractor ``` Then I got: 'false' but no main parameter was defined in your arg class Environment Description Hudi version : 0.13.0 Spark version : 3.4.1 Hive version : 0.13.1 Hadoop version : Storage (HDFS/S3/GCS..) : S3 Running on Docker? (yes/no) : NO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1961961656 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22588) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
hudi-bot commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1961953455 ## CI report: * 988039cbb47927ba6f0ef3e0c2f77e0736d3cc36 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961941627 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22584) * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22587) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1961941516 ## CI report: * bdf483d0c96502fc888e7dc7f2fe087f7643ecb6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22585) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
yihua commented on PR #10740: URL: https://github.com/apache/hudi/pull/10740#issuecomment-1961919983 New comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7438] Fix Azure CI report check with new issue comments [hudi]
yihua opened a new pull request, #10740: URL: https://github.com/apache/hudi/pull/10740 ### Change Logs This PR fixes Azure CI report check with new issue comments. ### Impact Fixes bug and improves Azure CI check. ### Risk level none ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-196125 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22584) * dbe9cea4f203fe6f056b1f1e1f639e7ad775736c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-1961878815 ## CI report: * a592ad1361583635a55ead7b634d2b20a92c239f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22586) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-1961869252 ## CI report: * d0faf8850bf513fb1d610831b3459680c244 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22552) * a592ad1361583635a55ead7b634d2b20a92c239f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22586) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1961816513 ## CI report: * 0fcfe358f651975c5276f7030ebb81b0011e5d5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22536) * bdf483d0c96502fc888e7dc7f2fe087f7643ecb6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22585) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-1961815729 ## CI report: * d0faf8850bf513fb1d610831b3459680c244 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22552) * a592ad1361583635a55ead7b634d2b20a92c239f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7430] Fix empty schema issue for compactor [hudi]
hudi-bot commented on PR #10718: URL: https://github.com/apache/hudi/pull/10718#issuecomment-1961806589 ## CI report: * 0fcfe358f651975c5276f7030ebb81b0011e5d5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22536) * bdf483d0c96502fc888e7dc7f2fe087f7643ecb6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961715953 ## CI report: * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22584) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961554948 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22578) * 2b66c852e373113c8bd1bd66bd0376a8f537044e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22584) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1961542149 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22578) * 2b66c852e373113c8bd1bd66bd0376a8f537044e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Spark Write into MoR type hudi table small parquets issue + Athena Internal Error [hudi]
huliwuli commented on issue #10716: URL: https://github.com/apache/hudi/issues/10716#issuecomment-1961532330 @ad1happy2go Thanks for the reply. I used insert operation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7439) Remove redundant logs from hive server from Azure CI 4th module
[
https://issues.apache.org/jira/browse/HUDI-7439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Liu updated HUDI-7439:
--
Description:
When there is an error or failure, we can see at the bottom there are some
redundant logs from hive server, which is confusing.
{code:java}
2024-02-23T14:16:56.5032362Z ##[error]SLF4J: Class path contains multiple SLF4J
bindings.
2024-02-23T14:16:56.5298013Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5299627Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5300659Z ##[error]SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2024-02-23T14:16:56.5303968Z ##[error]ng is of type
[org.slf4j.impl.Reload4jLoggerFactory]
2024-02-23T14:16:56.5307486Z ##[error]SLF4J: Class path contains multiple SLF4J
bindings.
2024-02-23T14:16:56.5311372Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5312540Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5314223Z ##[error]SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2024-02-23T14:16:56.5319464Z ##[error]SLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]
2024-02-23T14:16:56.5325886Z ##[error]log4j:WARN No appenders could be found
for logger (org.apache.hudi.hadoop.fs.HadoopFSUtils).
2024-02-23T14:16:56.5328637Z ##[error]perly.
2024-02-23T14:16:56.5329457Z ##[error]log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2024-02-23T14:16:56.5333098Z ##[error]SLF4J: Class path contains multiple SLF4J
bindings.
2024-02-23T14:16:56.5334067Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5335118Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5336283Z ##[error]SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2024-02-23T14:16:56.5337413Z ##[error]SLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]
2024-02-23T14:16:56.5338200Z ##[error]SLF4J: Class path contains multiple SLF4J
bindings.
2024-02-23T14:16:56.5339865Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5341013Z ##[error]SLF4J: Found binding in
[jar:file:/root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.2/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2024-02-23T14:16:56.5341919Z ##[error]SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2024-02-23T14:16:56.5342719Z ##[error]SLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]r=2024-02-23
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
{code}
was:
When there is an error or failure, we can see at the bottom there are some
redundant logs from hive server, which is confusing.
{code:java}
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]r=2024-02-23
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
{code}
> Remove redundant logs from hive server from Azure CI 4th module
> ---
>
> Key: HUDI-7439
> URL: https://issues.apache.org/jira/browse/HUDI-7439
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Lin Liu
>Assignee: Lin Liu
>Priority: Major
>
> When there is an error or failure, we can see at the bottom there are some
> redundant logs from hive server, which is
[jira] [Created] (HUDI-7439) Remove logs from hive server from Azure CI 4th module
Lin Liu created HUDI-7439:
-
Summary: Remove logs from hive server from Azure CI 4th module
Key: HUDI-7439
URL: https://issues.apache.org/jira/browse/HUDI-7439
Project: Apache Hudi
Issue Type: Bug
Reporter: Lin Liu
Assignee: Lin Liu
When there is an error or failure, we can see at the bottom there are some
redundant logs from hive server, which is confusing.
{code:java}
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]r=2024-02-23
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
##[error]OK
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HUDI-7439) Remove redundant logs from hive server from Azure CI 4th module
[
https://issues.apache.org/jira/browse/HUDI-7439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Liu updated HUDI-7439:
--
Summary: Remove redundant logs from hive server from Azure CI 4th module
(was: Remove logs from hive server from Azure CI 4th module)
> Remove redundant logs from hive server from Azure CI 4th module
> ---
>
> Key: HUDI-7439
> URL: https://issues.apache.org/jira/browse/HUDI-7439
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Lin Liu
>Assignee: Lin Liu
>Priority: Major
>
> When there is an error or failure, we can see at the bottom there are some
> redundant logs from hive server, which is confusing.
>
> {code:java}
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]r=2024-02-23
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> ##[error]OK
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
hudi-bot commented on PR #10728: URL: https://github.com/apache/hudi/pull/10728#issuecomment-1961420614 ## CI report: * 22f875240369cab37842e58c9d504873643f10e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22583) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-6089][DOCS] update default value of hoodie.merge.allow.duplicate.on.inserts to true [hudi]
wombatu-kun opened a new pull request, #10739: URL: https://github.com/apache/hudi/pull/10739 ### Change Logs Update documentation: update default value of hoodie.merge.allow.duplicate.on.inserts to true ### Impact none ### Risk level (write none, low medium or high below) none ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-4444] Refactor DataSourceInternalWriterHelper [hudi]
wombatu-kun commented on code in PR #10715: URL: https://github.com/apache/hudi/pull/10715#discussion_r1500706025 ## hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java: ## @@ -66,13 +66,11 @@ public DataSourceInternalWriterHelper(String instantTime, HoodieWriteConfig writ this.extraMetadata = extraMetadata; this.writeClient = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(new JavaSparkContext(sparkSession.sparkContext())), writeConfig); this.writeClient.setOperationType(operationType); -this.writeClient.startCommitWithTime(instantTime); this.writeClient.initTable(operationType, Option.of(instantTime)); Review Comment: Definitely, initTable does not require the requested commit file to be existed. That's why i moved invocation of startCommitWithTime out of constructor right to that place, where it is needed (first statement of createInflightCommit method). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
hudi-bot commented on PR #10728: URL: https://github.com/apache/hudi/pull/10728#issuecomment-1961258025 ## CI report: * 6348547bbb296493bb2d137a9764199117784e10 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22555) * 22f875240369cab37842e58c9d504873643f10e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22583) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6089] Handle default insert behaviour to ingest duplicates [hudi]
hudi-bot commented on PR #10728: URL: https://github.com/apache/hudi/pull/10728#issuecomment-1961247518 ## CI report: * 6348547bbb296493bb2d137a9764199117784e10 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22555) * 22f875240369cab37842e58c9d504873643f10e1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7275] Separate use of HoodieTimelineTimeZone.UTC and LOCAL in tests to prevent infinite loops [hudi]
hudi-bot commented on PR #10738: URL: https://github.com/apache/hudi/pull/10738#issuecomment-1961075063 ## CI report: * 18435cdea361b920b8ff01e4ded0143d94c6d6f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22582) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Test PR [hudi]
hudi-bot commented on PR #10737: URL: https://github.com/apache/hudi/pull/10737#issuecomment-1960909243 ## CI report: * 0f4271e8c543fd2cda736a2af5c9356533c48cba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22581) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data [hudi]
hudi-bot commented on PR #10736: URL: https://github.com/apache/hudi/pull/10736#issuecomment-1960909193 ## CI report: * 6c41fe5e29de60a4d33701ebfd4aefc898cc605f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22578) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
