[GitHub] [hudi] hudi-bot commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995518802 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4365) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot removed a comment on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995474370 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4365) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3036) Enhance Cleaner Docs
Kyle Weller created HUDI-3036: - Summary: Enhance Cleaner Docs Key: HUDI-3036 URL: https://issues.apache.org/jira/browse/HUDI-3036 Project: Apache Hudi Issue Type: Sub-task Reporter: Kyle Weller Assignee: Kyle Weller Fix For: 0.11.0 This blog has rich info that should be in the docs: [https://hudi.apache.org/blog/2021/06/10/employing-right-configurations-for-hudi-cleaner/] Slack disc mention: https://apache-hudi.slack.com/archives/C4D716NPQ/p1639497026391400 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] lamberken commented on pull request #4291: [HUDI-2990] Sync to HMS when deleting partitions
lamberken commented on pull request #4291: URL: https://github.com/apache/hudi/pull/4291#issuecomment-995515379 NPE? it's hard to analyze it. @XuQianJin-Stars -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4287: [DO NOT MERGE] 0.10.0 release patch for flink
hudi-bot removed a comment on pull request #4287: URL: https://github.com/apache/hudi/pull/4287#issuecomment-995512935 ## CI report: * ef2184559bb19768efcb16869027c81e54f873fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4329) * 3cd93c6f3070a75a7ee65207e2961381d5fc3990 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4287: [DO NOT MERGE] 0.10.0 release patch for flink
hudi-bot commented on pull request #4287: URL: https://github.com/apache/hudi/pull/4287#issuecomment-995514386 ## CI report: * ef2184559bb19768efcb16869027c81e54f873fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4329) * 3cd93c6f3070a75a7ee65207e2961381d5fc3990 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4371) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3011) Add support to start incremental consumption from begin time rather than latest commit time with S3EventsHoodieIncrSource
[ https://issues.apache.org/jira/browse/HUDI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3011: -- Fix Version/s: 0.11.0 > Add support to start incremental consumption from begin time rather than > latest commit time with S3EventsHoodieIncrSource > - > > Key: HUDI-3011 > URL: https://issues.apache.org/jira/browse/HUDI-3011 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: core-flow-ds, pull-request-available, sev:normal > Fix For: 0.11.0 > > > with S3EventsHoodieIncrSource, if begin time is not provided to consume from > 1st meta table, we fallback to latest commit time. we should support starting > from earliest instant. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1604) Fix archival max log size and potentially a bug in archival
[
https://issues.apache.org/jira/browse/HUDI-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1604:
--
Labels: core-flow-ds sev:high user-support-issues (was: core-flow-ds
sev:critical sev:high user-support-issues)
> Fix archival max log size and potentially a bug in archival
> ---
>
> Key: HUDI-1604
> URL: https://issues.apache.org/jira/browse/HUDI-1604
> Project: Apache Hudi
> Issue Type: Bug
> Components: Cleaner
>Affects Versions: 0.7.0
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: core-flow-ds, sev:high, user-support-issues
>
> Gist of the issue from Udit
>
> I took a deeper look at this. For you this seems to be happening in the
> archival code path:
>
> {{ at
> org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:309)
> at
> org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:282)
> at
> org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:133)
> at
> org.apache.hudi.client.HoodieWriteClient.postCommit(HoodieWriteClient.java:381)}}
> In {{HoodieTimelineArchiveLog}} where it needs to write log files with commit
> record, similar to how log files are written for MOR tables. However, in this
> code I notice a couple of issues:
> * The default maximum log block size of 256 MB defined
> [here|https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieStorageConfig.java#L51],
> is not utilized for this class and is only used for the MOR log blocks
> writing case. As a result, there is no real control over the block size that
> it can end up writing which can potentially overflow
> {{ByteArrayOutputStream}} whose maximum size is {{Integer.MAX_VALE - 8}}.
> That is what seems to be happening in this scenario here because of an
> integer overflow following that code path inside {{ByteArrayOutputStream}}.
> So we need to use the maximum block size concept here as well.
> * In addition I see a bug in code
> [here|https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L302]
> where even after flushing out the records into a file after a batch size of
> 10 (default) it is not clearing the list and just goes on accumulating the
> records. This seems logically wrong as well (duplication), apart from the
> fact that it would keep increasing the log file blocks size it is writing.
> Reference: https://github.com/apache/hudi/issues/2408#issuecomment-758320870
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4287: [DO NOT MERGE] 0.10.0 release patch for flink
hudi-bot commented on pull request #4287: URL: https://github.com/apache/hudi/pull/4287#issuecomment-995512935 ## CI report: * ef2184559bb19768efcb16869027c81e54f873fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4329) * 3cd93c6f3070a75a7ee65207e2961381d5fc3990 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4287: [DO NOT MERGE] 0.10.0 release patch for flink
hudi-bot removed a comment on pull request #4287: URL: https://github.com/apache/hudi/pull/4287#issuecomment-994800082 ## CI report: * ef2184559bb19768efcb16869027c81e54f873fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4329) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3776: [HUDI-2543]: Added guides section
pratyakshsharma commented on pull request #3776: URL: https://github.com/apache/hudi/pull/3776#issuecomment-995512290 looks like there was some issue with the build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot removed a comment on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995510054 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) * 0ca37e34896362e020502bc1cd0008bca050752a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot commented on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995511563 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) * 0ca37e34896362e020502bc1cd0008bca050752a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4370) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-3015) Implement #reset and #sync for metadata filesystem view
[ https://issues.apache.org/jira/browse/HUDI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460450#comment-17460450 ] Danny Chen commented on HUDI-3015: -- Fixed via master branch: ea2eba1a5532d0f0f66268a45683763dec08 > Implement #reset and #sync for metadata filesystem view > --- > > Key: HUDI-3015 > URL: https://issues.apache.org/jira/browse/HUDI-3015 > Project: Apache Hudi > Issue Type: Task > Components: Common Core >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot removed a comment on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995493823 ## CI report: * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) * 70503ca127d5638bf49440d630ae90de41712d0f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HUDI-3015) Implement #reset and #sync for metadata filesystem view
[ https://issues.apache.org/jira/browse/HUDI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-3015. -- > Implement #reset and #sync for metadata filesystem view > --- > > Key: HUDI-3015 > URL: https://issues.apache.org/jira/browse/HUDI-3015 > Project: Apache Hudi > Issue Type: Task > Components: Common Core >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot removed a comment on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995488504 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot commented on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995510054 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) * 0ca37e34896362e020502bc1cd0008bca050752a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot commented on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995509877 ## CI report: * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) * 70503ca127d5638bf49440d630ae90de41712d0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4369) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
danny0405 merged pull request #4307: URL: https://github.com/apache/hudi/pull/4307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (f5b07a7 -> ea2eba1)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from f5b07a7 [HUDI-2998] claiming rfc number for consistent hashing index (#4303) add ea2eba1 [HUDI-3015] Implement #reset and #sync for metadata filesystem view (#4307) No new revisions were added by this update. Summary of changes: .../hudi/client/TestHoodieClientMultiWriter.java | 59 ++ .../table/view/AbstractTableFileSystemView.java| 5 +- .../table/view/PriorityBasedFileSystemView.java| 4 +- .../view/RemoteHoodieTableFileSystemView.java | 3 +- .../metadata/FileSystemBackedTableMetadata.java| 5 ++ .../hudi/metadata/HoodieBackedTableMetadata.java | 6 +++ .../metadata/HoodieMetadataFileSystemView.java | 12 + .../apache/hudi/metadata/HoodieTableMetadata.java | 5 ++ .../view/TestPriorityBasedFileSystemView.java | 4 +- 9 files changed, 63 insertions(+), 40 deletions(-)
[GitHub] [hudi] yuzhaojing commented on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
yuzhaojing commented on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995508694 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot commented on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995506702 ## CI report: * bdcac7c5442d3f79cd75fc02e13d606e4bd6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4364) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot removed a comment on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995463456 ## CI report: * a157c751a957cfbc82cf699b479c376e93549ce4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4359) * bdcac7c5442d3f79cd75fc02e13d606e4bd6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4364) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
danny0405 commented on a change in pull request #4307:
URL: https://github.com/apache/hudi/pull/4307#discussion_r770275770
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java
##
@@ -100,38 +100,33 @@ public void
testHoodieClientBasicMultiWriter(HoodieTableType tableType) throws E
.build()).withAutoCommit(false).withProperties(properties).build();
// Create the first commit
createCommitWithInserts(cfg, getHoodieWriteClient(cfg), "000", "001", 200);
-try {
- ExecutorService executors = Executors.newFixedThreadPool(2);
- SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
- SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
- Future future1 = executors.submit(() -> {
-String newCommitTime = "004";
-int numRecords = 100;
-String commitTimeBetweenPrevAndNew = "002";
-try {
- createCommitWithUpserts(cfg, client1, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
-} catch (Exception e1) {
- assertTrue(e1 instanceof HoodieWriteConflictException);
- throw new RuntimeException(e1);
-}
- });
- Future future2 = executors.submit(() -> {
-String newCommitTime = "005";
-int numRecords = 100;
-String commitTimeBetweenPrevAndNew = "002";
-try {
- createCommitWithUpserts(cfg, client2, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
-} catch (Exception e2) {
- assertTrue(e2 instanceof HoodieWriteConflictException);
- throw new RuntimeException(e2);
-}
- });
- future1.get();
- future2.get();
- fail("Should not reach here, this means concurrent writes were handled
incorrectly");
-} catch (Exception e) {
- // Expected to fail due to overlapping commits
-}
+ExecutorService executors = Executors.newFixedThreadPool(2);
+SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+Future future1 = executors.submit(() -> {
+ String newCommitTime = "004";
+ int numRecords = 100;
+ String commitTimeBetweenPrevAndNew = "002";
+ try {
+createCommitWithUpserts(cfg, client1, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
+ } catch (Exception e1) {
+assertTrue(e1 instanceof HoodieWriteConflictException);
+throw new RuntimeException(e1);
+ }
+});
+Future future2 = executors.submit(() -> {
+ String newCommitTime = "005";
+ int numRecords = 100;
+ String commitTimeBetweenPrevAndNew = "002";
+ try {
+createCommitWithUpserts(cfg, client2, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
+ } catch (Exception e2) {
+assertTrue(e2 instanceof HoodieWriteConflictException);
+throw new RuntimeException(e2);
+ }
+});
+future1.get();
Review comment:
Because the fix of the metadata sync. The writer can resolve the
conflicts correctly now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a change in pull request #4254: [HUDI-2547] Schedule Flink compaction in service
yuzhaojing commented on a change in pull request #4254:
URL: https://github.com/apache/hudi/pull/4254#discussion_r770273132
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/compact/HoodieFlinkCompactor.java
##
@@ -18,141 +18,299 @@
package org.apache.hudi.sink.compact;
+import org.apache.hudi.async.HoodieAsyncService;
import org.apache.hudi.avro.model.HoodieCompactionPlan;
import org.apache.hudi.client.HoodieFlinkWriteClient;
import org.apache.hudi.common.table.HoodieTableMetaClient;
import org.apache.hudi.common.table.timeline.HoodieInstant;
import org.apache.hudi.common.table.timeline.HoodieTimeline;
import org.apache.hudi.common.util.CompactionUtils;
import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
import org.apache.hudi.table.HoodieFlinkTable;
import org.apache.hudi.util.CompactionUtil;
import org.apache.hudi.util.StreamerUtil;
import com.beust.jcommander.JCommander;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.operators.ProcessOperator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
/**
* Flink hudi compaction program that can be executed manually.
*/
public class HoodieFlinkCompactor {
protected static final Logger LOG =
LoggerFactory.getLogger(HoodieFlinkCompactor.class);
+ /**
+ * Flink Compaction Config.
+ */
+ private final FlinkCompactionConfig cfg;
+
+ /**
+ * Flink Execution Environment.
+ */
+ private final AsyncCompactionService compactionScheduleService;
+
+ public HoodieFlinkCompactor(FlinkCompactionConfig cfg, Configuration conf,
StreamExecutionEnvironment env) throws Exception {
+this.cfg = cfg;
+this.compactionScheduleService = new AsyncCompactionService(cfg, conf,
env);
+ }
+
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+FlinkCompactionConfig cfg = getFlinkCompactionConfig(args);
+Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+
+new HoodieFlinkCompactor(cfg, conf, env).start();
+ }
+
+ /**
+ * Main method to start compaction service.
+ */
+ public void start() throws Exception {
+if (cfg.serviceMode) {
+ compactionScheduleService.start(null);
+ try {
+compactionScheduleService.waitForShutdown();
+ } catch (Exception e) {
+throw new HoodieException(e.getMessage(), e);
+ } finally {
+LOG.info("Shut down hoodie flink compactor");
+ }
+} else {
+ LOG.info("Hoodie Flink Compactor running only single round");
+ try {
+compactionScheduleService.scheduleCompaction();
+ } catch (Exception e) {
+LOG.error("Got error running delta sync once. Shutting down", e);
+throw e;
+ } finally {
+LOG.info("Shut down hoodie flink compactor");
+ }
+}
+ }
+
+ public static FlinkCompactionConfig getFlinkCompactionConfig(String[] args) {
FlinkCompactionConfig cfg = new FlinkCompactionConfig();
JCommander cmd = new JCommander(cfg, null, args);
if (cfg.help || args.length == 0) {
cmd.usage();
System.exit(1);
}
+return cfg;
+ }
-Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+ // -
+ // Inner Class
+ // -
+
+ /**
+ * Schedule compaction in service.
+ */
+ public static class AsyncCompactionService extends HoodieAsyncService {
+private static final long serialVersionUID = 1L;
+
+/**
+ * Flink Compaction Config.
+ */
+private final FlinkCompactionConfig cfg;
+
+/**
+ * Flink Config.
+ */
+private final Configuration conf;
+
+/**
+ * Meta Client.
+ */
+private final HoodieTableMetaClient metaClient;
+
+/**
+ * Write Client.
+ */
+private final HoodieFlinkWriteClient writeClient;
+
+/**
+ * The hoodie table.
+ */
+private final HoodieFlinkTable table;
+
+/**
+ * Flink Execution Environment.
+ */
+private final StreamExecutionEnvironment env;
+
+/**
+ * Executor Service.
+ */
+private final ExecutorService executor;
+
+public AsyncCompactionService(FlinkCompactionConfig cfg, Configuration
conf, StreamExecutionEnvironment env) throws Exception {
+ this.cfg = cfg;
+ this.conf = conf;
+ this.env
[GitHub] [hudi] mincwang commented on a change in pull request #4254: [HUDI-2547] Schedule Flink compaction in service
mincwang commented on a change in pull request #4254:
URL: https://github.com/apache/hudi/pull/4254#discussion_r770271063
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/compact/HoodieFlinkCompactor.java
##
@@ -18,141 +18,299 @@
package org.apache.hudi.sink.compact;
+import org.apache.hudi.async.HoodieAsyncService;
import org.apache.hudi.avro.model.HoodieCompactionPlan;
import org.apache.hudi.client.HoodieFlinkWriteClient;
import org.apache.hudi.common.table.HoodieTableMetaClient;
import org.apache.hudi.common.table.timeline.HoodieInstant;
import org.apache.hudi.common.table.timeline.HoodieTimeline;
import org.apache.hudi.common.util.CompactionUtils;
import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
import org.apache.hudi.table.HoodieFlinkTable;
import org.apache.hudi.util.CompactionUtil;
import org.apache.hudi.util.StreamerUtil;
import com.beust.jcommander.JCommander;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.operators.ProcessOperator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
/**
* Flink hudi compaction program that can be executed manually.
*/
public class HoodieFlinkCompactor {
protected static final Logger LOG =
LoggerFactory.getLogger(HoodieFlinkCompactor.class);
+ /**
+ * Flink Compaction Config.
+ */
+ private final FlinkCompactionConfig cfg;
+
+ /**
+ * Flink Execution Environment.
+ */
+ private final AsyncCompactionService compactionScheduleService;
+
+ public HoodieFlinkCompactor(FlinkCompactionConfig cfg, Configuration conf,
StreamExecutionEnvironment env) throws Exception {
+this.cfg = cfg;
+this.compactionScheduleService = new AsyncCompactionService(cfg, conf,
env);
+ }
+
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+FlinkCompactionConfig cfg = getFlinkCompactionConfig(args);
+Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+
+new HoodieFlinkCompactor(cfg, conf, env).start();
+ }
+
+ /**
+ * Main method to start compaction service.
+ */
+ public void start() throws Exception {
+if (cfg.serviceMode) {
+ compactionScheduleService.start(null);
+ try {
+compactionScheduleService.waitForShutdown();
+ } catch (Exception e) {
+throw new HoodieException(e.getMessage(), e);
+ } finally {
+LOG.info("Shut down hoodie flink compactor");
+ }
+} else {
+ LOG.info("Hoodie Flink Compactor running only single round");
+ try {
+compactionScheduleService.scheduleCompaction();
+ } catch (Exception e) {
+LOG.error("Got error running delta sync once. Shutting down", e);
+throw e;
+ } finally {
+LOG.info("Shut down hoodie flink compactor");
+ }
+}
+ }
+
+ public static FlinkCompactionConfig getFlinkCompactionConfig(String[] args) {
FlinkCompactionConfig cfg = new FlinkCompactionConfig();
JCommander cmd = new JCommander(cfg, null, args);
if (cfg.help || args.length == 0) {
cmd.usage();
System.exit(1);
}
+return cfg;
+ }
-Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+ // -
+ // Inner Class
+ // -
+
+ /**
+ * Schedule compaction in service.
+ */
+ public static class AsyncCompactionService extends HoodieAsyncService {
+private static final long serialVersionUID = 1L;
+
+/**
+ * Flink Compaction Config.
+ */
+private final FlinkCompactionConfig cfg;
+
+/**
+ * Flink Config.
+ */
+private final Configuration conf;
+
+/**
+ * Meta Client.
+ */
+private final HoodieTableMetaClient metaClient;
+
+/**
+ * Write Client.
+ */
+private final HoodieFlinkWriteClient writeClient;
+
+/**
+ * The hoodie table.
+ */
+private final HoodieFlinkTable table;
+
+/**
+ * Flink Execution Environment.
+ */
+private final StreamExecutionEnvironment env;
+
+/**
+ * Executor Service.
+ */
+private final ExecutorService executor;
+
+public AsyncCompactionService(FlinkCompactionConfig cfg, Configuration
conf, StreamExecutionEnvironment env) throws Exception {
+ this.cfg = cfg;
+ this.conf = conf;
+ this.env =
[GitHub] [hudi] danny0405 commented on a change in pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
danny0405 commented on a change in pull request #4307:
URL: https://github.com/apache/hudi/pull/4307#discussion_r769572485
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java
##
@@ -100,38 +100,33 @@ public void
testHoodieClientBasicMultiWriter(HoodieTableType tableType) throws E
.build()).withAutoCommit(false).withProperties(properties).build();
// Create the first commit
createCommitWithInserts(cfg, getHoodieWriteClient(cfg), "000", "001", 200);
-try {
- ExecutorService executors = Executors.newFixedThreadPool(2);
- SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
- SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
- Future future1 = executors.submit(() -> {
-String newCommitTime = "004";
-int numRecords = 100;
-String commitTimeBetweenPrevAndNew = "002";
-try {
- createCommitWithUpserts(cfg, client1, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
-} catch (Exception e1) {
- assertTrue(e1 instanceof HoodieWriteConflictException);
- throw new RuntimeException(e1);
-}
- });
- Future future2 = executors.submit(() -> {
-String newCommitTime = "005";
-int numRecords = 100;
-String commitTimeBetweenPrevAndNew = "002";
-try {
- createCommitWithUpserts(cfg, client2, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
-} catch (Exception e2) {
- assertTrue(e2 instanceof HoodieWriteConflictException);
- throw new RuntimeException(e2);
-}
- });
- future1.get();
- future2.get();
- fail("Should not reach here, this means concurrent writes were handled
incorrectly");
-} catch (Exception e) {
- // Expected to fail due to overlapping commits
-}
+ExecutorService executors = Executors.newFixedThreadPool(2);
+SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+Future future1 = executors.submit(() -> {
+ String newCommitTime = "004";
+ int numRecords = 100;
+ String commitTimeBetweenPrevAndNew = "002";
+ try {
+createCommitWithUpserts(cfg, client1, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
+ } catch (Exception e1) {
+assertTrue(e1 instanceof HoodieWriteConflictException);
+throw new RuntimeException(e1);
+ }
+});
+Future future2 = executors.submit(() -> {
+ String newCommitTime = "005";
+ int numRecords = 100;
+ String commitTimeBetweenPrevAndNew = "002";
+ try {
+createCommitWithUpserts(cfg, client2, "002",
commitTimeBetweenPrevAndNew, newCommitTime, numRecords);
+ } catch (Exception e2) {
+assertTrue(e2 instanceof HoodieWriteConflictException);
+throw new RuntimeException(e2);
+ }
+});
+future1.get();
+future2.get();
Review comment:
The conflict can now resolve correctly with the view refreshed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot commented on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995493823 ## CI report: * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) * 70503ca127d5638bf49440d630ae90de41712d0f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot removed a comment on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995492550 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) * 70503ca127d5638bf49440d630ae90de41712d0f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot commented on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995492550 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) * 70503ca127d5638bf49440d630ae90de41712d0f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot removed a comment on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995477399 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995489753 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 1a390d804e170c0fd42014211db65fd44a3d054e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4363) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995438872 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 00df74e3e4f6039008cd26b789218b48e6ea0d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4349) * 1a390d804e170c0fd42014211db65fd44a3d054e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4363) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4301: [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir
hudi-bot commented on pull request #4301: URL: https://github.com/apache/hudi/pull/4301#issuecomment-995489672 ## CI report: * 64ea4442ad179875fe2dbaa7c6a001b4000b4524 UNKNOWN * b40b14614bd79ff3834efa30de3273d3ec84264f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4348) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4357) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4368) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4301: [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir
hudi-bot removed a comment on pull request #4301: URL: https://github.com/apache/hudi/pull/4301#issuecomment-995454806 ## CI report: * 64ea4442ad179875fe2dbaa7c6a001b4000b4524 UNKNOWN * b40b14614bd79ff3834efa30de3273d3ec84264f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4348) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4357) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4301: [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir
zhangyue19921010 commented on pull request #4301: URL: https://github.com/apache/hudi/pull/4301#issuecomment-995488725 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot removed a comment on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995431681 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot commented on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995488504 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot removed a comment on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995461338 ## CI report: * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN * 6e8aba03f04cc266b5e14f7f991770f1e024 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot commented on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995484875 ## CI report: * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN * 6e8aba03f04cc266b5e14f7f991770f1e024 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4367) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4306: [HUDI-3014] add table option to set utc timezone
hudi-bot commented on pull request #4306: URL: https://github.com/apache/hudi/pull/4306#issuecomment-995478697 ## CI report: * a39258ca69c6302da42cdb1fe1a0794676480952 UNKNOWN * a1ba1e2c81b74948a93589c3192ab24ef320107b UNKNOWN * c347bb78b3c799dce34db7a00c7f6a07c95ec777 UNKNOWN * d6a0ac9027bf12362b56729a86e9755dbe1c21db UNKNOWN * 4fd974b6b45e337f75bfaa9e6d54dc7e82cf1473 UNKNOWN * 1e7fadfb38c1e412e0cb13cd977599ea60bd16b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4360) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4306: [HUDI-3014] add table option to set utc timezone
hudi-bot removed a comment on pull request #4306: URL: https://github.com/apache/hudi/pull/4306#issuecomment-995431624 ## CI report: * a39258ca69c6302da42cdb1fe1a0794676480952 UNKNOWN * a1ba1e2c81b74948a93589c3192ab24ef320107b UNKNOWN * c347bb78b3c799dce34db7a00c7f6a07c95ec777 UNKNOWN * d6a0ac9027bf12362b56729a86e9755dbe1c21db UNKNOWN * 4fd974b6b45e337f75bfaa9e6d54dc7e82cf1473 UNKNOWN * b6be10b65ceac5c3b975469ede0c166b6b108202 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4354) * 1e7fadfb38c1e412e0cb13cd977599ea60bd16b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4360) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a change in pull request #4334: [HUDI-3011] Adding ability to read entire data with HoodieIncrSource with empty checkpoint
harsh1231 commented on a change in pull request #4334:
URL: https://github.com/apache/hudi/pull/4334#discussion_r770255320
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.IncrSourceHelper;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class TestHoodieIncrSource extends HoodieClientTestHarness {
+
+ @BeforeEach
+ public void setUp() throws IOException {
+initResources();
+ }
+
+ @AfterEach
+ public void tearDown() throws IOException {
+cleanupResources();
+ }
+
+ @Test
+ public void testHoodieIncrSource() throws IOException {
Review comment:
should we add parameterized test for configs
`archive.min_commits=2
clean.retain_commits=2`
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot commented on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995477399 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4366) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot removed a comment on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995476027 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot removed a comment on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-989644500 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4254: [HUDI-2547] Schedule Flink compaction in service
hudi-bot commented on pull request #4254: URL: https://github.com/apache/hudi/pull/4254#issuecomment-995476027 ## CI report: * 2d9c2c349db10c753557585bde1f4250b6fa14fe Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4120) * 9fc8ed74086ee6d5f516f209dca518a03986cc80 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995474370 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4365) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot removed a comment on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995471339 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
xiarixiaoyao commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995473588 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995471339 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot removed a comment on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995469911 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot removed a comment on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995415718 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
hudi-bot commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995469911 ## CI report: * cd1d155561a5b29061ba7aa27b3801377ef2d4b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4350) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #4253: [HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, when using bulkinsert to insert data which contains decimalType
xiarixiaoyao commented on pull request #4253: URL: https://github.com/apache/hudi/pull/4253#issuecomment-995469819 @hudi-bot run azur -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot commented on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995463456 ## CI report: * a157c751a957cfbc82cf699b479c376e93549ce4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4359) * bdcac7c5442d3f79cd75fc02e13d606e4bd6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4364) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot removed a comment on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995462363 ## CI report: * 09a4a81b8d3dd50f0d714492f6dae827a2df1f2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4356) * a157c751a957cfbc82cf699b479c376e93549ce4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4359) * bdcac7c5442d3f79cd75fc02e13d606e4bd6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot commented on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995462363 ## CI report: * 09a4a81b8d3dd50f0d714492f6dae827a2df1f2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4356) * a157c751a957cfbc82cf699b479c376e93549ce4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4359) * bdcac7c5442d3f79cd75fc02e13d606e4bd6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4307: [HUDI-3015] Implement #reset and #sync for metadata filesystem view
hudi-bot removed a comment on pull request #4307: URL: https://github.com/apache/hudi/pull/4307#issuecomment-995415781 ## CI report: * 09a4a81b8d3dd50f0d714492f6dae827a2df1f2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4356) * a157c751a957cfbc82cf699b479c376e93549ce4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4359) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot removed a comment on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995459038 ## CI report: * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot commented on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995461338 ## CI report: * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN * 6e8aba03f04cc266b5e14f7f991770f1e024 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot removed a comment on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995457987 ## CI report: * 8d4d953b74870a41c20fbac81e8c9bd4f86cd2e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4358) * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot commented on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995459038 ## CI report: * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot removed a comment on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995431667 ## CI report: * 8d4d953b74870a41c20fbac81e8c9bd4f86cd2e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4358) * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot commented on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995457987 ## CI report: * 8d4d953b74870a41c20fbac81e8c9bd4f86cd2e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4358) * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) * 6d39c38f416f2e0f8249f8bcff2434c07fe929aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4301: [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir
hudi-bot removed a comment on pull request #4301: URL: https://github.com/apache/hudi/pull/4301#issuecomment-995395605 ## CI report: * 64ea4442ad179875fe2dbaa7c6a001b4000b4524 UNKNOWN * b40b14614bd79ff3834efa30de3273d3ec84264f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4348) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4357) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4301: [Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir
hudi-bot commented on pull request #4301: URL: https://github.com/apache/hudi/pull/4301#issuecomment-995454806 ## CI report: * 64ea4442ad179875fe2dbaa7c6a001b4000b4524 UNKNOWN * b40b14614bd79ff3834efa30de3273d3ec84264f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4348) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4357) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4317: [WIP][Minor] Debug TestHoodieDeltaStreamerWithMultiWriter
hudi-bot removed a comment on pull request #4317: URL: https://github.com/apache/hudi/pull/4317#issuecomment-995385150 ## CI report: * 69c02777482f14cdde476554f29de83a2518e466 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4315) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4322) * 43917292ef69d1892d822e358b16e57315f172f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4355) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4317: [WIP][Minor] Debug TestHoodieDeltaStreamerWithMultiWriter
hudi-bot commented on pull request #4317: URL: https://github.com/apache/hudi/pull/4317#issuecomment-995443126 ## CI report: * 43917292ef69d1892d822e358b16e57315f172f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4355) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995438872 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 00df74e3e4f6039008cd26b789218b48e6ea0d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4349) * 1a390d804e170c0fd42014211db65fd44a3d054e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4363) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995426398 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 00df74e3e4f6039008cd26b789218b48e6ea0d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4349) * 1a390d804e170c0fd42014211db65fd44a3d054e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4308: [HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields
hudi-bot commented on pull request #4308: URL: https://github.com/apache/hudi/pull/4308#issuecomment-995436122 ## CI report: * 1be190032b1b6da5c07fc81bb2fdd42903a48185 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4338) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4353) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4308: [HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields
hudi-bot removed a comment on pull request #4308: URL: https://github.com/apache/hudi/pull/4308#issuecomment-995368807 ## CI report: * 1be190032b1b6da5c07fc81bb2fdd42903a48185 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4338) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4353) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3034: - Fix Version/s: 0.11.0 > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot removed a comment on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995430589 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot commented on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995431667 ## CI report: * 8d4d953b74870a41c20fbac81e8c9bd4f86cd2e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4358) * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4361) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot commented on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995431681 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4362) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4336: [HUDI-3032] Do not clean the log files right after compaction for met…
hudi-bot removed a comment on pull request #4336: URL: https://github.com/apache/hudi/pull/4336#issuecomment-995411297 ## CI report: * 8d4d953b74870a41c20fbac81e8c9bd4f86cd2e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4358) * 8f454b734d8848aee4cb6883999a658a7f007fc2 UNKNOWN * d47b689915b04e25b3cff8249478284b2522b44d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4306: [HUDI-3014] add table option to set utc timezone
hudi-bot commented on pull request #4306: URL: https://github.com/apache/hudi/pull/4306#issuecomment-995431624 ## CI report: * a39258ca69c6302da42cdb1fe1a0794676480952 UNKNOWN * a1ba1e2c81b74948a93589c3192ab24ef320107b UNKNOWN * c347bb78b3c799dce34db7a00c7f6a07c95ec777 UNKNOWN * d6a0ac9027bf12362b56729a86e9755dbe1c21db UNKNOWN * 4fd974b6b45e337f75bfaa9e6d54dc7e82cf1473 UNKNOWN * b6be10b65ceac5c3b975469ede0c166b6b108202 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4354) * 1e7fadfb38c1e412e0cb13cd977599ea60bd16b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4360) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4306: [HUDI-3014] add table option to set utc timezone
hudi-bot removed a comment on pull request #4306: URL: https://github.com/apache/hudi/pull/4306#issuecomment-995411249 ## CI report: * a39258ca69c6302da42cdb1fe1a0794676480952 UNKNOWN * a1ba1e2c81b74948a93589c3192ab24ef320107b UNKNOWN * c347bb78b3c799dce34db7a00c7f6a07c95ec777 UNKNOWN * d6a0ac9027bf12362b56729a86e9755dbe1c21db UNKNOWN * 4fd974b6b45e337f75bfaa9e6d54dc7e82cf1473 UNKNOWN * b6be10b65ceac5c3b975469ede0c166b6b108202 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4354) * 1e7fadfb38c1e412e0cb13cd977599ea60bd16b9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4337: [HUDI-3034] Add option for parallel mvn install
hudi-bot commented on pull request #4337: URL: https://github.com/apache/hudi/pull/4337#issuecomment-995430589 ## CI report: * 8f36ab59b05bed9cd29a60f73d611026cefd2588 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3034: - Labels: pull-request-available (was: ) > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Raymond Xu >Priority: Major > Labels: pull-request-available > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4298: [HUDI-3001]clean up the marker directory when finish bootstrap operat…
hudi-bot removed a comment on pull request #4298: URL: https://github.com/apache/hudi/pull/4298#issuecomment-995369926 ## CI report: * 8dd35055d6206b3a916775f9685b41f11d1c72f0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4237) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4238) * 8d76af397158cc8c19f5deab92de35f8437ec8ef Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4352) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4298: [HUDI-3001]clean up the marker directory when finish bootstrap operat…
hudi-bot commented on pull request #4298: URL: https://github.com/apache/hudi/pull/4298#issuecomment-995428425 ## CI report: * 8d76af397158cc8c19f5deab92de35f8437ec8ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4352) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3035) Unify Parquet writers
[ https://issues.apache.org/jira/browse/HUDI-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3035: -- Priority: Blocker (was: Major) > Unify Parquet writers > - > > Key: HUDI-3035 > URL: https://issues.apache.org/jira/browse/HUDI-3035 > Project: Apache Hudi > Issue Type: Bug >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Blocker > > Currently we have at least 3 implementations of the ParquetWriters (which is > 3x more than we actually need): > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowParquetWriter.java] > > Implementations (while identical in principle) have diverged, essentially > living their own lifecycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3035) Unify Parquet writers
[ https://issues.apache.org/jira/browse/HUDI-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3035: -- Fix Version/s: 0.11.0 > Unify Parquet writers > - > > Key: HUDI-3035 > URL: https://issues.apache.org/jira/browse/HUDI-3035 > Project: Apache Hudi > Issue Type: Bug >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Blocker > Fix For: 0.11.0 > > > Currently we have at least 3 implementations of the ParquetWriters (which is > 3x more than we actually need): > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowParquetWriter.java] > > Implementations (while identical in principle) have diverged, essentially > living their own lifecycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3035) Unify Parquet writers
[ https://issues.apache.org/jira/browse/HUDI-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-3035: - Assignee: Alexey Kudinkin > Unify Parquet writers > - > > Key: HUDI-3035 > URL: https://issues.apache.org/jira/browse/HUDI-3035 > Project: Apache Hudi > Issue Type: Bug >Reporter: Alexey Kudinkin >Assignee: Alexey Kudinkin >Priority: Major > > Currently we have at least 3 implementations of the ParquetWriters (which is > 3x more than we actually need): > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataParquetWriter.java] > [https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowParquetWriter.java] > > Implementations (while identical in principle) have diverged, essentially > living their own lifecycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3035) Unify Parquet writers
Alexey Kudinkin created HUDI-3035: - Summary: Unify Parquet writers Key: HUDI-3035 URL: https://issues.apache.org/jira/browse/HUDI-3035 Project: Apache Hudi Issue Type: Bug Reporter: Alexey Kudinkin Currently we have at least 3 implementations of the ParquetWriters (which is 3x more than we actually need): [https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieParquetWriter.java] [https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataParquetWriter.java] [https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieInternalRowParquetWriter.java] Implementations (while identical in principle) have diverged, essentially living their own lifecycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995426398 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 00df74e3e4f6039008cd26b789218b48e6ea0d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4349) * 1a390d804e170c0fd42014211db65fd44a3d054e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-995404984 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * 00df74e3e4f6039008cd26b789218b48e6ea0d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4349) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-3034: --- Assignee: Raymond Xu (was: Raymond Xu) > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Raymond Xu >Priority: Major > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3034: Description: We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. Locally on MacBook, this reduces the build time by 3-4 minutes. > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Priority: Major > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-3034: --- Assignee: Raymond Xu (was: Ethan Guo) > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Raymond Xu >Priority: Major > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HUDI-3034) Enable maven parallel build in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-3034: --- Assignee: Ethan Guo > Enable maven parallel build in Azure CI > --- > > Key: HUDI-3034 > URL: https://issues.apache.org/jira/browse/HUDI-3034 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > > We may try `mvn -T 1C clean package -DskipTests` in CI to speed up the build. > Locally on MacBook, this reduces the build time by 3-4 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3034) Enable maven parallel build in Azure CI
Ethan Guo created HUDI-3034: --- Summary: Enable maven parallel build in Azure CI Key: HUDI-3034 URL: https://issues.apache.org/jira/browse/HUDI-3034 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3033) Rebalance tests in Azure CI
[
https://issues.apache.org/jira/browse/HUDI-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3033:
Component/s: Testing
> Rebalance tests in Azure CI
> ---
>
> Key: HUDI-3033
> URL: https://issues.apache.org/jira/browse/HUDI-3033
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Testing
>Reporter: Ethan Guo
>Priority: Major
> Attachments: Screen Shot 2021-12-15 at 20.07.20.png
>
>
> We may want to rebalance the tests to avoid timeout of 1 hour. Currently,
> "{{{}IT"{}}} and "{{{}UT FT clients & cli & utilities & sync/hive-sync"{}}}
> take less than 40 mins while "{{{}UT FT other modules"{}}} almost takes an
> hour
> !Screen Shot 2021-12-15 at 20.07.20.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HUDI-3033) Rebalance tests in Azure CI
[
https://issues.apache.org/jira/browse/HUDI-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3033:
Description:
We may want to rebalance the tests to avoid timeout of 1 hour. Currently,
"{{{}IT"{}}} and "{{{}UT FT clients & cli & utilities & sync/hive-sync"{}}}
take less than 40 mins while "{{{}UT FT other modules"{}}} almost takes an hour
!Screen Shot 2021-12-15 at 20.07.20.png!
> Rebalance tests in Azure CI
> ---
>
> Key: HUDI-3033
> URL: https://issues.apache.org/jira/browse/HUDI-3033
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: Ethan Guo
>Priority: Major
> Attachments: Screen Shot 2021-12-15 at 20.07.20.png
>
>
> We may want to rebalance the tests to avoid timeout of 1 hour. Currently,
> "{{{}IT"{}}} and "{{{}UT FT clients & cli & utilities & sync/hive-sync"{}}}
> take less than 40 mins while "{{{}UT FT other modules"{}}} almost takes an
> hour
> !Screen Shot 2021-12-15 at 20.07.20.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HUDI-3033) Rebalance tests in Azure CI
[ https://issues.apache.org/jira/browse/HUDI-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3033: Attachment: Screen Shot 2021-12-15 at 20.07.20.png > Rebalance tests in Azure CI > --- > > Key: HUDI-3033 > URL: https://issues.apache.org/jira/browse/HUDI-3033 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Priority: Major > Attachments: Screen Shot 2021-12-15 at 20.07.20.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3033) Rebalance tests in Azure CI
Ethan Guo created HUDI-3033: --- Summary: Rebalance tests in Azure CI Key: HUDI-3033 URL: https://issues.apache.org/jira/browse/HUDI-3033 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.1#820001)
