Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859720659 ## CI report: * 50121f9c130e543642c114d8b96b72e88273d730 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7213] When using wrong tabe.type value in hudi catalog happends npe [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10300: URL: https://github.com/apache/hudi/pull/10300#issuecomment-1859720455 ## CI report: * 9194aa91d029ea3f4c701390bae75bf625b782c1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7204] Improve fg reader test framework [hudi]

2023-12-17 Thread via GitHub
linliu-code commented on code in PR #10324: URL: https://github.com/apache/hudi/pull/10324#discussion_r1429614415 ## hudi-common/src/main/java/org/apache/hudi/common/model/EmptyHoodieRecordPayload.java: ## @@ -18,35 +18,69 @@ package org.apache.hudi.common.model; +import or

Re: [PR] [HUDI-7204] Improve fg reader test framework [hudi]

2023-12-17 Thread via GitHub
linliu-code commented on code in PR #10324: URL: https://github.com/apache/hudi/pull/10324#discussion_r1429614415 ## hudi-common/src/main/java/org/apache/hudi/common/model/EmptyHoodieRecordPayload.java: ## @@ -18,35 +18,69 @@ package org.apache.hudi.common.model; +import or

(hudi) branch release-0.14.0-siva-0.14.1 updated (68f37119ad1 -> d1a43dc3694)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git from 68f37119ad1 [HUDI-7199] Optimize contains impl with HoodieDefaultTimeline (#10284) add 7cef

(hudi) 06/07: [HUDI-7183] Fix static insert overwrite partitions issue (#10254)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit e4fd81f1b2549baf5f51211ec11d22718e05b9c1 Author: Wechar Yu AuthorDate: Sun Dec 17 11:32:30 2023 +08

(hudi) 05/07: [MINOR] NPE fix while adding projection field & added its test cases (#10313)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit bd59a866ea8c19f2ab99751f6a82870832210cca Author: Prathit malik <53890994+prathi...@users.noreply.git

(hudi) 02/07: [HUDI-7040] Handle dropping of partition columns in BulkInsertDataInternalWriterHelper::write(...) (#10272)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 790903712ecd5ee65850673141227698ea0ced26 Author: bhat-vinay <152183592+bhat-vi...@users.noreply.gith

(hudi) 01/07: [HUDI-7171] Fix 'show partitions' not display rewritten partitions (#10242)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 75d06238e3daf640bc71af86ddc6559f1d15164f Author: Wechar Yu AuthorDate: Mon Dec 11 11:43:16 2023 +08

(hudi) 03/07: [HUDI-7210] In CleanFunction#open, triggers the cleaning under option 'clean.async.enabled' (#10298)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 080d2f9f08ff95646ec13864b3eb416cf94d817b Author: Danny Chan AuthorDate: Tue Dec 12 09:50:33 2023 +0

(hudi) 07/07: [HUDI-7223] Cleaner KEEP_LATEST_BY_HOURS should retain latest commit before earliest commit to retain (#10307)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit d1a43dc3694b6a51aa830fe2b78340503c6909b5 Author: Tim Brown AuthorDate: Sun Dec 17 12:21:15 2023 -06

(hudi) 04/07: [HUDI-7132] Data may be lost for flink task failure (#10312)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.14.0-siva-0.14.1 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 549a80bf865012e707abd045597fb9ca5a0a12a4 Author: Danny Chan AuthorDate: Tue Dec 12 18:26:03 2023 +0

Re: [PR] [HUDI-7204] Improve fg reader test framework [hudi]

2023-12-17 Thread via GitHub
codope commented on code in PR #10324: URL: https://github.com/apache/hudi/pull/10324#discussion_r1429604793 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java: ## @@ -179,6 +180,9 @@ private Schema generateRequiredSchema() { for (Stri

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859673961 ## CI report: * 8d999c7e7946d2dc3d05e8bd7ebf53d5d5e8a57a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7236] Allow MIT to change partition path when using global index [hudi]

2023-12-17 Thread via GitHub
xushiyan commented on code in PR #10337: URL: https://github.com/apache/hudi/pull/10337#discussion_r1429556607 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java: ## @@ -251,9 +305,14 @@ private static Option> mergeIncomingWithExistingR

[I] [SUPPORT] maintain backward compatibility for HeaderMetadataType? [hudi]

2023-12-17 Thread via GitHub
waitingF opened a new issue, #10348: URL: https://github.com/apache/hudi/issues/10348 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subsc

Re: [I] Querying Hudi Table Created With Version 0.12.3 Not Working on Trino 430 [hudi]

2023-12-17 Thread via GitHub
Amar1404 commented on issue #10228: URL: https://github.com/apache/hudi/issues/10228#issuecomment-1859552349 Hi @ad1happy2go - I have found the issue in Syncing of Table in Catalog, Since I am using GLue Catalog. But when I tried creating a table using the HudiSyncTool class the table is no

Re: [I] The Schema Evolution Not working For Hudi 0.12.3 [hudi]

2023-12-17 Thread via GitHub
Amar1404 commented on issue #10309: URL: https://github.com/apache/hudi/issues/10309#issuecomment-1859549331 Hi @ad1happy2go - Please find below the configurations "hoodie.schema.on.read.enable": "true" "hoodie.cleaner.commits.retained": "3", "hoodie.datasource.

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859518478 ## CI report: * 1c4c39a20e16f110f6d45e3315287e00b2b1c4a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7213] When using wrong tabe.type value in hudi catalog happends npe [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10300: URL: https://github.com/apache/hudi/pull/10300#issuecomment-1859518369 ## CI report: * f7b85354a801289e20e1c8ac2b803c2ee9d5ec6c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859513164 ## CI report: * 1c4c39a20e16f110f6d45e3315287e00b2b1c4a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859513120 ## CI report: * 16f3ba28795eca2e871a31fbbc68ee0edc4721b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7213] When using wrong tabe.type value in hudi catalog happends npe [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10300: URL: https://github.com/apache/hudi/pull/10300#issuecomment-1859513050 ## CI report: * f7b85354a801289e20e1c8ac2b803c2ee9d5ec6c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859508766 ## CI report: * 16f3ba28795eca2e871a31fbbc68ee0edc4721b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [I] [SUPPORT] The stage "Building workload profile" [hudi]

2023-12-17 Thread via GitHub
zyclove closed issue #10346: [SUPPORT] The stage "Building workload profile" URL: https://github.com/apache/hudi/issues/10346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] [SUPPORT] Flink to hudi ,but the partitioned table query is not expected [hudi]

2023-12-17 Thread via GitHub
young138120 commented on issue #10327: URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859507966 > so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the

Re: [PR] [HUDI-6979][RFC-76] support event time based compaction strategy [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10266: URL: https://github.com/apache/hudi/pull/10266#discussion_r1429454087 ## rfc/rfc-76/rfc-76.md: ## @@ -0,0 +1,143 @@ + +# RFC-[76]: [support EventTimeBasedCompactionStrategy and metric for RO table] + +## Proposers + +- @waitingF + +## Ap

Re: [I] [SUPPORT] Flink to hudi ,but the partitioned table query is not expected [hudi]

2023-12-17 Thread via GitHub
young138120 commented on issue #10327: URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859503906 > so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429453468 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint.

Re: [PR] [HUDI-6979][RFC-76] support event time based compaction strategy [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10266: URL: https://github.com/apache/hudi/pull/10266#discussion_r1429452701 ## rfc/rfc-76/rfc-76.md: ## @@ -0,0 +1,143 @@ + +# RFC-[76]: [support EventTimeBasedCompactionStrategy and metric for RO table] + +## Proposers + +- @waitingF + +## Ap

[I] [SUPPORT] Can the process of "Building workload profile" be optimized? Too slow, half the time is spent in this stage [hudi]

2023-12-17 Thread via GitHub
zyclove opened a new issue, #10347: URL: https://github.com/apache/hudi/issues/10347 **Describe the problem you faced** ![image](https://github.com/apache/hudi/assets/15028279/859c528d-ab93-4823-8213-a9697c05f407) ``` private Pair, WorkloadStat> buildProfile(HoodieData>

Re: [PR] [HUDI-6979][RFC-76] support event time based compaction strategy [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10266: URL: https://github.com/apache/hudi/pull/10266#discussion_r1429450376 ## rfc/rfc-76/rfc-76.md: ## @@ -0,0 +1,143 @@ + +# RFC-[76]: [support EventTimeBasedCompactionStrategy and metric for RO table] + +## Proposers + +- @waitingF + +## Ap

[I] [SUPPORT] The stage "Building workload profile" [hudi]

2023-12-17 Thread via GitHub
zyclove opened a new issue, #10346: URL: https://github.com/apache/hudi/issues/10346 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10340: URL: https://github.com/apache/hudi/pull/10340#discussion_r1429418791 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatReader.java: ## @@ -104,8 +90,6 @@ public boolean hasNext() { // First close prev

Re: [I] [SUPPORT] Flink to hudi ,but the partitioned table query is not expected [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on issue #10327: URL: https://github.com/apache/hudi/issues/10327#issuecomment-1859489273 so you are not syncing any meta into hive or something, and you are using the Flink in-memory catalog, did you query the table directly from Hudi with spark (i.e. not query the hudi

Re: [PR] [HUDI-7213] When using wrong tabe.type value in hudi catalog happends npe [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on PR #10300: URL: https://github.com/apache/hudi/pull/10300#issuecomment-1859486382 There are still test failures: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=21528&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bbbd7bcc-ae73-56b8-8

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10344: URL: https://github.com/apache/hudi/pull/10344#discussion_r1429440167 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -217,11 +219,11 @@ public R put(T key, R value) { this.cur

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10344: URL: https://github.com/apache/hudi/pull/10344#discussion_r1429438784 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -217,11 +219,11 @@ public R put(T key, R value) { this.curr

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10345: URL: https://github.com/apache/hudi/pull/10345#discussion_r1429435343 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/LazyFileIterable.java: ## @@ -128,7 +132,10 @@ private void closeHandle() { } private v

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10340: URL: https://github.com/apache/hudi/pull/10340#discussion_r1429433785 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatReader.java: ## @@ -104,8 +90,6 @@ public boolean hasNext() { // First close pre

Re: [PR] [HUDI-7208] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10297: URL: https://github.com/apache/hudi/pull/10297#discussion_r1429433856 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -294,6 +295,9 @@ private Option prepareRecord(HoodieRecord hoodieRec

Re: [PR] Incoming batch schema is not compatible with the table's one #9980 [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10308: URL: https://github.com/apache/hudi/pull/10308#discussion_r1429432577 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala: ## @@ -1092,6 +1092,10 @@ class HoodieSparkSqlWriterInternal {

Re: [I] [SUPPORT] Inconsistent Checkpoint Size in Flink Applications with MoR [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on issue #10329: URL: https://github.com/apache/hudi/issues/10329#issuecomment-1859477654 It's the mapping from hoodie record key to location, for a location it is comprised by a partition path and file group id. -- This is an automated message from the Apache Git Serv

Re: [PR] [HUDI-7204] Improve fg reader test framework [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10324: URL: https://github.com/apache/hudi/pull/10324#discussion_r1429429815 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java: ## @@ -179,6 +180,9 @@ private Schema generateRequiredSchema() { for (S

Re: [PR] [HUDI-7204] Improve fg reader test framework [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10324: URL: https://github.com/apache/hudi/pull/10324#discussion_r1429428502 ## hudi-common/src/main/java/org/apache/hudi/common/model/EmptyHoodieRecordPayload.java: ## @@ -18,35 +18,69 @@ package org.apache.hudi.common.model; +import org.

Re: [I] Single table multiple write conflicts! [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on issue #10333: URL: https://github.com/apache/hudi/issues/10333#issuecomment-1859472658 You can have a try of our new NB-CC concurrency mode, here is what it looks like with Flink options: ```sql -- NB-CC demo CREATE TABLE sourceT ( uuid varchar(20

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10340: URL: https://github.com/apache/hudi/pull/10340#discussion_r1429418791 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatReader.java: ## @@ -104,8 +90,6 @@ public boolean hasNext() { // First close prev

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
danny0405 commented on code in PR #10340: URL: https://github.com/apache/hudi/pull/10340#discussion_r1429418791 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatReader.java: ## @@ -104,8 +90,6 @@ public boolean hasNext() { // First close prev

Re: [PR] [HUDI-7190] Fix nested columns vectorized read for spark33+ legacy formats [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10265: URL: https://github.com/apache/hudi/pull/10265#issuecomment-1859468760 ## CI report: * ba0c6c35920e115cfc8d14726c6047de6881c298 UNKNOWN * 939389e022837e408c6f3efef2c5e9c7219455af Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429412658 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint. */

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429412658 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint. */

Re: [PR] [WIP] Log record reader memory investigate [hudi]

2023-12-17 Thread via GitHub
nsivabalan closed pull request #10338: [WIP] Log record reader memory investigate URL: https://github.com/apache/hudi/pull/10338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [HUDI-7190] Fix nested columns vectorized read for spark33+ legacy formats [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10265: URL: https://github.com/apache/hudi/pull/10265#issuecomment-1859431778 ## CI report: * ba0c6c35920e115cfc8d14726c6047de6881c298 UNKNOWN * 939389e022837e408c6f3efef2c5e9c7219455af Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7190] Fix nested columns vectorized read for spark33+ legacy formats [hudi]

2023-12-17 Thread via GitHub
stream2000 commented on PR #10265: URL: https://github.com/apache/hudi/pull/10265#issuecomment-1859430055 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859364591 ## CI report: * b6ebd945a7d980ee5044edcd5b4f8390f6bd8adf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859294990 ## CI report: * 1c4c39a20e16f110f6d45e3315287e00b2b1c4a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859284996 ## CI report: * ef6698a802f5ac6752b806f5acc7c15a57ed2f36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859283257 ## CI report: * ef6698a802f5ac6752b806f5acc7c15a57ed2f36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10342: URL: https://github.com/apache/hudi/pull/10342#issuecomment-1859281330 ## CI report: * cb62ad9bed32bf3acc6f8227e5e824cb73e8f0e4 UNKNOWN * 3f0829263192c35ae636e707106a97d7c0142ff7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859265143 ## CI report: * 16f3ba28795eca2e871a31fbbc68ee0edc4721b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10342: URL: https://github.com/apache/hudi/pull/10342#issuecomment-1859254794 ## CI report: * cb62ad9bed32bf3acc6f8227e5e824cb73e8f0e4 UNKNOWN * 7c3ea778cc509ea71d9b837d4b228bb07abf18b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859253158 ## CI report: * 88c31b51d63041996222b0af3710f6612934451d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10342: URL: https://github.com/apache/hudi/pull/10342#issuecomment-1859253145 ## CI report: * cb62ad9bed32bf3acc6f8227e5e824cb73e8f0e4 UNKNOWN * 7c3ea778cc509ea71d9b837d4b228bb07abf18b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429243393 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -1010,8 +1008,9 @@ private void reInitWriteClient(Schema sourceSche

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859251453 ## CI report: * 88c31b51d63041996222b0af3710f6612934451d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859251440 ## CI report: * 0f37535e445fcfe7294bd696c7ecbae3835a1901 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859249897 ## CI report: * ef6698a802f5ac6752b806f5acc7c15a57ed2f36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429241599 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint.

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429238447 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint. */

(hudi) branch master updated: [HUDI-7223] Cleaner KEEP_LATEST_BY_HOURS should retain latest commit before earliest commit to retain (#10307)

2023-12-17 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 50f0d9f3bae [HUDI-7223] Cleaner KEEP_LATEST_BY_

Re: [PR] [HUDI-7223] Cleaner KEEP_LATEST_BY_HOURS should retain latest commit before earliest commit to retain [hudi]

2023-12-17 Thread via GitHub
nsivabalan merged PR #10307: URL: https://github.com/apache/hudi/pull/10307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apa

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader (0.x branch) [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10339: URL: https://github.com/apache/hudi/pull/10339#issuecomment-1859235750 ## CI report: * 1a2b5d4c84b5fae678a8e0f0e4edcac870cb6e8f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown commented on code in PR #10344: URL: https://github.com/apache/hudi/pull/10344#discussion_r1429212571 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -217,11 +219,11 @@ public R put(T key, R value) {

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader (0.x branch) [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10339: URL: https://github.com/apache/hudi/pull/10339#issuecomment-1859222894 ## CI report: * a4a30405db038272466adea3ee51c2923fcd944e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859222907 ## CI report: * 53875b1c03bc5d2c761e57c998a3fb1c2e038b01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10345: URL: https://github.com/apache/hudi/pull/10345#discussion_r1429207366 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java: ## @@ -112,16 +114,25 @@ private List addAdditionalMetricsExporters(HoodieWrite

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10344: URL: https://github.com/apache/hudi/pull/10344#discussion_r1429206050 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -217,11 +219,11 @@ public R put(T key, R value) { this.cur

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429204766 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint.

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859213210 ## CI report: * ef6698a802f5ac6752b806f5acc7c15a57ed2f36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859213189 ## CI report: * 53875b1c03bc5d2c761e57c998a3fb1c2e038b01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7237] Hudi Streamer: Handle edge case with null schema, minor cleanups [hudi]

2023-12-17 Thread via GitHub
nsivabalan commented on code in PR #10342: URL: https://github.com/apache/hudi/pull/10342#discussion_r1429201396 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -998,9 +996,9 @@ public void runMetaSync() { * this constraint. */

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader (0.x branch) [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10339: URL: https://github.com/apache/hudi/pull/10339#issuecomment-1859213165 ## CI report: * a4a30405db038272466adea3ee51c2923fcd944e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10345: URL: https://github.com/apache/hudi/pull/10345#issuecomment-1859211299 ## CI report: * ef6698a802f5ac6752b806f5acc7c15a57ed2f36 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859211222 ## CI report: * 53875b1c03bc5d2c761e57c998a3fb1c2e038b01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7228] Fix eager closure of log reader input streams with log record reader [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10340: URL: https://github.com/apache/hudi/pull/10340#issuecomment-1859209237 ## CI report: * 53875b1c03bc5d2c761e57c998a3fb1c2e038b01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

[PR] [MINOR] Avoid resource leaks [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown opened a new pull request, #10345: URL: https://github.com/apache/hudi/pull/10345 ### Change Logs This PR aims to track down more places there may be leaks in our tests while also cleaning up a couple of spots that were easier to track down. - Added logs to

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859207566 ## CI report: * 88c31b51d63041996222b0af3710f6612934451d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859197263 ## CI report: * 88c31b51d63041996222b0af3710f6612934451d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21

Re: [PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
hudi-bot commented on PR #10344: URL: https://github.com/apache/hudi/pull/10344#issuecomment-1859195374 ## CI report: * 88c31b51d63041996222b0af3710f6612934451d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [I] [SUPPORT] Spark planner choose broadcast hash join for large HUDI data source [hudi]

2023-12-17 Thread via GitHub
beyond1920 commented on issue #10343: URL: https://github.com/apache/hudi/issues/10343#issuecomment-1859191229 @xuzifu666 @codope Please help me confirm whether my analysis of this issue is correct. Is the `FileIndex#sizeInBytes` better to return the `Long.MAX` instead of 0 if `FileIndex`

[jira] [Updated] (HUDI-7238) Ensure ExternalSpillableMaps are properly closed

2023-12-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7238: - Labels: pull-request-available (was: ) > Ensure ExternalSpillableMaps are properly closed >

[PR] [HUDI-7238] Bug fixes and optimization of ExternalSpillableMap [hudi]

2023-12-17 Thread via GitHub
the-other-tim-brown opened a new pull request, #10344: URL: https://github.com/apache/hudi/pull/10344 ### Change Logs - The `ExternalSpillableMap` is now marked as `Closeable` as a reminder to developers that it needs to be closed and so we can use try-with-resources with this object

[I] [SUPPORT] Spark planner choose broadcast hash join for large HUDI data source [hudi]

2023-12-17 Thread via GitHub
beyond1920 opened a new issue, #10343: URL: https://github.com/apache/hudi/issues/10343 After apply [HUDI-6941](https://issues.apache.org/jira/browse/HUDI-6941) in the internal HUDI version(based on 0.14.0 version), there is a frequent occurrence of the execution plan selecting "broadcast h

[jira] [Created] (HUDI-7238) Ensure ExternalSpillableMaps are properly closed

2023-12-17 Thread Timothy Brown (Jira)
Timothy Brown created HUDI-7238: --- Summary: Ensure ExternalSpillableMaps are properly closed Key: HUDI-7238 URL: https://issues.apache.org/jira/browse/HUDI-7238 Project: Apache Hudi Issue Type:

[jira] [Updated] (HUDI-7237) Minor Improvements to Schema Handling in Delta Sync

2023-12-17 Thread Timothy Brown (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Brown updated HUDI-7237: Priority: Minor (was: Major) > Minor Improvements to Schema Handling in Delta Sync > --

Re: [PR] [HUDI-7183] Fix static insert overwrite partitions issue [hudi]

2023-12-17 Thread via GitHub
beyond1920 commented on code in PR #10254: URL: https://github.com/apache/hudi/pull/10254#discussion_r1429169068 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala: ## @@ -334,42 +336,51 @@ trait ProvidesHoodieConfig ex