[GitHub] [hudi] chenbodeng719 commented on pull request #7706: [HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error

2023-03-15 Thread via GitHub
chenbodeng719 commented on PR #7706: URL: https://github.com/apache/hudi/pull/7706#issuecomment-1471417671 Which released version includes this feature realesed? Hudi 0.12 version has to stick with this problem? -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471405017 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8179: [HUDI-5932] Make the combine step in Call run_bootstrap Procedure optional

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8179: URL: https://github.com/apache/hudi/pull/8179#issuecomment-1471404959 ## CI report: * 088416b705b091b4a46031437afd164571808025 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1572

[GitHub] [hudi] hudi-bot commented on pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8165: URL: https://github.com/apache/hudi/pull/8165#issuecomment-1471404882 ## CI report: * a1c2ad7c90846fd9aad643c762d2398a4bf9874f UNKNOWN * 23a988f9444c3159858883e0dca2a39a181863ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471399887 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8179: [HUDI-5932] Make the combine step in Call run_bootstrap Procedure optional

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8179: URL: https://github.com/apache/hudi/pull/8179#issuecomment-1471399828 ## CI report: * 088416b705b091b4a46031437afd164571808025 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1572

[GitHub] [hudi] hudi-bot commented on pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8165: URL: https://github.com/apache/hudi/pull/8165#issuecomment-1471399707 ## CI report: * a1c2ad7c90846fd9aad643c762d2398a4bf9874f UNKNOWN * 23a988f9444c3159858883e0dca2a39a181863ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471394037 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8165: URL: https://github.com/apache/hudi/pull/8165#issuecomment-1471393919 ## CI report: * a1c2ad7c90846fd9aad643c762d2398a4bf9874f UNKNOWN * 23a988f9444c3159858883e0dca2a39a181863ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Updated] (HUDI-5903) Make number of max concurrent glue connections configurable

2023-03-15 Thread Pratyaksh Sharma (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-5903: --- Status: In Progress (was: Open) > Make number of max concurrent glue connections configurable

[GitHub] [hudi] vinothchandar commented on pull request #7984: hash partition in spark datasoure

2023-03-15 Thread via GitHub
vinothchandar commented on PR #7984: URL: https://github.com/apache/hudi/pull/7984#issuecomment-1471373856 @lvhu-goodluck Apologize for the delays here. Most of us were heads down with 0.13. Let me respond on the dev list. -- This is an automated message from the Apache Git Service. To re

[GitHub] [hudi] maheshguptags commented on issue #8195: Clustering is not happening on Flink Hudi

2023-03-15 Thread via GitHub
maheshguptags commented on issue #8195: URL: https://github.com/apache/hudi/issues/8195#issuecomment-1471357761 Hi @danny0405, I tried with the `MOR` table but the result is still the same not performing clustering. ``` 'table.type' = 'MERGE_ON_READ', 'hoodie.compact.inline'= 'tr

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471345330 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] maheshguptags commented on issue #8195: Clustering is not happening on Flink Hudi

2023-03-15 Thread via GitHub
maheshguptags commented on issue #8195: URL: https://github.com/apache/hudi/issues/8195#issuecomment-1471343339 I was trying with the `COW` table but let me try it out with the `MOR` table. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hudi] danny0405 commented on pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-15 Thread via GitHub
danny0405 commented on PR #8165: URL: https://github.com/apache/hudi/pull/8165#issuecomment-1471334736 > > There are some tests failures, can you help to confirm? > > All checks have passed, am I missing something? There are some error codes on the master that triggers failures,

[GitHub] [hudi] danny0405 commented on a diff in pull request #8190: [HUDI-5936] Fix serialization problem when FileStatus is not serializable

2023-03-15 Thread via GitHub
danny0405 commented on code in PR #8190: URL: https://github.com/apache/hudi/pull/8190#discussion_r1138139043 ## hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java: ## @@ -106,9 +106,9 @@ private List getPartitionPathWithPathPrefix(String rela

[jira] [Commented] (HUDI-5931) Improve the description of operation in HoodieDeltaStreamer

2023-03-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700983#comment-17700983 ] Danny Chen commented on HUDI-5931: -- Fixed via master branch: b79ce80f709ddffb5ad7aead1071

[jira] [Resolved] (HUDI-5931) Improve the description of operation in HoodieDeltaStreamer

2023-03-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-5931. -- > Improve the description of operation in HoodieDeltaStreamer >

[jira] [Updated] (HUDI-5931) Improve the description of operation in HoodieDeltaStreamer

2023-03-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5931: - Fix Version/s: 0.13.1 > Improve the description of operation in HoodieDeltaStreamer >

[hudi] branch master updated: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer (#8175)

2023-03-15 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new b79ce80f709 [HUDI-5931] Improve the description

[GitHub] [hudi] danny0405 merged pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer

2023-03-15 Thread via GitHub
danny0405 merged PR #8175: URL: https://github.com/apache/hudi/pull/8175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] danny0405 closed issue #8180: [SUPPORT] Snapshot query of MOR table in HUDI Flink DataSource does not behave as expected

2023-03-15 Thread via GitHub
danny0405 closed issue #8180: [SUPPORT] Snapshot query of MOR table in HUDI Flink DataSource does not behave as expected URL: https://github.com/apache/hudi/issues/8180 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Closed] (HUDI-5937) Archived log file never rollover

2023-03-15 Thread chenzhiming (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenzhiming closed HUDI-5937. - Fix Version/s: 0.12.1 Resolution: Fixed > Archived log file never rollover > --

[hudi] branch master updated: [HUDI-5939] Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)" (#8197)

2023-03-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d889cc77347 [HUDI-5939] Revert "[HUDI-4675] add uni

[GitHub] [hudi] yihua merged pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
yihua merged PR #8197: URL: https://github.com/apache/hudi/pull/8197 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[GitHub] [hudi] 1032851561 closed pull request #8194: [HUDI-5937] Archived log file never rollover

2023-03-15 Thread via GitHub
1032851561 closed pull request #8194: [HUDI-5937] Archived log file never rollover URL: https://github.com/apache/hudi/pull/8194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] 1032851561 commented on a diff in pull request #8194: [HUDI-5937] Archived log file never rollover

2023-03-15 Thread via GitHub
1032851561 commented on code in PR #8194: URL: https://github.com/apache/hudi/pull/8194#discussion_r1138123092 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java: ## @@ -216,9 +216,10 @@ private int getLogBlockLength(int contentLength, int

[GitHub] [hudi] danny0405 commented on issue #8195: Clustering is not happening on Flink Hudi

2023-03-15 Thread via GitHub
danny0405 commented on issue #8195: URL: https://github.com/apache/hudi/issues/8195#issuecomment-1471316428 The clustering only works for `MOR` table with `INSERT` operation, what is your table type then? -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [hudi] maheshguptags commented on issue #8195: Clustering is not happening on Flink Hudi

2023-03-15 Thread via GitHub
maheshguptags commented on issue #8195: URL: https://github.com/apache/hudi/issues/8195#issuecomment-1471302720 Hi @hbgstc123, I have tried with your suggestion and updated the config but I am still not getting the clustering inflight and .replace file. I am also attaching the

[GitHub] [hudi] coffee34 commented on issue #8132: [SUPPORT] data loss in new base file after compaction

2023-03-15 Thread via GitHub
coffee34 commented on issue #8132: URL: https://github.com/apache/hudi/issues/8132#issuecomment-1471301250 Thanks for reply. Currently, we have only one writer running, and it has been running without any errors for over half a year. However, I have set up a monitoring system to detect i

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1471293198 ## CI report: * 2d95079c7eecd8f4a06ed3e496535b206d5d3eb3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1471256417 ## CI report: * 2d95079c7eecd8f4a06ed3e496535b206d5d3eb3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #7955: [HUDI-5649] Unify all the loggers to slf4j

2023-03-15 Thread via GitHub
hudi-bot commented on PR #7955: URL: https://github.com/apache/hudi/pull/7955#issuecomment-1471255998 ## CI report: * 272dcfc8a4dfd42cf6ff1342b29c3799240f5c77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1572

[GitHub] [hudi] hudi-bot commented on pull request #7955: [HUDI-5649] Unify all the loggers to slf4j

2023-03-15 Thread via GitHub
hudi-bot commented on PR #7955: URL: https://github.com/apache/hudi/pull/7955#issuecomment-1471249981 ## CI report: * 272dcfc8a4dfd42cf6ff1342b29c3799240f5c77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1572

[GitHub] [hudi] hudi-bot commented on pull request #7362: [HUDI-5315] The record size is dynamically estimated when the table i…

2023-03-15 Thread via GitHub
hudi-bot commented on PR #7362: URL: https://github.com/apache/hudi/pull/7362#issuecomment-1471249454 ## CI report: * 4ace11977a64bc8bee549351ac815a9cdb00aa33 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1521

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1471244353 ## CI report: * 2d95079c7eecd8f4a06ed3e496535b206d5d3eb3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] weimingdiit commented on pull request #7362: [HUDI-5315] The record size is dynamically estimated when the table i…

2023-03-15 Thread via GitHub
weimingdiit commented on PR #7362: URL: https://github.com/apache/hudi/pull/7362#issuecomment-1471243966 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[jira] [Updated] (HUDI-5944) Added the ability to fix partitiion missing in hivesynctool

2023-03-15 Thread weiming (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weiming updated HUDI-5944: -- Description: When occ is enabled, hivesynctool causes partiton metadata to be lost. We added some repair logic.

[jira] [Updated] (HUDI-5944) Added the ability to fix partitiion missing in hivesynctool

2023-03-15 Thread weiming (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weiming updated HUDI-5944: -- Description: When occ is enabled, hivesynctool causes partiton metadata to be lost. We added some repair logic.

[jira] [Commented] (HUDI-5904) support more than one update actions in merge into table

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700949#comment-17700949 ] xy commented on HUDI-5904: -- master commit:f51a45589d72a12c4e894e17bb90a56bca44fca6 > support mor

[jira] [Updated] (HUDI-5944) Added the ability to fix partitiion missing in hivesynctool

2023-03-15 Thread weiming (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weiming updated HUDI-5944: -- Attachment: clipboard_image_1678936408619.png clipboard_image_1678936464376.png > Added the abil

[jira] [Updated] (HUDI-5904) support more than one update actions in merge into table

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-5904: - Summary: support more than one update actions in merge into table (was: Only one updating action is supported in MERGE IN

[jira] [Resolved] (HUDI-5904) Only one updating action is supported in MERGE INTO statement

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy resolved HUDI-5904. -- > Only one updating action is supported in MERGE INTO statement > -

[jira] [Updated] (HUDI-5904) Only one updating action is supported in MERGE INTO statement

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-5904: - Status: In Progress (was: Open) > Only one updating action is supported in MERGE INTO statement > --

[GitHub] [hudi] SteNicholas commented on a diff in pull request #8188: [MINOR] Improve instantToWrite

2023-03-15 Thread via GitHub
SteNicholas commented on code in PR #8188: URL: https://github.com/apache/hudi/pull/8188#discussion_r1138039079 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/TimeWait.java: ## @@ -67,7 +68,7 @@ public void waitFor() { * Builder. */ publi

[jira] [Updated] (HUDI-5904) Only one updating action is supported in MERGE INTO statement

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-5904: - Affects Version/s: 0.13.0 0.12.2 0.12.1 > Only one updating action is suppor

[jira] [Updated] (HUDI-5904) Only one updating action is supported in MERGE INTO statement

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-5904: - Fix Version/s: 0.13.1 > Only one updating action is supported in MERGE INTO statement > -

[jira] [Assigned] (HUDI-5904) Only one updating action is supported in MERGE INTO statement

2023-03-15 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy reassigned HUDI-5904: Assignee: xy > Only one updating action is supported in MERGE INTO statement > ---

[GitHub] [hudi] SteNicholas commented on a diff in pull request #8188: [MINOR] Improve instantToWrite

2023-03-15 Thread via GitHub
SteNicholas commented on code in PR #8188: URL: https://github.com/apache/hudi/pull/8188#discussion_r1138038570 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriteFunction.java: ## @@ -195,7 +200,7 @@ private String instantToWrite() {

[GitHub] [hudi] SteNicholas commented on a diff in pull request #8188: [MINOR] Improve instantToWrite

2023-03-15 Thread via GitHub
SteNicholas commented on code in PR #8188: URL: https://github.com/apache/hudi/pull/8188#discussion_r1138037717 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/TimeWait.java: ## @@ -54,10 +54,11 @@ public static Builder builder() { public void wai

[jira] [Updated] (HUDI-5938) No need to init the properties for each data flushing

2023-03-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5938: - Fix Version/s: 0.14.0 > No need to init the properties for each data flushing > --

[jira] [Closed] (HUDI-5938) No need to init the properties for each data flushing

2023-03-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-5938. Resolution: Fixed Fixed via master branch: eb921591562ae178031c406b48489c30fa587d06 > No need to init the p

[hudi] branch master updated: [HUDI-5938] No need to init the properties for each data flushing (#8192)

2023-03-15 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new eb921591562 [HUDI-5938] No need to init the pro

[GitHub] [hudi] danny0405 commented on pull request #8192: [HUDI-5938] No need to init the properties for each data flushing

2023-03-15 Thread via GitHub
danny0405 commented on PR #8192: URL: https://github.com/apache/hudi/pull/8192#issuecomment-1471217300 The spark test failure is not caused by this minor change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] danny0405 merged pull request #8192: [HUDI-5938] No need to init the properties for each data flushing

2023-03-15 Thread via GitHub
danny0405 merged PR #8192: URL: https://github.com/apache/hudi/pull/8192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] gudladona opened a new issue, #8199: [SUPPORT] OOM during a Sync/Async clean operation

2023-03-15 Thread via GitHub
gudladona opened a new issue, #8199: URL: https://github.com/apache/hudi/issues/8199 **OOM during a Sync or Async clean operation** ENV: Hudi version: 0.11.1 Java Version 1.8 Spark Version: 3.1.2 EMR version: 6.4 Clean Policy: KEEP_LATEST_BY_HOURS -- 24 hours

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471202064 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8198: URL: https://github.com/apache/hudi/pull/8198#issuecomment-1471197058 ## CI report: * 3a073e42e1acae0cb3f08e6688b1d9121edee9cb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8128: [HUDI-5782] Tweak defaults and remove unnecessary configs after config review

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8128: URL: https://github.com/apache/hudi/pull/8128#issuecomment-1471191739 ## CI report: * 894861b03430217482771663639c9e413b0dca3b UNKNOWN * 5331412a3bfb887c22641ee6bbcb7307498fb02b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Updated] (HUDI-5943) Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5943: - Labels: pull-request-available (was: ) > Support bootstrap produce to synchronize to multiple met

[GitHub] [hudi] huangxiaopingRD opened a new pull request, #8198: [HUDI-5943] Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread via GitHub
huangxiaopingRD opened a new pull request, #8198: URL: https://github.com/apache/hudi/pull/8198 ### Change Logs In bootstrap produce, currently, only Hive Metastore is supported to synchronize metadata. This PR is to support synchronization to multiple metastores ### I

[jira] [Created] (HUDI-5944) Added the ability to fix partitiion missing in hivesynctool

2023-03-15 Thread weiming (Jira)
weiming created HUDI-5944: - Summary: Added the ability to fix partitiion missing in hivesynctool Key: HUDI-5944 URL: https://issues.apache.org/jira/browse/HUDI-5944 Project: Apache Hudi Issue Type:

[jira] [Created] (HUDI-5943) Support bootstrap produce to synchronize to multiple metastores

2023-03-15 Thread xiaoping.huang (Jira)
xiaoping.huang created HUDI-5943: Summary: Support bootstrap produce to synchronize to multiple metastores Key: HUDI-5943 URL: https://issues.apache.org/jira/browse/HUDI-5943 Project: Apache Hudi

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1471156546 ## CI report: * 2d95079c7eecd8f4a06ed3e496535b206d5d3eb3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[jira] [Updated] (HUDI-5942) Support for spark 3.0

2023-03-15 Thread longyin319 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] longyin319 updated HUDI-5942: - Issue Type: Improvement (was: New Feature) > Support for spark 3.0 > - > >

[jira] [Reopened] (HUDI-5942) Support for spark 3.0

2023-03-15 Thread longyin319 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] longyin319 reopened HUDI-5942: -- > Support for spark 3.0 > - > > Key: HUDI-5942 > URL: ht

[jira] [Resolved] (HUDI-5942) Support for spark 3.0

2023-03-15 Thread longyin319 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] longyin319 resolved HUDI-5942. -- > Support for spark 3.0 > - > > Key: HUDI-5942 > URL: ht

[jira] [Updated] (HUDI-5942) Support for spark 3.0

2023-03-15 Thread longyin319 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] longyin319 updated HUDI-5942: - Description: Spark 3.0 still has a significant user base, but hudi 0.13.0 no longer provides support for S

[jira] [Created] (HUDI-5942) Support for spark 3.0

2023-03-15 Thread longyin319 (Jira)
longyin319 created HUDI-5942: Summary: Support for spark 3.0 Key: HUDI-5942 URL: https://issues.apache.org/jira/browse/HUDI-5942 Project: Apache Hudi Issue Type: New Feature Reporter:

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1471048470 ## CI report: * 8a5b6969e8c5e7805a63d192ea7d8b4f92216de5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC

2023-03-15 Thread via GitHub
nsivabalan commented on PR #7469: URL: https://github.com/apache/hudi/pull/7469#issuecomment-1471040984 @LinMingQiang : I assume this is intermittant right. i.e we hit this when we have concurrent writers and each triggered cleaning concurrently and incidentally happened to rollback at simi

[GitHub] [hudi] nsivabalan commented on pull request #7687: [HUDI-5606] Update to handle deletes in postgres debezium

2023-03-15 Thread via GitHub
nsivabalan commented on PR #7687: URL: https://github.com/apache/hudi/pull/7687#issuecomment-1471024806 hey @BalaMahesh : we have already fix it here https://github.com/apache/hudi/pull/7944 please re-open if this is a diff PR. -- This is an automated message from the Apache Git Serv

[GitHub] [hudi] nsivabalan closed pull request #7687: [HUDI-5606] Update to handle deletes in postgres debezium

2023-03-15 Thread via GitHub
nsivabalan closed pull request #7687: [HUDI-5606] Update to handle deletes in postgres debezium URL: https://github.com/apache/hudi/pull/7687 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [hudi] nsivabalan commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

2023-03-15 Thread via GitHub
nsivabalan commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1471021896 @fengjian428 : any updates here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[jira] [Created] (HUDI-5941) Support savepoint CALL procedure with table base path

2023-03-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5941: --- Summary: Support savepoint CALL procedure with table base path Key: HUDI-5941 URL: https://issues.apache.org/jira/browse/HUDI-5941 Project: Apache Hudi Issue Type: New

[jira] [Updated] (HUDI-5941) Support savepoint CALL procedure with table base path

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5941: Fix Version/s: 0.14.0 > Support savepoint CALL procedure with table base path >

[jira] [Assigned] (HUDI-5941) Support savepoint CALL procedure with table base path

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5941: --- Assignee: Ethan Guo > Support savepoint CALL procedure with table base path > ---

[GitHub] [hudi] nsivabalan closed pull request #8044: [HUDI-5859] Adding standalone restore tool

2023-03-15 Thread via GitHub
nsivabalan closed pull request #8044: [HUDI-5859] Adding standalone restore tool URL: https://github.com/apache/hudi/pull/8044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [hudi] nsivabalan commented on a diff in pull request #8194: [HUDI-5937] Archived log file never rollover

2023-03-15 Thread via GitHub
nsivabalan commented on code in PR #8194: URL: https://github.com/apache/hudi/pull/8194#discussion_r1137911092 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java: ## @@ -216,9 +216,10 @@ private int getLogBlockLength(int contentLength, int

[GitHub] [hudi] nsivabalan commented on pull request #8194: [HUDI-5937] Archived log file never rollover

2023-03-15 Thread via GitHub
nsivabalan commented on PR #8194: URL: https://github.com/apache/hudi/pull/8194#issuecomment-1471015862 @1032851561 : is it possible to write a test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Updated] (HUDI-5940) Support predicates with target table fields in matched conditions in Spark SQL MERGE INTO

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5940: Fix Version/s: 0.14.0 > Support predicates with target table fields in matched conditions in Spark > SQL ME

[jira] [Updated] (HUDI-5940) Support predicates with target table fields in matched conditions in Spark SQL MERGE INTO

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5940: Description: See [https://github.com/apache/hudi/issues/6037] merge into delete_error_test target using (se

[GitHub] [hudi] yihua commented on issue #6037: [QUESTION] about the implementation of spark merge into

2023-03-15 Thread via GitHub
yihua commented on issue #6037: URL: https://github.com/apache/hudi/issues/6037#issuecomment-1471014100 I created HUDI-5940 for supporting predicates with target table fields in matched conditions in Spark SQL MERGE INTO. -- This is an automated message from the Apache Git Service. To res

[jira] [Created] (HUDI-5940) Support predicates with target table fields in matched conditions in Spark SQL MERGE INTO

2023-03-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5940: --- Summary: Support predicates with target table fields in matched conditions in Spark SQL MERGE INTO Key: HUDI-5940 URL: https://issues.apache.org/jira/browse/HUDI-5940 Project:

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1470993391 ## CI report: * 8a5b6969e8c5e7805a63d192ea7d8b4f92216de5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1470986623 ## CI report: * 8a5b6969e8c5e7805a63d192ea7d8b4f92216de5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8128: [HUDI-5782] Tweak defaults and remove unnecessary configs after config review

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8128: URL: https://github.com/apache/hudi/pull/8128#issuecomment-1470986502 ## CI report: * 894861b03430217482771663639c9e413b0dca3b UNKNOWN * b4c5a2a6026645077caaa012ccbf234fbece43ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] yihua commented on pull request #6458: [HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex

2023-03-15 Thread via GitHub
yihua commented on PR #6458: URL: https://github.com/apache/hudi/pull/6458#issuecomment-1470986050 Hi @fengjian428 This PR causes Azure CI to fail. I'm reverting the merged commit on master (#8197). Could you put up a PR again and fix the CI failure before merging? -- This is an automa

[GitHub] [hudi] hudi-bot commented on pull request #8128: [HUDI-5782] Tweak defaults and remove unnecessary configs after config review

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8128: URL: https://github.com/apache/hudi/pull/8128#issuecomment-1470982009 ## CI report: * 894861b03430217482771663639c9e413b0dca3b UNKNOWN * b4c5a2a6026645077caaa012ccbf234fbece43ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1470977376 ## CI report: * 8a5b6969e8c5e7805a63d192ea7d8b4f92216de5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[jira] [Assigned] (HUDI-5939) Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5939: --- Assignee: Ethan Guo > Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)" >

[jira] [Updated] (HUDI-5939) Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5939: Priority: Blocker (was: Major) > Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#645

[jira] [Updated] (HUDI-5939) Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5939: Story Points: 0.5 > Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)" >

[jira] [Updated] (HUDI-5939) Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5939: Fix Version/s: 0.14.0 > Revert "[HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458)" >

[jira] [Updated] (HUDI-5912) Update snapshot_exporter to reflect the corrent jar name.md

2023-03-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5912: - Labels: pull-request-available (was: ) > Update snapshot_exporter to reflect the corrent jar name

[hudi] branch asf-site updated: [HUDI-5912] Update snapshot_exporter.md (#8142)

2023-03-15 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 7620e6b346b [HUDI-5912] Update snapshot_exp

[GitHub] [hudi] nsivabalan merged pull request #8142: [HUDI-5912] Update snapshot_exporter to reflect the corrent jar name.md

2023-03-15 Thread via GitHub
nsivabalan merged PR #8142: URL: https://github.com/apache/hudi/pull/8142 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

[GitHub] [hudi] nsivabalan commented on pull request #8029: [HUDI-5832] add relocated prefix for hbase classes in hbase-site.xml

2023-03-15 Thread via GitHub
nsivabalan commented on PR #8029: URL: https://github.com/apache/hudi/pull/8029#issuecomment-1470905344 they @stayrascal : thanks for the contribution. can you give us an idea of the impact. What is failing as of now w/o this fix? and do we have any work around for existing users (even i

[GitHub] [hudi] hudi-bot commented on pull request #8197: [HUDI-5939] Revert "HUDI-4675 add unittest for RebalancedSparkHoodieHBaseIndex (#6458)"

2023-03-15 Thread via GitHub
hudi-bot commented on PR #8197: URL: https://github.com/apache/hudi/pull/8197#issuecomment-1470887107 ## CI report: * 8a5b6969e8c5e7805a63d192ea7d8b4f92216de5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1573

[hudi] branch master updated (d760ed99734 -> ca552c6a8ae)

2023-03-15 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from d760ed99734 [HUDI-5933] Revert "[HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreame

  1   2   >