Re: [PR] [HUDI-7769] Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11242: URL: https://github.com/apache/hudi/pull/11242#issuecomment-2114199755 ## CI report: * 922efda55e668b992e1b12b873be49c7f1645fba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR][TESTING][DNM] 0.15.0 RC1 testing [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11244: URL: https://github.com/apache/hudi/pull/11244#issuecomment-2114097437 ## CI report: * c028842814f48d8229802df9572a89c0dbfd688e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR][TESTING][DNM] 0.15.0 RC1 testing [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11244: URL: https://github.com/apache/hudi/pull/11244#issuecomment-2114089531 ## CI report: * c028842814f48d8229802df9572a89c0dbfd688e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [I] [SUPPORT] After upgrading hudi version 0.9.0 -> 0.13.1, it is slower and had mermory issue. [hudi]

2024-05-15 Thread via GitHub
ssilb4 commented on issue #11241: URL: https://github.com/apache/hudi/issues/11241#issuecomment-2114089485 I think hoodie.metadata.enable is problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [HUDI-7146] Implement secondary index write path [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11146: URL: https://github.com/apache/hudi/pull/11146#issuecomment-2114089178 ## CI report: * 8a6a98e1a8f0f65df59dddf663b0ef231f4c01ee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR][Testing][DO NOT MERGE] 0.15.0 RC testing [hudi]

2024-05-15 Thread via GitHub
yihua closed pull request #11227: [MINOR][Testing][DO NOT MERGE] 0.15.0 RC testing URL: https://github.com/apache/hudi/pull/11227 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] [SUPPORT] RLI index slowing down [hudi]

2024-05-15 Thread via GitHub
manishgaurav84 commented on issue #11243: URL: https://github.com/apache/hudi/issues/11243#issuecomment-2114039570 [Uploading DOC-20240516-WA0005.zip…]() -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] [HUDI-7769] Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11242: URL: https://github.com/apache/hudi/pull/11242#issuecomment-2114039251 ## CI report: * 922efda55e668b992e1b12b873be49c7f1645fba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[PR] [MINOR][TESTING][DNM] 0.15.0 RC1 testing [hudi]

2024-05-15 Thread via GitHub
yihua opened a new pull request, #11244: URL: https://github.com/apache/hudi/pull/11244 ### Change Logs As above. ### Impact Testing only. ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read th

[I] [SUPPORT] RLI index slowing down [hudi]

2024-05-15 Thread via GitHub
manishgaurav84 opened a new issue, #11243: URL: https://github.com/apache/hudi/issues/11243 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev

Re: [PR] [HUDI-7769] Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11242: URL: https://github.com/apache/hudi/pull/11242#issuecomment-2114032260 ## CI report: * 922efda55e668b992e1b12b873be49c7f1645fba UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [MINOR][DO NOT MERGE] Test 0.15.0 with reverting Spark versions [hudi]

2024-05-15 Thread via GitHub
yihua closed pull request #11231: [MINOR][DO NOT MERGE] Test 0.15.0 with reverting Spark versions URL: https://github.com/apache/hudi/pull/11231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

(hudi) branch branch-0.x updated: [HUDI-6386] Branch 0.x failing tests test multi writer archival (#11239)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/branch-0.x by this push: new 2815aef3101 [HUDI-6386] Branch 0.x failing

Re: [PR] [HUDI-6386] Branch 0.x failing tests test multi writer archival [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11239: URL: https://github.com/apache/hudi/pull/11239 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

(hudi) branch branch-0.x updated: [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 (#11240)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/branch-0.x by this push: new 2b81e6bf96e [HUDI-7771] Making OverwriteWit

Re: [PR] [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11240: URL: https://github.com/apache/hudi/pull/11240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11226: URL: https://github.com/apache/hudi/pull/11226#issuecomment-2114025455 ## CI report: * c7b402870d78e079662a5d810f7484e39dc20f83 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Updated] (HUDI-7769) Fix Hudi CDC read on Spark 3.3.4 and 3.4.3

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7769: - Labels: pull-request-available (was: ) > Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 > ---

[PR] [HUDI-7769] Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 [hudi]

2024-05-15 Thread via GitHub
yihua opened a new pull request, #11242: URL: https://github.com/apache/hudi/pull/11242 ### Change Logs The CDC releation expects `InternalRow` from the base and log files for merging, so we have to explicitly turn off `spark.sql.parquet.enableVectorizedReader`. Otherwise, the error

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hwani3142 commented on code in PR #11226: URL: https://github.com/apache/hudi/pull/11226#discussion_r1602606957 ## hudi-common/src/main/java/org/apache/hudi/metrics/JmxMetricsReporter.java: ## @@ -72,9 +70,29 @@ public JmxMetricsReporter(HoodieMetricsConfig config, MetricRegist

[jira] [Commented] (HUDI-7717) hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates)

2024-05-15 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846808#comment-17846808 ] Geser Dugarov commented on HUDI-7717: - MR with fix is under review. > hoodie.combine.

[jira] [Updated] (HUDI-7769) Fix Hudi CDC read on Spark 3.3.4 and 3.4.3

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7769: Summary: Fix Hudi CDC read on Spark 3.3.4 and 3.4.3 (was: Fix Hudi read on Spark 3.3.4 and 3.4.3) > Fix Hu

[jira] [Assigned] (HUDI-7769) Fix Hudi read on Spark 3.3.4 and 3.4.3

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7769: --- Assignee: Ethan Guo > Fix Hudi read on Spark 3.3.4 and 3.4.3 > --

[jira] [Updated] (HUDI-7717) hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates)

2024-05-15 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-7717: Fix Version/s: 1.0.0 (was: 0.15.0) > hoodie.combine.before.insert silently br

Re: [PR] [HUDI-7652] Add new `HoodieMergeKey` API to support simple and composite keys [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on code in PR #11077: URL: https://github.com/apache/hudi/pull/11077#discussion_r1602600466 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java: ## @@ -222,7 +223,8 @@ public Iterator iterator() { } public M

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on code in PR #11226: URL: https://github.com/apache/hudi/pull/11226#discussion_r1602597067 ## hudi-common/src/main/java/org/apache/hudi/metrics/JmxMetricsReporter.java: ## @@ -72,9 +70,29 @@ public JmxMetricsReporter(HoodieMetricsConfig config, MetricRegist

(hudi) branch master updated: [MINOR] Rebalance CI with tests in hudi-hadoop-common module (#11230)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 5bf509014a6 [MINOR] Rebalance CI with tests in hudi

Re: [PR] [MINOR] Rebalance CI with tests in hudi-hadoop-common module [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11230: URL: https://github.com/apache/hudi/pull/11230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-7146] Implement secondary index write path [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11146: URL: https://github.com/apache/hudi/pull/11146#issuecomment-2113987490 ## CI report: * f232b46fcd23d960efc587a624c2e9d69d3d7e9e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7146] Implement secondary index write path [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11146: URL: https://github.com/apache/hudi/pull/11146#issuecomment-2113981548 ## CI report: * f232b46fcd23d960efc587a624c2e9d69d3d7e9e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11226: URL: https://github.com/apache/hudi/pull/11226#issuecomment-2113930983 ## CI report: * 76e758753be4c817618c5e371e99cbd52fb09a46 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11226: URL: https://github.com/apache/hudi/pull/11226#issuecomment-2113925223 ## CI report: * 76e758753be4c817618c5e371e99cbd52fb09a46 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT]Flink Streaming Read hudi table which is in clustering,encounterd file not exists. [hudi]

2024-05-15 Thread via GitHub
weitianpei commented on issue #11090: URL: https://github.com/apache/hudi/issues/11090#issuecomment-2113917175 > 是否必须加read.streaming.skip_clustering = true,当我读 上游开启了异步cluster的数据? 就是我前几天遇到的问题。我发现没有开启,貌似数据没有重复读。 -- This is an automated message from the Apache Git Service. To respond

Re: [I] [SUPPORT]Flink Streaming Read hudi table which is in clustering,encounterd file not exists. [hudi]

2024-05-15 Thread via GitHub
weitianpei commented on issue #11090: URL: https://github.com/apache/hudi/issues/11090#issuecomment-2113916438 是否必须加read.streaming.skip_clustering = true,当我读 上游开启了异步cluster的数据? 就是我前几天遇到的问题。我发现没有开启,貌似数据没有重复读。 > 2024年5月10日 17:03,Danny Chan ***@***.***> 写道: > > >

Re: [I] [SUPPORT]Flink Streaming Read hudi table which is in clustering,encounterd file not exists. [hudi]

2024-05-15 Thread via GitHub
weitianpei commented on issue #11090: URL: https://github.com/apache/hudi/issues/11090#issuecomment-2113914086 是否必须加合格参数呢?我在下游读的时候 read.skip_clustering > 2024年5月10日 17:03,Danny Chan ***@***.***> 写道: > > > Thanks for the feedback, feel free to reop

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hwani3142 commented on code in PR #11226: URL: https://github.com/apache/hudi/pull/11226#discussion_r1602512348 ## hudi-common/src/main/java/org/apache/hudi/metrics/JmxMetricsReporter.java: ## @@ -72,9 +69,25 @@ public JmxMetricsReporter(HoodieMetricsConfig config, MetricRegist

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
hwani3142 commented on code in PR #11226: URL: https://github.com/apache/hudi/pull/11226#discussion_r1602512348 ## hudi-common/src/main/java/org/apache/hudi/metrics/JmxMetricsReporter.java: ## @@ -72,9 +69,25 @@ public JmxMetricsReporter(HoodieMetricsConfig config, MetricRegist

Re: [PR] [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11240: URL: https://github.com/apache/hudi/pull/11240#issuecomment-2113799542 ## CI report: * 50bb1a0344e395c50e1d54c1b14ff332ce63a6d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[I] [SUPPORT] After upgrading hudi version 0.9.0 -> 0.13.1, it is slower and had mermory issue. [hudi]

2024-05-15 Thread via GitHub
ssilb4 opened a new issue, #11241: URL: https://github.com/apache/hudi/issues/11241 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr.

Re: [PR] [HUDI-7652] Add new `HoodieMergeKey` API to support simple and composite keys [hudi]

2024-05-15 Thread via GitHub
codope commented on code in PR #11077: URL: https://github.com/apache/hudi/pull/11077#discussion_r1602430560 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java: ## @@ -222,7 +223,8 @@ public Iterator iterator() { } public Map

Re: [PR] [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11240: URL: https://github.com/apache/hudi/pull/11240#issuecomment-2113723990 ## CI report: * 50bb1a0344e395c50e1d54c1b14ff332ce63a6d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]

2024-05-15 Thread via GitHub
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2113723533 Hi @ad1happy2go, yes, because we set "hoodie.cleaner.policy.failed.writes" = "NEVER", so it's not running rollback. And the reason we manually deleted the delta commit file is

Re: [PR] [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11240: URL: https://github.com/apache/hudi/pull/11240#issuecomment-2113717846 ## CI report: * 50bb1a0344e395c50e1d54c1b14ff332ce63a6d5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-6386] Branch 0.x failing tests test multi writer archival [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11239: URL: https://github.com/apache/hudi/pull/11239#issuecomment-2113717808 ## CI report: * ebea9b7c5152d135610bb35de89cf1d9e1ab1449 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Updated] (HUDI-7771) Make default hoodie record payload as OverwriteWithLatestPayload for 0.15.0

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7771: - Labels: pull-request-available (was: ) > Make default hoodie record payload as OverwriteWithLates

[PR] [HUDI-7771] Making OverwriteWithLatestPayload as default payload in 0.15.0 [hudi]

2024-05-15 Thread via GitHub
nsivabalan opened a new pull request, #11240: URL: https://github.com/apache/hudi/pull/11240 ### Change Logs Making OverwriteWithLatestPayload as default payload in 0.15.0 ### Impact Making OverwriteWithLatestPayload as default payload in 0.15.0 ### Risk level (wri

Re: [PR] [HUDI-7763] Fix that multiple jmx reporter can exist if metadata enables [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on code in PR #11226: URL: https://github.com/apache/hudi/pull/11226#discussion_r1602402567 ## hudi-common/src/main/java/org/apache/hudi/metrics/JmxMetricsReporter.java: ## @@ -72,9 +69,25 @@ public JmxMetricsReporter(HoodieMetricsConfig config, MetricRegist

[jira] [Assigned] (HUDI-7771) Make default hoodie record payload as OverwriteWithLatestPayload for 0.15.0

2024-05-15 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-7771: - Assignee: sivabalan narayanan > Make default hoodie record payload as OverwriteWi

[jira] [Updated] (HUDI-7771) Make default hoodie record payload as OverwriteWithLatestPayload for 0.15.0

2024-05-15 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-7771: -- Fix Version/s: 0.15.0 > Make default hoodie record payload as OverwriteWithLatestPayload

Re: [PR] [HUDI-7762] Optimizing Hudi Table Check with Delta Lake by Refining Class Name Checks In Spark3.5 [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on PR #11224: URL: https://github.com/apache/hudi/pull/11224#issuecomment-2113697495 > When executed on a Delta table, this may result in an error. What action are we executing here? -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Created] (HUDI-7771) Make default hoodie record payload as OverwriteWithLatestPayload for 0.15.0

2024-05-15 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7771: - Summary: Make default hoodie record payload as OverwriteWithLatestPayload for 0.15.0 Key: HUDI-7771 URL: https://issues.apache.org/jira/browse/HUDI-7771 Pro

Re: [PR] [HUDI-7758] Only consider files in Hudi partitions when initializing MDT [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on code in PR #11219: URL: https://github.com/apache/hudi/pull/11219#discussion_r1602400778 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -2000,16 +2000,15 @@ public DirectoryInfo(String relativePath, List pathInfo

Re: [PR] [HUDI-7652] Add new `HoodieMergeKey` API to support simple and composite keys [hudi]

2024-05-15 Thread via GitHub
danny0405 commented on code in PR #11077: URL: https://github.com/apache/hudi/pull/11077#discussion_r1602397269 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java: ## @@ -222,7 +223,8 @@ public Iterator iterator() { } public M

Re: [PR] [HUDI-6386] Branch 0.x failing tests test multi writer archival [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11239: URL: https://github.com/apache/hudi/pull/11239#issuecomment-2113672052 ## CI report: * ebea9b7c5152d135610bb35de89cf1d9e1ab1449 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7766] Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11234: URL: https://github.com/apache/hudi/pull/11234#issuecomment-2113665875 ## CI report: * f762633ac16c1072963f4846d70686d67e6d8063 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6386] Branch 0.x failing tests test multi writer archival [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11239: URL: https://github.com/apache/hudi/pull/11239#issuecomment-2113665931 ## CI report: * ebea9b7c5152d135610bb35de89cf1d9e1ab1449 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Created] (HUDI-7770) Bootstrap read tries to parse partition from the bootstrap base path

2024-05-15 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-7770: - Summary: Bootstrap read tries to parse partition from the bootstrap base path Key: HUDI-7770 URL: https://issues.apache.org/jira/browse/HUDI-7770 Project: Apache Hu

[PR] [HUDI-6386] Branch 0.x failing tests test multi writer archival [hudi]

2024-05-15 Thread via GitHub
nsivabalan opened a new pull request, #11239: URL: https://github.com/apache/hudi/pull/11239 ### Change Logs Branch 0.x failing tests test multi writer archival(disabling flaky test) ### Impact Branch 0.x failing tests test multi writer archival(disabling flaky test)

(hudi) 01/01: Disabling flaky tests

2024-05-15 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch branch-0.x-failing-tests-test-mult-writer-archival in repository https://gitbox.apache.org/repos/asf/hudi.git commit ebea9b7c5152d135610bb35de89cf1d9e1ab1449 Author: sivabalan AuthorDate: Wed

(hudi) branch branch-0.x-failing-tests-test-mult-writer-archival created (now ebea9b7c515)

2024-05-15 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch branch-0.x-failing-tests-test-mult-writer-archival in repository https://gitbox.apache.org/repos/asf/hudi.git at ebea9b7c515 Disabling flaky tests This branch includes the following ne

Re: [PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11237: URL: https://github.com/apache/hudi/pull/11237#issuecomment-2113612999 ## CI report: * 985234dd8da51ee1e5a4fda66b3eef28bdef6d0a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[PR] [WIP] Get Hudi 1.x reader to read 0.14+ writer [hudi]

2024-05-15 Thread via GitHub
bvaradar opened a new pull request, #11238: URL: https://github.com/apache/hudi/pull/11238 ### Change Logs When deploying 1.x, readers will be upgraded first. This PR is to ensure 1.x Reader be able to correctly read 0.14+ datasets. This is a WIP, Will update this PR with more fix

Re: [PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
jonvex commented on code in PR #11237: URL: https://github.com/apache/hudi/pull/11237#discussion_r1602333216 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBootstrapRelation.scala: ## @@ -115,8 +113,8 @@ abstract class BaseHoodieBootstrapRelation

Re: [PR] [HUDI-7766] Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11234: URL: https://github.com/apache/hudi/pull/11234#issuecomment-2113533524 ## CI report: * f762633ac16c1072963f4846d70686d67e6d8063 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11237: URL: https://github.com/apache/hudi/pull/11237#issuecomment-2113533568 ## CI report: * 985234dd8da51ee1e5a4fda66b3eef28bdef6d0a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
yihua commented on code in PR #11237: URL: https://github.com/apache/hudi/pull/11237#discussion_r1602304217 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBootstrapRelation.scala: ## @@ -115,8 +113,8 @@ abstract class BaseHoodieBootstrapRelation(

Re: [PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11237: URL: https://github.com/apache/hudi/pull/11237#issuecomment-2113524977 ## CI report: * 985234dd8da51ee1e5a4fda66b3eef28bdef6d0a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7766] Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11234: URL: https://github.com/apache/hudi/pull/11234#issuecomment-2113524931 ## CI report: * f762633ac16c1072963f4846d70686d67e6d8063 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

(hudi) branch branch-0.x updated: [HUDI-7767] Revert Spark 3.3 and 3.4 upgrades (#11235)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/branch-0.x by this push: new c4ca02812f5 [HUDI-7767] Revert Spark 3.3 an

Re: [PR] [HUDI-7767][0.x] Revert Spark 3.3 and 3.4 upgrades [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11235: URL: https://github.com/apache/hudi/pull/11235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

(hudi) branch branch-0.x updated (5f65aac5e21 -> 98e9cb16ef3)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git from 5f65aac5e21 [HUDI-7768] Fixing failing tests of async compaction metadata for 0.15.0 (#11232) add 98e9cb16ef3 [

Re: [PR] [HUDI-7765][branch-0.x] Turn off native HFile reader for 0.15.0 release [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11233: URL: https://github.com/apache/hudi/pull/11233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

[PR] fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
jonvex opened a new pull request, #11237: URL: https://github.com/apache/hudi/pull/11237 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

Re: [PR] Fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
jonvex closed pull request #11236: Fix bootstrap issue URL: https://github.com/apache/hudi/pull/11236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: co

(hudi) branch branch-0.x updated (cc64cd82747 -> 5f65aac5e21)

2024-05-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git from cc64cd82747 [HUDI-7532] Include only compaction instants for lastCompaction in getDeltaCommitsSinceLatestCompaction

Re: [PR] [HUDI-7768] Branch 0.x failingtests async compaction metadata num commits check [hudi]

2024-05-15 Thread via GitHub
yihua merged PR #11232: URL: https://github.com/apache/hudi/pull/11232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

[jira] [Assigned] (HUDI-7768) Fix failing tests for 0.15.0 release (async compaction and metadata num commits check)

2024-05-15 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-7768: - Assignee: sivabalan narayanan > Fix failing tests for 0.15.0 release (async compa

[PR] Fix bootstrap issue [hudi]

2024-05-15 Thread via GitHub
jonvex opened a new pull request, #11236: URL: https://github.com/apache/hudi/pull/11236 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[jira] [Updated] (HUDI-7767) Revert Spark 3.3 and 3.4 upgrades

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7767: - Labels: pull-request-available (was: ) > Revert Spark 3.3 and 3.4 upgrades > ---

[jira] [Created] (HUDI-7769) Fix Hudi read on Spark 3.3.4 and 3.4.3

2024-05-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7769: --- Summary: Fix Hudi read on Spark 3.3.4 and 3.4.3 Key: HUDI-7769 URL: https://issues.apache.org/jira/browse/HUDI-7769 Project: Apache Hudi Issue Type: Improvement

[PR] [HUDI-7767][0.x] Revert Spark 3.3 and 3.4 upgrades [hudi]

2024-05-15 Thread via GitHub
yihua opened a new pull request, #11235: URL: https://github.com/apache/hudi/pull/11235 ### Change Logs As above, to avoid read failure on Spark 3.3.4 and 3.4.3. HUDI-7769 as a follow-up to fix this. ### Impact Bug fix to avoid regression. ### Risk level n

[jira] [Updated] (HUDI-7766) Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7766: - Labels: pull-request-available (was: ) > Adding staging jar deployment command for Spark 3.5 and

[PR] [HUDI-7766] Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile [hudi]

2024-05-15 Thread via GitHub
yihua opened a new pull request, #11234: URL: https://github.com/apache/hudi/pull/11234 ### Change Logs This PR adds the staging jar deployment command for Spark 3.5 and Scala 2.13 profile to the release script. ### Impact Release jars for Spark 3.5 and Scala 2.13 profil

[jira] [Updated] (HUDI-7765) Turn off native HFile reader for 0.15.0 release

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7765: - Labels: pull-request-available (was: ) > Turn off native HFile reader for 0.15.0 release > --

[PR] [HUDI-7765] Turn off native HFile reader for 0.15.0 release [hudi]

2024-05-15 Thread via GitHub
yihua opened a new pull request, #11233: URL: https://github.com/apache/hudi/pull/11233 ### Change Logs As above. ### Impact New feature turned off by default. ### Risk level none ### Documentation Update Config docs will be updated automatical

[jira] [Updated] (HUDI-7768) Fix failing tests for 0.15.0 release (async compaction and metadata num commits check)

2024-05-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7768: - Labels: pull-request-available (was: ) > Fix failing tests for 0.15.0 release (async compaction a

[PR] [HUDI-7768] Branch 0.x failingtests async compaction metadata num commits check [hudi]

2024-05-15 Thread via GitHub
nsivabalan opened a new pull request, #11232: URL: https://github.com/apache/hudi/pull/11232 ### Change Logs Fix failing tests for 0.15.0 branch. More details in the linked jira. ### Impact Fix failing tests for 0.15.0 branch. More details in the linked jira. ###

[jira] [Created] (HUDI-7768) Fix failing tests for 0.15.0 release (async compaction and metadata num commits check)

2024-05-15 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7768: - Summary: Fix failing tests for 0.15.0 release (async compaction and metadata num commits check) Key: HUDI-7768 URL: https://issues.apache.org/jira/browse/HUDI-7768

[jira] [Created] (HUDI-7767) Revert Spark 3.3 and 3.4 upgrades

2024-05-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7767: --- Summary: Revert Spark 3.3 and 3.4 upgrades Key: HUDI-7767 URL: https://issues.apache.org/jira/browse/HUDI-7767 Project: Apache Hudi Issue Type: Improvement

Re: [PR] [MINOR][DO NOT MERGE] Test 0.15.0 with reverting Spark versions [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11231: URL: https://github.com/apache/hudi/pull/11231#issuecomment-2113450499 ## CI report: * 872cbabc5f27a5335e3a322a49ed8be8bfc8f158 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Updated] (HUDI-7766) Adding staging jar deployment commandfor Spark 3.5 and Scala 2.13

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7766: Summary: Adding staging jar deployment commandfor Spark 3.5 and Scala 2.13 (was: Adding staging jar deploym

[jira] [Updated] (HUDI-7766) Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7766: Summary: Adding staging jar deployment command for Spark 3.5 and Scala 2.13 profile (was: Adding staging ja

[jira] [Created] (HUDI-7766) Adding staging jar deployment for Spark 3.5 and Scala 2.13

2024-05-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7766: --- Summary: Adding staging jar deployment for Spark 3.5 and Scala 2.13 Key: HUDI-7766 URL: https://issues.apache.org/jira/browse/HUDI-7766 Project: Apache Hudi Issue Type

[jira] [Created] (HUDI-7765) Turn off native HFile reader for 0.15.0

2024-05-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7765: --- Summary: Turn off native HFile reader for 0.15.0 Key: HUDI-7765 URL: https://issues.apache.org/jira/browse/HUDI-7765 Project: Apache Hudi Issue Type: Improvement

[jira] [Updated] (HUDI-7765) Turn off native HFile reader for 0.15.0 release

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7765: Summary: Turn off native HFile reader for 0.15.0 release (was: Turn off native HFile reader for 0.15.0) >

[jira] [Updated] (HUDI-7765) Turn off native HFile reader for 0.15.0 release

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7765: Fix Version/s: 0.15.0 > Turn off native HFile reader for 0.15.0 release > --

[jira] [Assigned] (HUDI-7765) Turn off native HFile reader for 0.15.0 release

2024-05-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7765: --- Assignee: Ethan Guo > Turn off native HFile reader for 0.15.0 release > -

Re: [PR] [MINOR] Rebalance CI with tests in hudi-hadoop-common module [hudi]

2024-05-15 Thread via GitHub
hudi-bot commented on PR #11230: URL: https://github.com/apache/hudi/pull/11230#issuecomment-2113439887 ## CI report: * 7b7ebea95b3f60be5fa8de317ac669d253a33a86 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7764] Add DefaultHoodieRecordPayload to the list of projection compatible [hudi]

2024-05-15 Thread via GitHub
jonvex closed pull request #11229: [HUDI-7764] Add DefaultHoodieRecordPayload to the list of projection compatible URL: https://github.com/apache/hudi/pull/11229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[jira] [Commented] (HUDI-7764) DefaultHoodieRecordPayload should be projection compatible

2024-05-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846745#comment-17846745 ] Jonathan Vexler commented on HUDI-7764: --- Changing this leads to OOM issues with spar

  1   2   3   4   >