Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
danny0405 merged PR #10886: URL: https://github.com/apache/hudi/pull/10886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066536128 ## CI report: * cc4c48076b11d9a97fdb7fd0f6f0a5253d530ff1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066365873 ## CI report: * af5d107b867fd97362710bc032a95743eb5d33a8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066355763 ## CI report: * af5d107b867fd97362710bc032a95743eb5d33a8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
danny0405 commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066283997 You can rebase with the latest master now to resolve the compile error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
wecharyu commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1572163511 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -94,36 +96,32 @@ public int getPartitionDepth() { /** * Write the m

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
danny0405 commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066274001 The master build is broken and here is the fix: https://github.com/apache/hudi/pull/11056, you may need to await for this patch and rebase with the latest master again~ -- This is an

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066250342 ## CI report: * af5d107b867fd97362710bc032a95743eb5d33a8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1572070120 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -94,36 +96,32 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066077325 ## CI report: * 7b04755aa308766f3b0f0d5292ed9476630da90d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2066065344 ## CI report: * 7b04755aa308766f3b0f0d5292ed9476630da90d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2065710966 ## CI report: * 7b04755aa308766f3b0f0d5292ed9476630da90d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2065675635 ## CI report: * 522a68cb3ea8dc725418eb9b811a03b5c86c694b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2065670991 ## CI report: * 522a68cb3ea8dc725418eb9b811a03b5c86c694b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
wecharyu commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1571697843 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,37 +94,33 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1571623276 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1571623276 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1571564604 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,37 +94,33 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2063823692 ## CI report: * 522a68cb3ea8dc725418eb9b811a03b5c86c694b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2063621252 ## CI report: * aadcb616ac338ef60c5799414bef660a19135c06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2063608614 ## CI report: * aadcb616ac338ef60c5799414bef660a19135c06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
wecharyu commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1570475426 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1570413829 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
Tartarus0zm commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1570153651 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-18 Thread via GitHub
Tartarus0zm commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1570153651 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-17 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1568425096 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-17 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1568425096 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-17 Thread via GitHub
boneanxs commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1568323048 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-16 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1568071623 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-16 Thread via GitHub
Tartarus0zm commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1567119140 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-15 Thread via GitHub
boneanxs commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1566657358 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-15 Thread via GitHub
Tartarus0zm commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1565699528 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-09 Thread via GitHub
danny0405 commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2044649201 > > Before throw exception finally, we could retry for several times. > > @beyond1920 It sounds a good idea here. Added a retry logic. cc: @boneanxs @danny0405 There is a `R

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-04-09 Thread via GitHub
wecharyu commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2044635984 > Before throw exception finally, we could retry for several times. @beyond1920 It sounds a good idea here. Added a retry logic. cc: @boneanxs @danny0405 -- This is an automate

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-27 Thread via GitHub
beyond1920 commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2023032742 @wecharyu > We did not notice this possible failure, it may not happen frequently because we have a pre-checked if this metadata file exists. OTOH, I think it's also OK to remove th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-26 Thread via GitHub
wecharyu commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2020686744 > Would the failure probability increasing? We did not notice this possible failure, it may not happen frequently because we have a pre-checked if this metadata file exists. OTOH, I

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-21 Thread via GitHub
beyond1920 commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2012131723 @wecharyu Thanks for your contribution. The pull request could resolve the problem, of course. But I am not sure whether that the solution would increase the probability of job fai

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-20 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1533289294 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-20 Thread via GitHub
beyond1920 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1532422677 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write t

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-20 Thread via GitHub
wecharyu commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1532367467 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1531326627 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
wecharyu commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1530312027 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write the

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2007036084 ## CI report: * aadcb616ac338ef60c5799414bef660a19135c06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
danny0405 commented on code in PR #10886: URL: https://github.com/apache/hudi/pull/10886#discussion_r1530176270 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java: ## @@ -92,11 +92,12 @@ public int getPartitionDepth() { /** * Write th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2006901528 ## CI report: * aadcb616ac338ef60c5799414bef660a19135c06 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
hudi-bot commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2006875566 ## CI report: * aadcb616ac338ef60c5799414bef660a19135c06 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
wecharyu commented on PR #10886: URL: https://github.com/apache/hudi/pull/10886#issuecomment-2006795897 For https://github.com/apache/hudi/issues/10885, cc: @beyond1920 @boneanxs @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] [HUDI-7515] Fix partition metadata write failure [hudi]

2024-03-19 Thread via GitHub
wecharyu opened a new pull request, #10886: URL: https://github.com/apache/hudi/pull/10886 ### Change Logs When `spark.speculation` is enabled, if the write metadata operation become slow for some reason, a speculative will be started to write the same metadata file concurrently.