Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751617070 > I think timestamp(0) type is the culprit, in PR https://github.com/apache/hudi/pull/8418/files, we adapter to timestamp(3) and timestamp(6) but any precision other than that is wri

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751613732 > I think timestamp(0) type is the culprit, in PR https://github.com/apache/hudi/pull/8418/files, we adapter to timestamp(3) and timestamp(6) but any precision other than that is wri

Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]

2023-10-06 Thread via GitHub
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751610045 > What I did according to this article :https://hudi.apache.org/cn/docs/0.13.0/docker_demo#testing-hudi-in-local-docker-environment What I did according to this article :https

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751609504 > Contributor I try try timestamp(3) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751608503 Maybe you can analyze you context why the schema is not set up correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751608161 I think timestamp(0) type is the culprit, in PR https://github.com/apache/hudi/pull/8418/files, we adapter to timestamp(3) and timestamp(6) but any precision other than that is writte

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349456496 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/BaseHoodieCompactionPlanGenerator.java: ## @@ -118,8 +120,21 @@ publ

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751586925 > > You first error strack trace indicates that you were doing an offline compaction job, and the exception is thrown when the `TableSchemaResolver` decodes the table schema, is that

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751586535 > You first error strack trace indicates that you were doing an offline compaction job, and the exception is thrown when the `TableSchemaResolver` decodes the table schema, is that t

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349449472 ## hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/timeline/TestCompletionTimeQueryView.java: ## Review Comment: We can not do that, some classe

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349447360 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream baseFile

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349445871 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream baseFile

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444563 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java: ## @@ -516,10 +516,13 @@ public void testMultiWriterWithAsyncTab

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444521 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void tes

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444353 ## hudi-common/src/main/java/org/apache/hudi/common/util/HoodieRecordUtils.java: ## @@ -109,4 +110,11 @@ public static boolean recordTypeCompatibleEngine(HoodieRecordTy

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349444096 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

[jira] [Created] (HUDI-6921) Make the CompletionTimeQueryView loading incremental

2023-10-06 Thread Danny Chen (Jira)
Danny Chen created HUDI-6921: Summary: Make the CompletionTimeQueryView loading incremental Key: HUDI-6921 URL: https://issues.apache.org/jira/browse/HUDI-6921 Project: Apache Hudi Issue Type: Im

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349443600 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -138,6 +142,20 @@ protected void refreshTimeline(HoodieTimeline v

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442571 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -69,6 +84,64 @@ public CompletionTimeQueryView(HoodieTableMetaClie

Re: [I] [SUPPORT]flink[1.15.4] & hudi[0.12.3],after flink bulk_insert & bucket_index, when flink trigger compaction, get exception [hudi]

2023-10-06 Thread via GitHub
li-ang-666 commented on issue #9828: URL: https://github.com/apache/hudi/issues/9828#issuecomment-1751573818 the mysql table is: CREATE TABLE `ratio_path_company` ( `id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT '主键id', `company_id` bigint NOT NULL DEFAULT '0' COMMENT '公司id

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442374 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java: ## @@ -90,15 +101,25 @@ public void withOutputStream(FSDataOutputStream output)

Re: [I] [SUPPORT] CDC file clean not work [hudi]

2023-10-06 Thread via GitHub
zealjoanna commented on issue #9557: URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751573310 > @zealjoanna Do you still have spare time for contributing the fix ? sorry for late replying , I've been quite busy lately and I'm not available for giving a PR -- This

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442256 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */ -packag

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349442027 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java: ## @@ -844,6 +847,47 @@ private Pair, List> twoUpsertCommitDataW

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441899 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,27 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /** -

Re: [I] [SUPPORT]Data loss occurs when using bulkinsert [hudi]

2023-10-06 Thread via GitHub
blackcheckren commented on issue #9748: URL: https://github.com/apache/hudi/issues/9748#issuecomment-1751572069 @ad1happy2go What other information do I need to provide in order to troubleshoot the problem? -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [I] [SUPPORT] CDC file clean not work [hudi]

2023-10-06 Thread via GitHub
zealjoanna commented on issue #9557: URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751572036 > @zealjoanna Do you still have spare time for contributing the fix ? > ![image](https://user-images.githubusercontent.com/21325163/263876807-eff4cfbd-e140-4f93-bc60-5a

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441742 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,27 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /** -

[I] [SUPPORT]flink[1.15.4] & hudi[0.12.3],after flink bulk_insert & bucket_index, when flink trigger compaction, get exception [hudi]

2023-10-06 Thread via GitHub
li-ang-666 opened a new issue, #9828: URL: https://github.com/apache/hudi/issues/9828 the flink-sql bulk_insert is : CREATE TABLE source_table( id DECIMAL(20, 0), company_id BIGINT, shareholder_id STRING, shareholder_

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349441335 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -164,66 +163,68 @@ public HoodieAppendHandle(HoodieWriteConfig config,

[jira] [Created] (HUDI-6920) Move the special handing for instant time in HoodieAppendHandle out

2023-10-06 Thread Danny Chen (Jira)
Danny Chen created HUDI-6920: Summary: Move the special handing for instant time in HoodieAppendHandle out Key: HUDI-6920 URL: https://issues.apache.org/jira/browse/HUDI-6920 Project: Apache Hudi

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440980 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -726,19 +727,24 @@ protected void reconcileAgainstMarkers(HoodieEngineCont

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440950 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java: ## @@ -233,41 +234,45 @@ private static Schema getWriteSchema(HoodieWriteConfi

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440717 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -164,66 +163,68 @@ public HoodieAppendHandle(HoodieWriteConfig config,

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349440642 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -951,12 +951,6 @@ private void startCommit(String instantTime, S

Re: [I] [SUPPORT]hudi[0.13.1] on flink[1.16.2], after bulk_insert & bucket_index, get int96 exception when flink trigger compaction [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9804: URL: https://github.com/apache/hudi/issues/9804#issuecomment-1751568479 You first error strack trace indicates that you were doing an offline compaction job, and the exception is thrown when the `TableSchemaResolver` decodes the table schema, is that the

Re: [PR] [HUDI-6495][RFC-66] Non-blocking Concurrency Control [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #7907: URL: https://github.com/apache/hudi/pull/7907#discussion_r1339600606 ## rfc/rfc-66/rfc-66.md: ## @@ -0,0 +1,318 @@ +# RFC-66: Non-blocking Concurrency Control + +## Proposers +- @danny0405 +- @ForwardXu + +## Approvers +- + +## Status + +

Re: [PR] [HUDI-6495][RFC-66] Non-blocking Concurrency Control [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #7907: URL: https://github.com/apache/hudi/pull/7907#discussion_r1349439025 ## rfc/rfc-61/rfc-61.md: ## @@ -0,0 +1,98 @@ +# RFC-61: Lockless Multi Writer + +## Proposers +- @danny0405 +- @ForwardXu +- @SteNicholas + +## Approvers +- + +## Status

Re: [I] [SUPPORT] Spark job stuck after completion, due to some non daemon threads still running [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9826: URL: https://github.com/apache/hudi/issues/9826#issuecomment-1751565733 @yihua Any good ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]

2023-10-06 Thread via GitHub
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751564997 > `hoodie.deltastreamer.schemaprovider.source.schema.file` did you create your table with SQL ddl, did you set up any schema params from cmd line? I don‘t create the table, don

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1349438084 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/HoodieFileSliceReader.java: ## @@ -19,47 +19,80 @@ package org.apache.hudi.common.tab

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1349437382 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/CachingIterator.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1349437382 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/CachingIterator.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] [SUPPORT]Data loss occurs when using bulkinsert [hudi]

2023-10-06 Thread via GitHub
blackcheckren commented on issue #9748: URL: https://github.com/apache/hudi/issues/9748#issuecomment-1751562608 @ad1happy2go yes,Yes, I tried to insert data using the bulk_insert operation type many times, and the result was a fixed number of missing data. -- This is an automated message

Re: [I] [SUPPORT] org.apache.hudi.exception.HoodieKeyException: Record key has to be non-null! with HUDI 0.12.1 and bulk_insert [hudi]

2023-10-06 Thread via GitHub
jefflee-jco commented on issue #9799: URL: https://github.com/apache/hudi/issues/9799#issuecomment-1751562249 Hi @codope , Wanted to follow-up on this ticket, if there is an estimate on the triage and any advise for a workaround to address this? Thanks! -- This is an automat

Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751562256 `hoodie.deltastreamer.schemaprovider.source.schema.file` did you create your table with SQL ddl, did you set up any schema params from cmd line? -- This is an automated message from

Re: [I] [SUPPORT] CDC file clean not work [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9557: URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751561400 @zealjoanna Do you still have spare time for contributing the fix ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [I] [SUPPORT] project hudi-common: Compilation failure: Compilation failure [hudi]

2023-10-06 Thread via GitHub
danny0405 commented on issue #9744: URL: https://github.com/apache/hudi/issues/9744#issuecomment-1751560247 Did you ever tried JDK 8 instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[jira] [Closed] (HUDI-6523) Fix get valid checkpoint for current writer

2023-10-06 Thread eric (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric closed HUDI-6523. -- Fix Version/s: 1.0.0 (was: 0.14.1) Resolution: Fixed > Fix get valid checkpoint for curren

[jira] [Commented] (HUDI-6523) Fix get valid checkpoint for current writer

2023-10-06 Thread eric (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772731#comment-17772731 ] eric commented on HUDI-6523: This issue has been resolved by https://github.com/apache/hudi/pu

Re: [PR] [HUDI-6832] Ensure other table services with correct path are not affect… [hudi]

2023-10-06 Thread via GitHub
leesf commented on code in PR #9652: URL: https://github.com/apache/hudi/pull/9652#discussion_r1349426615 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java: ## @@ -44,22 +46,43 @@ * Utils for executing multi-table services. */

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751421795 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 20243aca927344468345dd1033495082f8d5a6c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Updated] (HUDI-6919) Support Partial Merging in HoodieFileGroupReader

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu updated HUDI-6919: -- Summary: Support Partial Merging in HoodieFileGroupReader (was: Support Partial Merging) > Support Partial Mer

[jira] [Created] (HUDI-6919) Support Partial Merging

2023-10-06 Thread Lin Liu (Jira)
Lin Liu created HUDI-6919: - Summary: Support Partial Merging Key: HUDI-6919 URL: https://issues.apache.org/jira/browse/HUDI-6919 Project: Apache Hudi Issue Type: New Feature Reporter: Lin

[jira] [Assigned] (HUDI-6919) Support Partial Merging

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-6919: - Assignee: Lin Liu > Support Partial Merging > --- > > Key: HUDI-6919

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751373291 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 51fabad4f3ee6b602e7bcc886e8df7d4445a07c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751364519 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 51fabad4f3ee6b602e7bcc886e8df7d4445a07c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Assigned] (HUDI-6786) Integrate FileGroupReader with NewHoodieParquetFileFormat for Spark MOR Snapshot Query

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-6786: - Assignee: Jonathan Vexler (was: Lin Liu) > Integrate FileGroupReader with NewHoodieParquetFileFormat for

[jira] [Assigned] (HUDI-5807) HoodieSparkParquetReader is not appending partition-path values

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-5807: - Assignee: Jonathan Vexler (was: Lin Liu) > HoodieSparkParquetReader is not appending partition-path valu

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1751335152 LGTM. The GH CI failure is pretty annoying though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9774: URL: https://github.com/apache/hudi/pull/9774#issuecomment-1751300814 ## CI report: * 25d900c216bdeb1e8bd55ce4533f0ce865c27999 UNKNOWN * 658c987d20c827851b05cd530d3adfda5038df7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751300668 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 51fabad4f3ee6b602e7bcc886e8df7d4445a07c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349225544 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/ListBasedHoodieBloomIndexHelper.java: ## @@ -64,22 +64,23 @@ public HoodiePairData findMat

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349221039 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java: ## @@ -173,18 +173,18 @@ public static HoodieRecord tagRecord(HoodieReco

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349221039 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java: ## @@ -173,18 +173,18 @@ public static HoodieRecord tagRecord(HoodieReco

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349217024 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -752,6 +752,14 @@ public class HoodieWriteConfig extends HoodieCon

[jira] [Commented] (HUDI-6702) Extend merge API to support all merging operations

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772664#comment-17772664 ] Lin Liu commented on HUDI-6702: --- After supporting custom delete, this task is done. > Exten

Re: [PR] [HUDI-6784] Support deletion logic in merger [hudi]

2023-10-06 Thread via GitHub
linliu-code closed pull request #9593: [HUDI-6784] Support deletion logic in merger URL: https://github.com/apache/hudi/pull/9593 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[jira] [Resolved] (HUDI-6784) Clean Merger API and its invocations

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu resolved HUDI-6784. --- > Clean Merger API and its invocations > > > Key: HUDI-6784 >

[jira] [Closed] (HUDI-6784) Clean Merger API and its invocations

2023-10-06 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu closed HUDI-6784. - Resolution: Done > Clean Merger API and its invocations > > >

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9774: URL: https://github.com/apache/hudi/pull/9774#issuecomment-1751228833 ## CI report: * 25d900c216bdeb1e8bd55ce4533f0ce865c27999 UNKNOWN * ea1001ccef41ff55b0fb003bd8d1245800e4b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9774: URL: https://github.com/apache/hudi/pull/9774#issuecomment-1751218297 ## CI report: * 25d900c216bdeb1e8bd55ce4533f0ce865c27999 UNKNOWN * ea1001ccef41ff55b0fb003bd8d1245800e4b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751218140 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 70c2646c2d4a2ec604be3edf5649df2d6bbad8fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6832] Ensure other table services with correct path are not affect… [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9652: URL: https://github.com/apache/hudi/pull/9652#issuecomment-1751217975 ## CI report: * 30c9eb809bc9ef11b8b22d413373f6019b835a83 UNKNOWN * b9d50d1a44fc15c3cd9a001442a40d7fe0fbcece Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [I] [SUPPORT] DeltaStreamer non-continuous behavior for S3EventsSource + S3EventsHoodieIncrSource [hudi]

2023-10-06 Thread via GitHub
nickrvieira commented on issue #9398: URL: https://github.com/apache/hudi/issues/9398#issuecomment-1751178436 @ad1happy2go Thanks for the support and good to know about the new strategy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1751167107 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * 70c2646c2d4a2ec604be3edf5649df2d6bbad8fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
jonvex commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1349135494 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/LogFileIterator.java: ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
jonvex commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1349097170 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/HoodieFileSliceReader.java: ## @@ -19,47 +19,80 @@ package org.apache.hudi.common.table.

Re: [PR] [HUDI-6832] Ensure other table services with correct path are not affect… [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9652: URL: https://github.com/apache/hudi/pull/9652#issuecomment-1751137228 ## CI report: * 30c9eb809bc9ef11b8b22d413373f6019b835a83 UNKNOWN * c2a6107bcf3a084147a5bf242a681eca15b40ef5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [I] [SUPPORT] Hudi Job fails fast in concurrent write even with high retries and long wait time [hudi]

2023-10-06 Thread via GitHub
psendyk commented on issue #9728: URL: https://github.com/apache/hudi/issues/9728#issuecomment-1751114980 @Jason-liujc I believe the issue you're seeing is unrelated to lock acquisition. From the [multi-writer docs](https://hudi.apache.org/docs/concurrency_control/#model-c-multi-writer):

Re: [PR] [HUDI-6832] Ensure other table services with correct path are not affect… [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9652: URL: https://github.com/apache/hudi/pull/9652#issuecomment-1751069193 ## CI report: * 30c9eb809bc9ef11b8b22d413373f6019b835a83 UNKNOWN * c2a6107bcf3a084147a5bf242a681eca15b40ef5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-06 Thread via GitHub
jonvex commented on code in PR #9774: URL: https://github.com/apache/hudi/pull/9774#discussion_r1348872942 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/common/table/log/CachingIterator.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundatio

[I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]

2023-10-06 Thread via GitHub
hanrongMan opened a new issue, #9827: URL: https://github.com/apache/hudi/issues/9827 OS : MacOS 14.0 (23A344) Chip : Apple M1 Docker Env: colima version 0.5.5 + Docker version 20.10.17, build 100c70180f All my operating steps are as follows: What I did according to this

Re: [I] [SUPPORT] project hudi-common: Compilation failure: Compilation failure [hudi]

2023-10-06 Thread via GitHub
maheshguptags commented on issue #9744: URL: https://github.com/apache/hudi/issues/9744#issuecomment-1750508296 @danny0405 I am using openjdk 11.0.20 2023-07-18 https://github.com/apache/hudi/assets/115445723/c4fe9948-e2c7-4287-b40f-5e2e5cdeb442";> -- This is an automated messag

Re: [I] [SUPPORT] CDC file clean not work [hudi]

2023-10-06 Thread via GitHub
Hans-Raintree commented on issue #9557: URL: https://github.com/apache/hudi/issues/9557#issuecomment-1750498971 Hey @zealjoanna, @ad1happy2go has there been any movement on this? I'm facing the same issue. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750448550 ## CI report: * a4985db0ce22fb4b4f2518ed70bd96890024a08b UNKNOWN * 22fb8e8c4f28a18967d7fe924e145c759d39e2af Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [I] [SUPPORT] Process Fallout: Unable to upsert due to continues Failure in Compaction [hudi]

2023-10-06 Thread via GitHub
ketkidev commented on issue #9822: URL: https://github.com/apache/hudi/issues/9822#issuecomment-1750438426 @ad1happy2go As disucssed, we are getting another error which suggesting that some file is missing: ` : org.apache.hudi.exception.HoodieCompactionException: Could not compact s3a

[I] [SUPPORT] Spark job stuck after completion, due to some non daemon threads still running [hudi]

2023-10-06 Thread via GitHub
pravin1406 opened a new issue, #9826: URL: https://github.com/apache/hudi/issues/9826 Hi, In one of the cases when we writing incremental data (upsert) in a hudi table, we failed to write deltacommit file on storage, which caused the HoodieIOException. After which spark context exited.

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750347206 ## CI report: * a4985db0ce22fb4b4f2518ed70bd96890024a08b UNKNOWN * 50ff40b83a9d930f30a5bb1b16bfa0fdce6b3a5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750282028 ## CI report: * a4985db0ce22fb4b4f2518ed70bd96890024a08b UNKNOWN * 50ff40b83a9d930f30a5bb1b16bfa0fdce6b3a5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750267668 ## CI report: * 37426857eb415c779b6670e47b44578e38b5426a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2022

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750204991 ## CI report: * 37426857eb415c779b6670e47b44578e38b5426a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2022

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
linliu-code commented on code in PR #9819: URL: https://github.com/apache/hudi/pull/9819#discussion_r1348399906 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java: ## @@ -63,7 +64,7 @@ * @param The type of engine-specific record represen

Re: [PR] [HUDI-5031] Fix MERGE INTO creates empty partition files when source table has partitions but target table does not [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #6983: URL: https://github.com/apache/hudi/pull/6983#issuecomment-1750173382 ## CI report: * 0593cd212628684db658d7a8bdd8fc320069d090 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2022

Re: [I] [SUPPORT] Failed Job - doing partition and writing data - in Hudi 0.11.0 [hudi]

2023-10-06 Thread via GitHub
nandubatchu commented on issue #6367: URL: https://github.com/apache/hudi/issues/6367#issuecomment-1750125391 @ad1happy2go - do you know what needs to be done in these scenarios where I want to do upserts on large datasets where column_stats indexing is enabled? -- This is an automated me

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1348351265 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -203,11 +221,10 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750122118 ## CI report: * 37426857eb415c779b6670e47b44578e38b5426a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2022

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-06 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1750110942 ## CI report: * 57920570e4b68c6582598a8d7fb5d52d6b31232e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2022

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-06 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1348276299 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java: ## @@ -101,12 +103,36 @@ public void addBaseFile(HoodieBaseFile dataFile) { /** * A