[GitHub] [hudi] hudi-bot commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
hudi-bot commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1128429540 ## CI report: * 986960516f86a1426725141cd7cb25e84d260020 UNKNOWN * c5fb81a0b229ded9a2b925790366b62f1bf7ade9 UNKNOWN * a77d750b95f7fbb37675923467ca4bd6b88b006f UNKNOWN *

[GitHub] [hudi] hudi-bot commented on pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
hudi-bot commented on PR #5443: URL: https://github.com/apache/hudi/pull/5443#issuecomment-1128427356 ## CI report: * 3961299bb684991bc34c44e4c25a340fc6bbaeb2 Azure:

[GitHub] [hudi] jinxing64 commented on pull request #5588: [HUDI-4100] CTAS failed to clean up when given an illegal MANAGED table definition

2022-05-16 Thread GitBox
jinxing64 commented on PR #5588: URL: https://github.com/apache/hudi/pull/5588#issuecomment-1128427149 > rebase this pr for CI's error @XuQianJin-Stars Thanks looking into this ~ I've rebased. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] hudi-bot commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
hudi-bot commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1128425434 ## CI report: * 986960516f86a1426725141cd7cb25e84d260020 UNKNOWN * c5fb81a0b229ded9a2b925790366b62f1bf7ade9 UNKNOWN * a77d750b95f7fbb37675923467ca4bd6b88b006f UNKNOWN *

[GitHub] [hudi] hudi-bot commented on pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
hudi-bot commented on PR #5443: URL: https://github.com/apache/hudi/pull/5443#issuecomment-1128425354 ## CI report: * 3961299bb684991bc34c44e4c25a340fc6bbaeb2 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
hudi-bot commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1128423677 ## CI report: * 986960516f86a1426725141cd7cb25e84d260020 UNKNOWN * c5fb81a0b229ded9a2b925790366b62f1bf7ade9 UNKNOWN * a77d750b95f7fbb37675923467ca4bd6b88b006f UNKNOWN *

[GitHub] [hudi] hudi-bot commented on pull request #5588: [HUDI-4100] CTAS failed to clean up when given an illegal MANAGED table definition

2022-05-16 Thread GitBox
hudi-bot commented on PR #5588: URL: https://github.com/apache/hudi/pull/5588#issuecomment-1128420135 ## CI report: * c06740700f51f557b77ffc19068e36a4cb19864a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
hudi-bot commented on PR #5443: URL: https://github.com/apache/hudi/pull/5443#issuecomment-1128419946 ## CI report: * 3961299bb684991bc34c44e4c25a340fc6bbaeb2 Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #3599: [HUDI-2207] Support independent flink hudi clustering function

2022-05-16 Thread GitBox
danny0405 commented on code in PR #3599: URL: https://github.com/apache/hudi/pull/3599#discussion_r874354784 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -702,22 +702,36 @@ public static Object

[GitHub] [hudi] hudi-bot commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
hudi-bot commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1128398641 ## CI report: * 986960516f86a1426725141cd7cb25e84d260020 UNKNOWN * c5fb81a0b229ded9a2b925790366b62f1bf7ade9 UNKNOWN * a77d750b95f7fbb37675923467ca4bd6b88b006f UNKNOWN *

[GitHub] [hudi] wzx140 commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
wzx140 commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1128397162 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] qjqqyy commented on a diff in pull request #5520: [HUDI-3922] parse record key + partition path config consistently between keygens and HiveSync

2022-05-16 Thread GitBox
qjqqyy commented on code in PR #5520: URL: https://github.com/apache/hudi/pull/5520#discussion_r874349668 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java: ## @@ -37,14 +36,10 @@ public class ComplexKeyGenerator extends

[GitHub] [hudi] qjqqyy commented on a diff in pull request #5520: [HUDI-3922] parse record key + partition path config consistently between keygens and HiveSync

2022-05-16 Thread GitBox
qjqqyy commented on code in PR #5520: URL: https://github.com/apache/hudi/pull/5520#discussion_r874349668 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java: ## @@ -37,14 +36,10 @@ public class ComplexKeyGenerator extends

[GitHub] [hudi] hudi-bot commented on pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
hudi-bot commented on PR #5532: URL: https://github.com/apache/hudi/pull/5532#issuecomment-1128389850 ## CI report: * 0141f3ba49a73277d49537494510790f31bdf386 Azure:

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874317481 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/RowDataProjection.java: ## @@ -61,7 +68,10 @@ public static RowDataProjection instance(LogicalType[]

[GitHub] [hudi] hudi-bot commented on pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
hudi-bot commented on PR #5443: URL: https://github.com/apache/hudi/pull/5443#issuecomment-1128364713 ## CI report: * 919429284dc76e73435825b978a3ba2e332087bf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
hudi-bot commented on PR #5443: URL: https://github.com/apache/hudi/pull/5443#issuecomment-1128362894 ## CI report: * 919429284dc76e73435825b978a3ba2e332087bf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
hudi-bot commented on PR #5567: URL: https://github.com/apache/hudi/pull/5567#issuecomment-1128361158 ## CI report: * 765c7c9d031deb7bdebead97032a61b4b9e5ad4a Azure:

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315880 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/CastMap.java: ## @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315880 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/CastMap.java: ## @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874317316 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java: ## @@ -390,4 +417,70 @@ private InflaterInputStreamFactory

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874317481 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/RowDataProjection.java: ## @@ -61,7 +68,10 @@ public static RowDataProjection instance(LogicalType[]

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874316400 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/SchemaEvoContext.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874316400 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/SchemaEvoContext.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315880 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/CastMap.java: ## @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315880 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/CastMap.java: ## @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315475 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java: ## @@ -447,6 +453,17 @@ private Schema inferSchemaFromDdl() { return

[GitHub] [hudi] trushev commented on a diff in pull request #5443: [HUDI-3982] Comprehensive schema evolution in flink when read/batch/cow/snapshot

2022-05-16 Thread GitBox
trushev commented on code in PR #5443: URL: https://github.com/apache/hudi/pull/5443#discussion_r874315347 ## hudi-flink-datasource/hudi-flink/pom.xml: ## @@ -265,6 +265,64 @@ + + + +org.apache.spark +

[GitHub] [hudi] xiarixiaoyao commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction

2022-05-16 Thread GitBox
xiarixiaoyao commented on issue #4978: URL: https://github.com/apache/hudi/issues/4978#issuecomment-1128340650 @nsivabalan i donot think 0.11 can solve this problem. @CrazyBeeline thanks for your help.could you pls raise a pr to solve this problem, thanks very much -- This is

[jira] [Commented] (HUDI-4101) BucketIndexPartitioner should take partition path for better dispersion

2022-05-16 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537884#comment-17537884 ] Danny Chen commented on HUDI-4101: -- Fixed via master branch: d52d13302db2eba94b25ddb680c58682760076e1 >

[hudi] branch master updated (fdd96cc97e -> d52d13302d)

2022-05-16 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from fdd96cc97e [HUDI-4104] DeltaWriteProfile includes the pending compaction file slice when deciding small buckets

[jira] [Commented] (HUDI-4104) DeltaWriteProfile includes the pending compaction file slice when deciding small buckets

2022-05-16 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537883#comment-17537883 ] Danny Chen commented on HUDI-4104: -- Fixed via master branch: fdd96cc97ef6a5033b9657e22278bdffd71a41f3 >

[GitHub] [hudi] danny0405 merged pull request #5590: [HUDI-4101] BucketIndexPartitioner should take partition path for bet…

2022-05-16 Thread GitBox
danny0405 merged PR #5590: URL: https://github.com/apache/hudi/pull/5590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Resolved] (HUDI-4104) DeltaWriteProfile includes the pending compaction file slice when deciding small buckets

2022-05-16 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-4104. -- > DeltaWriteProfile includes the pending compaction file slice when deciding > small buckets >

[hudi] branch master updated (ad773b3d96 -> fdd96cc97e)

2022-05-16 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from ad773b3d96 [HUDI-3654] Preparations for hudi metastore. (#5572) add fdd96cc97e [HUDI-4104] DeltaWriteProfile

[GitHub] [hudi] danny0405 merged pull request #5594: [HUDI-4104] DeltaWriteProfile includes the pending compaction file sl…

2022-05-16 Thread GitBox
danny0405 merged PR #5594: URL: https://github.com/apache/hudi/pull/5594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] xiarixiaoyao commented on issue #5489: [SUPPORT] Feature Comment sync not working

2022-05-16 Thread GitBox
xiarixiaoyao commented on issue #5489: URL: https://github.com/apache/hudi/issues/5489#issuecomment-1128337259 @parisni sorry i can not reproduce this problem, What version of hive do you use? i used hive3.1, here is the test result ```

[GitHub] [hudi] hudi-bot commented on pull request #5588: [HUDI-4100] CTAS failed to clean up when given an illegal MANAGED table definition

2022-05-16 Thread GitBox
hudi-bot commented on PR #5588: URL: https://github.com/apache/hudi/pull/5588#issuecomment-1128330653 ## CI report: * 783261cc306301a7392e7431c5281a4848f452c5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
hudi-bot commented on PR #5532: URL: https://github.com/apache/hudi/pull/5532#issuecomment-1128330585 ## CI report: * d6fad6c3d2db39df6410d1dcbba1bd05e8da9ad0 Azure:

[jira] [Resolved] (HUDI-3085) Refactor fileId & writeHandler logic into partitioner for bulk_insert

2022-05-16 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-3085. -- > Refactor fileId & writeHandler logic into partitioner for bulk_insert >

[GitHub] [hudi] hudi-bot commented on pull request #5588: [HUDI-4100] CTAS failed to clean up when given an illegal MANAGED table definition

2022-05-16 Thread GitBox
hudi-bot commented on PR #5588: URL: https://github.com/apache/hudi/pull/5588#issuecomment-1128328987 ## CI report: * 783261cc306301a7392e7431c5281a4848f452c5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
hudi-bot commented on PR #5532: URL: https://github.com/apache/hudi/pull/5532#issuecomment-1128328889 ## CI report: * d6fad6c3d2db39df6410d1dcbba1bd05e8da9ad0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
hudi-bot commented on PR #5567: URL: https://github.com/apache/hudi/pull/5567#issuecomment-1128326804 ## CI report: * 5a99503343f3f536b9c153ad4422e00eb0b3ff40 Azure:

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5572: [HUDI-3654] Preparations for hudi metastore.

2022-05-16 Thread GitBox
XuQianJin-Stars commented on code in PR #5572: URL: https://github.com/apache/hudi/pull/5572#discussion_r874291698 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java: ## @@ -459,6 +459,9 @@ private Stream

[GitHub] [hudi] xiarixiaoyao commented on pull request #3391: [HUDI-83] Fix Timestamp/Date type read by Hive3

2022-05-16 Thread GitBox
xiarixiaoyao commented on PR #3391: URL: https://github.com/apache/hudi/pull/3391#issuecomment-1128319931 @cdmikechen have some questions 1) Do we really need HudiAvroParquetInputFormat; how about modify HoodieRealtimeRecordReaderUtils.avroToArrayWritable directly 2) Do we support

[GitHub] [hudi] XuQianJin-Stars commented on issue #5586: [SUPPORT] 0.11.0 SparkSQL ParseException occurs in 0.11.0 when creating view with `timestamp as of`

2022-05-16 Thread GitBox
XuQianJin-Stars commented on issue #5586: URL: https://github.com/apache/hudi/issues/5586#issuecomment-1128319091 > > hi @gnailJC Thanks for the question, now `timestamp as of` only supports `table` operations, not `view` related operations yet. If there is a business scenario requirement,

[hudi] branch master updated: [HUDI-3654] Preparations for hudi metastore. (#5572)

2022-05-16 Thread forwardxu
This is an automated email from the ASF dual-hosted git repository. forwardxu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new ad773b3d96 [HUDI-3654] Preparations for hudi

[GitHub] [hudi] XuQianJin-Stars merged pull request #5572: [HUDI-3654] Preparations for hudi metastore.

2022-05-16 Thread GitBox
XuQianJin-Stars merged PR #5572: URL: https://github.com/apache/hudi/pull/5572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Assigned] (HUDI-4103) TestCreateTable failed CTAS when indicating hoodie.database.name in table properties

2022-05-16 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu reassigned HUDI-4103: Assignee: 董可伦 (was: Forward Xu) > TestCreateTable failed CTAS when indicating

[jira] [Assigned] (HUDI-4103) TestCreateTable failed CTAS when indicating hoodie.database.name in table properties

2022-05-16 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu reassigned HUDI-4103: Assignee: Forward Xu > TestCreateTable failed CTAS when indicating hoodie.database.name in table

[GitHub] [hudi] zhangyue19921010 commented on pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
zhangyue19921010 commented on PR #5567: URL: https://github.com/apache/hudi/pull/5567#issuecomment-1128310618 Hi @leesf Thanks a lot for your review. Really appreciate it ! All comments are addressed. PTAL -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
zhangyue19921010 commented on code in PR #5567: URL: https://github.com/apache/hudi/pull/5567#discussion_r874283970 ## rfc/rfc-53/rfc-53.md: ## @@ -0,0 +1,120 @@ + +# RFC-53: Use Lock-Free Message Queue Improving Hoodie Writing Efficiency + + +## Proposers +@zhangyue19921010 +

[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
zhangyue19921010 commented on code in PR #5567: URL: https://github.com/apache/hudi/pull/5567#discussion_r874282784 ## rfc/rfc-53/rfc-53.md: ## @@ -0,0 +1,120 @@ + +# RFC-53: Use Lock-Free Message Queue Improving Hoodie Writing Efficiency + + +## Proposers +@zhangyue19921010 +

[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
zhangyue19921010 commented on code in PR #5567: URL: https://github.com/apache/hudi/pull/5567#discussion_r874281672 ## rfc/rfc-53/rfc-53.md: ## @@ -0,0 +1,120 @@ + +# RFC-53: Use Lock-Free Message Queue Improving Hoodie Writing Efficiency + + +## Proposers +@zhangyue19921010 +

[GitHub] [hudi] hudi-bot commented on pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
hudi-bot commented on PR #5567: URL: https://github.com/apache/hudi/pull/5567#issuecomment-1128306460 ## CI report: * 5a99503343f3f536b9c153ad4422e00eb0b3ff40 Azure:

[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5567: [RFC-53] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency

2022-05-16 Thread GitBox
zhangyue19921010 commented on code in PR #5567: URL: https://github.com/apache/hudi/pull/5567#discussion_r874281338 ## rfc/rfc-53/rfc-53.md: ## @@ -0,0 +1,120 @@ + +# RFC-53: Use Lock-Free Message Queue Improving Hoodie Writing Efficiency + + +## Proposers +@zhangyue19921010 +

[GitHub] [hudi] huberylee commented on a diff in pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
huberylee commented on code in PR #5532: URL: https://github.com/apache/hudi/pull/5532#discussion_r874281023 ## hudi-sync/hudi-adb-sync/src/main/java/org/apache/hudi/sync/adb/HoodieAdbJdbcClient.java: ## @@ -0,0 +1,440 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] huberylee commented on a diff in pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
huberylee commented on code in PR #5532: URL: https://github.com/apache/hudi/pull/5532#discussion_r874277633 ## hudi-sync/hudi-adb-sync/src/main/java/org/apache/hudi/sync/adb/AdbSyncTool.java: ## @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] huberylee commented on a diff in pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
huberylee commented on code in PR #5532: URL: https://github.com/apache/hudi/pull/5532#discussion_r874277079 ## hudi-sync/hudi-adb-sync/src/main/java/org/apache/hudi/sync/adb/AdbSyncTool.java: ## @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] huberylee commented on a diff in pull request #5532: [HUDI-3985] Refactor DLASyncTool to support read hoodie table as spark datasource table

2022-05-16 Thread GitBox
huberylee commented on code in PR #5532: URL: https://github.com/apache/hudi/pull/5532#discussion_r874276530 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala: ## @@ -26,11 +26,11 @@ import

[GitHub] [hudi] leesf commented on a diff in pull request #5572: [HUDI-3654] Preparations for hudi metastore.

2022-05-16 Thread GitBox
leesf commented on code in PR #5572: URL: https://github.com/apache/hudi/pull/5572#discussion_r874273449 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java: ## @@ -459,6 +459,9 @@ private Stream getCommitInstantsToArchive() {

[jira] [Closed] (HUDI-4016) Prepare a document to list all tests to be done as part of release certification

2022-05-16 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-4016. Fix Version/s: (was: 0.12.0) Resolution: Done > Prepare a document to list all tests to be done

[GitHub] [hudi] hudi-bot commented on pull request #5597: [HUDI-4107] Added --sync-tool-classes config option in HoodieMultiTableDeltaStreamer

2022-05-16 Thread GitBox
hudi-bot commented on PR #5597: URL: https://github.com/apache/hudi/pull/5597#issuecomment-1128225742 ## CI report: * d312f8b888a463981dcc2abbb5d89cd99d584c62 Azure:

[jira] [Updated] (HUDI-4107) Introduce --sync-tool-classes parameter in HoodieMultiTableDeltaStreamer

2022-05-16 Thread Kumud Kumar Srivatsava Tirupati (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumud Kumar Srivatsava Tirupati updated HUDI-4107: -- Status: In Progress (was: Open) > Introduce

[GitHub] [hudi] kumudkumartirupati opened a new pull request, #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer

2022-05-16 Thread GitBox
kumudkumartirupati opened a new pull request, #5598: URL: https://github.com/apache/hudi/pull/5598 ## What is the purpose of the pull request Updates the missing content of DeltaStreamer documentation of v0.11.0. ## Brief change log * Adds the missing `--enable-sync` config

[GitHub] [hudi] hudi-bot commented on pull request #5596: Bump xercesImpl from 2.9.1 to 2.12.2

2022-05-16 Thread GitBox
hudi-bot commented on PR #5596: URL: https://github.com/apache/hudi/pull/5596#issuecomment-1128150591 ## CI report: * 2b40485c1ae9fafb28bc89af95ad24b1f341ff01 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5597: [HUDI-4107] Added --sync-tool-classes config option in HoodieMultiTableDeltaStreamer

2022-05-16 Thread GitBox
hudi-bot commented on PR #5597: URL: https://github.com/apache/hudi/pull/5597#issuecomment-1128147755 ## CI report: * d312f8b888a463981dcc2abbb5d89cd99d584c62 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5597: [HUDI-4107] Added --sync-tool-classes config option in HoodieMultiTableDeltaStreamer

2022-05-16 Thread GitBox
hudi-bot commented on PR #5597: URL: https://github.com/apache/hudi/pull/5597#issuecomment-1128144717 ## CI report: * d312f8b888a463981dcc2abbb5d89cd99d584c62 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-4107) Introduce --sync-tool-classes parameter in HoodieMultiTableDeltaStreamer

2022-05-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4107: - Labels: newbie pull-request-available (was: newbie) > Introduce --sync-tool-classes parameter in

[GitHub] [hudi] kumudkumartirupati opened a new pull request, #5597: [HUDI-4107] Added --sync-tool-classes config option in HoodieMultiTableDeltaStreamer

2022-05-16 Thread GitBox
kumudkumartirupati opened a new pull request, #5597: URL: https://github.com/apache/hudi/pull/5597 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is

[GitHub] [hudi] hudi-bot commented on pull request #5577: [WIP][HUDI-3991] Add hudi-integ-test slim bundle

2022-05-16 Thread GitBox
hudi-bot commented on PR #5577: URL: https://github.com/apache/hudi/pull/5577#issuecomment-1128100190 ## CI report: * e1988ddfac9458ffccf52a95b3155cc81edd267d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5572: [HUDI-3654] Preparations for hudi metastore.

2022-05-16 Thread GitBox
hudi-bot commented on PR #5572: URL: https://github.com/apache/hudi/pull/5572#issuecomment-1128090655 ## CI report: * 96bf68fdf1ed38c46ee3b5039b5f795f22285660 Azure:

[jira] [Closed] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-2875. - Resolution: Fixed > Concurrent call to HoodieMergeHandler cause parquet corruption >

[jira] [Updated] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2875: -- Fix Version/s: 0.11.1 0.12.0 (was: 0.11.0) >

[jira] [Updated] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2875: -- Status: Open (was: In Progress) > Concurrent call to HoodieMergeHandler cause parquet

[jira] [Closed] (HUDI-3849) AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3849. - Resolution: Fixed > AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration >

[jira] [Updated] (HUDI-3849) AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3849: -- Fix Version/s: 0.11.1 0.12.0 > AvroDeserializer supports

[jira] [Closed] (HUDI-4053) Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOptimized

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4053. - Fix Version/s: 0.11.1 Resolution: Fixed > Flaky

[jira] [Closed] (HUDI-4044) When reading data from flink-hudi to external storage, the result is incorrect

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4044. - Resolution: Fixed > When reading data from flink-hudi to external storage, the result is

[jira] [Closed] (HUDI-4055) use loop replace recursive call in ratelimiter

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4055. - Fix Version/s: 0.11.1 Resolution: Fixed > use loop replace recursive call in

[jira] [Closed] (HUDI-4079) Supports showing table comment for hudi with spark3

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4079. - Fix Version/s: 0.11.1 0.12.0 Resolution: Fixed > Supports

[jira] [Closed] (HUDI-4003) Flink offline compaction may cause NPE when log file only contain delete opereation

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4003. - Fix Version/s: 0.11.0 0.12.0 Resolution: Fixed > Flink offline

[jira] [Closed] (HUDI-4085) TestHoodieDeltastreamer is flaky in CI

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4085. - Resolution: Fixed > TestHoodieDeltastreamer is flaky in CI >

[jira] [Closed] (HUDI-3336) Configurations transferred through Flink SQL cannot take effect

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3336. - Fix Version/s: 0.11.1 (was: 0.11.0) Resolution: Fixed >

[GitHub] [hudi] hudi-bot commented on pull request #5522: [HUDI-3378][HUDI-3379][HUDI-3381] Rebasing usages of HoodieRecordPayload and raw Avro payload to rely on HoodieRecord instead

2022-05-16 Thread GitBox
hudi-bot commented on PR #5522: URL: https://github.com/apache/hudi/pull/5522#issuecomment-1127999109 ## CI report: * 986960516f86a1426725141cd7cb25e84d260020 UNKNOWN * c5fb81a0b229ded9a2b925790366b62f1bf7ade9 UNKNOWN * a77d750b95f7fbb37675923467ca4bd6b88b006f UNKNOWN *

[jira] [Closed] (HUDI-4097) Add table information to job status of spark job

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4097. - Fix Version/s: 0.11.1 Resolution: Fixed > Add table information to job status of

[jira] [Closed] (HUDI-3980) Suport kerberos hbase index

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3980. - Resolution: Fixed > Suport kerberos hbase index > --- > >

[jira] [Commented] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537713#comment-17537713 ] sivabalan narayanan commented on HUDI-3123: --- [~yuweixiao] : if there are any follow ups, do

[jira] [Updated] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3123: -- Fix Version/s: 0.11.1 0.12.0 > Consistent hashing index for

[jira] [Closed] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3123. - Resolution: Fixed > Consistent hashing index for upsert/insert write path >

[jira] [Closed] (HUDI-4001) "hoodie.datasource.write.operation" from table config should not be used as write operation

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4001. - Resolution: Fixed > "hoodie.datasource.write.operation" from table config should not be

[jira] [Closed] (HUDI-4098) Metadata table heartbeat for instant has expired, last heartbeat 0

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4098. - Resolution: Fixed > Metadata table heartbeat for instant has expired, last heartbeat 0 >

[jira] [Assigned] (HUDI-4098) Metadata table heartbeat for instant has expired, last heartbeat 0

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-4098: - Assignee: Danny Chen > Metadata table heartbeat for instant has expired, last

[jira] [Created] (HUDI-4107) Introduce --sync-tool-classes parameter in HoodieMultiTableDeltaStreamer

2022-05-16 Thread Kumud Kumar Srivatsava Tirupati (Jira)
Kumud Kumar Srivatsava Tirupati created HUDI-4107: - Summary: Introduce --sync-tool-classes parameter in HoodieMultiTableDeltaStreamer Key: HUDI-4107 URL:

[jira] [Reopened] (HUDI-4098) Metadata table heartbeat for instant has expired, last heartbeat 0

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-4098: --- > Metadata table heartbeat for instant has expired, last heartbeat 0 >

[jira] [Closed] (HUDI-4103) TestCreateTable failed CTAS when indicating hoodie.database.name in table properties

2022-05-16 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-4103. - Fix Version/s: 0.11.1 Resolution: Fixed > TestCreateTable failed CTAS when

[GitHub] [hudi] hudi-bot commented on pull request #5577: [WIP][HUDI-3991] Add hudi-integ-test slim bundle

2022-05-16 Thread GitBox
hudi-bot commented on PR #5577: URL: https://github.com/apache/hudi/pull/5577#issuecomment-1127983327 ## CI report: * 1cff39f163ab3febeaec31ab3b0b67caab0cea4b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5596: Bump xercesImpl from 2.9.1 to 2.12.2

2022-05-16 Thread GitBox
hudi-bot commented on PR #5596: URL: https://github.com/apache/hudi/pull/5596#issuecomment-1127946819 ## CI report: * 2b40485c1ae9fafb28bc89af95ad24b1f341ff01 Azure:

[GitHub] [hudi] Markc2001 commented on issue #5595: [SUPPORT]

2022-05-16 Thread GitBox
Markc2001 commented on issue #5595: URL: https://github.com/apache/hudi/issues/5595#issuecomment-1127944033 Hi, can you kindly remove the IP, and source code related to Stateauto please, it appears there are a number of references to systems and Stateauto when reviewing the site and

[GitHub] [hudi] hudi-bot commented on pull request #5596: Bump xercesImpl from 2.9.1 to 2.12.2

2022-05-16 Thread GitBox
hudi-bot commented on PR #5596: URL: https://github.com/apache/hudi/pull/5596#issuecomment-1127943240 ## CI report: * 2b40485c1ae9fafb28bc89af95ad24b1f341ff01 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

  1   2   3   4   >