[GitHub] [hudi] TengHuo commented on pull request #8161: [MINOR][DOCS] fix HiveMetastore lock provider package name

2023-03-13 Thread via GitHub
TengHuo commented on PR #8161: URL: https://github.com/apache/hudi/pull/8161#issuecomment-1467471095 Thanks guys -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [hudi] beyond1920 commented on a diff in pull request #8101: [HUDI-5879] Extends evaluators to support evaluate based on column values

2023-03-13 Thread via GitHub
beyond1920 commented on code in PR #8101: URL: https://github.com/apache/hudi/pull/8101#discussion_r1135032944 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java: ## @@ -156,12 +156,20 @@ public static Evaluator fromExpression(Call

[GitHub] [hudi] beyond1920 commented on a diff in pull request #8101: [HUDI-5879] Extends evaluators to support evaluate based on column values

2023-03-13 Thread via GitHub
beyond1920 commented on code in PR #8101: URL: https://github.com/apache/hudi/pull/8101#discussion_r1135032944 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java: ## @@ -156,12 +156,20 @@ public static Evaluator fromExpression(Call

[GitHub] [hudi] beyond1920 commented on a diff in pull request #8101: [HUDI-5879] Extends evaluators to support evaluate based on column values

2023-03-13 Thread via GitHub
beyond1920 commented on code in PR #8101: URL: https://github.com/apache/hudi/pull/8101#discussion_r1135032944 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java: ## @@ -156,12 +156,20 @@ public static Evaluator fromExpression(Call

[GitHub] [hudi] huangxiaopingRD commented on a diff in pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-13 Thread via GitHub
huangxiaopingRD commented on code in PR #8165: URL: https://github.com/apache/hudi/pull/8165#discussion_r1135025581 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientUtil.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [hudi] danny0405 commented on a diff in pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-13 Thread via GitHub
danny0405 commented on code in PR #8165: URL: https://github.com/apache/hudi/pull/8165#discussion_r1134972715 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientUtil.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] [hudi] danny0405 commented on a diff in pull request #8101: [HUDI-5879] Extends evaluators to support evaluate based on column values

2023-03-13 Thread via GitHub
danny0405 commented on code in PR #8101: URL: https://github.com/apache/hudi/pull/8101#discussion_r1134970324 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java: ## @@ -156,12 +156,20 @@ public static Evaluator fromExpression(CallE

[hudi] branch master updated: [MINOR] Ignoring warn msg for timeline server for metadata table (#8168)

2023-03-13 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 38e4078d23b [MINOR] Ignoring warn msg for timeline

[GitHub] [hudi] yihua merged pull request #8168: [MINOR] Ignoring warn msg for timeline server for mdt

2023-03-13 Thread via GitHub
yihua merged PR #8168: URL: https://github.com/apache/hudi/pull/8168 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1467336671 ## CI report: * 37322403cb6352bc74c008e1a80c71a953ff0602 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1570

[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #7463: [HUDI-2608] add json to avro converter

2023-03-13 Thread via GitHub
the-other-tim-brown commented on code in PR #7463: URL: https://github.com/apache/hudi/pull/7463#discussion_r1134910934 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/JsonSchemaToAvroSchemaConverter.java: ## @@ -0,0 +1,174 @@ +/* + * Licensed to the Apache Soft

[GitHub] [hudi] Guanpx closed pull request #4580: [MINOR][DOCS] Impala does NOT SUPPORT mor table with hudi 0.10-release

2023-03-13 Thread via GitHub
Guanpx closed pull request #4580: [MINOR][DOCS] Impala does NOT SUPPORT mor table with hudi 0.10-release URL: https://github.com/apache/hudi/pull/4580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] bvaradar commented on pull request #7984: hash partition in spark datasoure

2023-03-13 Thread via GitHub
bvaradar commented on PR #7984: URL: https://github.com/apache/hudi/pull/7984#issuecomment-1467287281 @lvhu-goodluck : Can you add the link to mini-RFC when it is ready ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [hudi] bvaradar commented on pull request #6151: [HUDI-4429] Make Spark 3.1 the default profile

2023-03-13 Thread via GitHub
bvaradar commented on PR #6151: URL: https://github.com/apache/hudi/pull/6151#issuecomment-1467286004 @yihua : should this PR be closed in light of https://github.com/apache/hudi/pull/6117 ? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [hudi] DavidZ1 commented on issue #8071: [SUPPORT]How to improve the speed of Flink writing to hudi ?

2023-03-13 Thread via GitHub
DavidZ1 commented on issue #8071: URL: https://github.com/apache/hudi/issues/8071#issuecomment-1467274664 Thx,I will read later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [hudi] bvaradar commented on pull request #5639: [HUDI-2516] Upgrade JUnit to 5.8.2

2023-03-13 Thread via GitHub
bvaradar commented on PR #5639: URL: https://github.com/apache/hudi/pull/5639#issuecomment-1467270423 @liujinhui1994 : Can you take a look at @yihua 's comment and let us know if this PR is still valid. Will timeout and close after 2 days. Thanks -- This is an automated me

[GitHub] [hudi] bvaradar commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit

2023-03-13 Thread via GitHub
bvaradar commented on code in PR #6926: URL: https://github.com/apache/hudi/pull/6926#discussion_r1134846459 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java: ## @@ -99,6 +99,206 @@ private static Stream argumentsForTest

[GitHub] [hudi] huangxiaopingRD commented on a diff in pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-13 Thread via GitHub
huangxiaopingRD commented on code in PR #8165: URL: https://github.com/apache/hudi/pull/8165#discussion_r1134832405 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientUtil.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [hudi] huangxiaopingRD commented on a diff in pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-13 Thread via GitHub
huangxiaopingRD commented on code in PR #8165: URL: https://github.com/apache/hudi/pull/8165#discussion_r1134832405 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientUtil.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1467252776 ## CI report: * 961aafcc251a8ed6bb18cc40c87365aa0a0924eb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1570

[jira] [Closed] (HUDI-5864) Update release notes regarding the HoodieMetadataFileSystemView regression

2023-03-13 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-5864. Resolution: Fixed > Update release notes regarding the HoodieMetadataFileSystemView regression > ---

[jira] [Commented] (HUDI-5463) Apply rollback commits from data table as rollbacks in MDT instead of Delta commit

2023-03-13 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699906#comment-17699906 ] Danny Chen commented on HUDI-5463: -- The failed instant in DT may never written into the M

[GitHub] [hudi] danny0405 commented on a diff in pull request #8163: [HUDI-5921] Partition path should be considered in BucketIndexConcurrentFileWritesConflictResolutionStrategy

2023-03-13 Thread via GitHub
danny0405 commented on code in PR #8163: URL: https://github.com/apache/hudi/pull/8163#discussion_r1134812808 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/TransactionUtils.java: ## @@ -78,7 +79,13 @@ public static Option resolveWriteConflictIfAny

[hudi] branch master updated (c48db6d8ebe -> c392fb7ff8c)

2023-03-13 Thread vbalaji
This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from c48db6d8ebe [HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex (#6458) add c392fb7ff8c [HUDI-4630] Add

[GitHub] [hudi] bvaradar merged pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer

2023-03-13 Thread via GitHub
bvaradar merged PR #6726: URL: https://github.com/apache/hudi/pull/6726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated (1b756d62e19 -> c48db6d8ebe)

2023-03-13 Thread vbalaji
This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 1b756d62e19 [HUDI-5915] Fixed load ckpMeatadata error when using minio (#8149) add c48db6d8ebe [HUDI-4675] add un

[GitHub] [hudi] bvaradar merged pull request #6458: [HUDI-4675] add unittest for RebalancedSparkHoodieHBaseIndex

2023-03-13 Thread via GitHub
bvaradar merged PR #6458: URL: https://github.com/apache/hudi/pull/6458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[GitHub] [hudi] danny0405 commented on issue #8164: [SUPPORT] Flink KryoSerializer cannot be cast to class org.apache.flink.table.runtime.typeutils.AbstractRowDataSerializer

2023-03-13 Thread via GitHub
danny0405 commented on issue #8164: URL: https://github.com/apache/hudi/issues/8164#issuecomment-1467229462 The input element of the `SortOperator` should be a `RowData`, because the serializer is hard coded into `BinaryRowDataSerializer`. -- This is an automated message from the Apache G

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1467213865 ## CI report: * d1eee370f542d5160632f4fdb5a7017e733f2cd7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[hudi] branch asf-site updated: [MINOR][DOCS] Fix HiveMetastore lock provider package name (#8161)

2023-03-13 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 64de8d0d9ba [MINOR][DOCS] Fix HiveMetastore

[GitHub] [hudi] danny0405 merged pull request #8161: [MINOR][DOCS] fix HiveMetastore lock provider package name

2023-03-13 Thread via GitHub
danny0405 merged PR #8161: URL: https://github.com/apache/hudi/pull/8161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[hudi] branch asf-site updated: [MINOR][DOCS] HMS mode with Spark datasource is already supported. (#8162)

2023-03-13 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 9b3b372cfae [MINOR][DOCS] HMS mode with Spa

[GitHub] [hudi] danny0405 merged pull request #8162: [MINOR][DOCS] HMS mode with Spark datasource is already supported.

2023-03-13 Thread via GitHub
danny0405 merged PR #8162: URL: https://github.com/apache/hudi/pull/8162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] danny0405 commented on issue #8071: [SUPPORT]How to improve the speed of Flink writing to hudi ?

2023-03-13 Thread via GitHub
danny0405 commented on issue #8071: URL: https://github.com/apache/hudi/issues/8071#issuecomment-1467208193 Maybe you can just take a reference of this doc: https://www.yuque.com/yuzhao-my9fz/kb/flqll8?#cJu7y -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1467207114 ## CI report: * d1eee370f542d5160632f4fdb5a7017e733f2cd7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[jira] [Updated] (HUDI-5915) listStatus error caused by minio storage

2023-03-13 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5915: - Fix Version/s: 0.13.1 0.14.0 > listStatus error caused by minio storage > -

[jira] [Closed] (HUDI-5915) listStatus error caused by minio storage

2023-03-13 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-5915. Resolution: Fixed Fixed via master branch: 1b756d62e19e0d3bd81ca61327758b90fbc6eddd > listStatus error caus

[hudi] branch master updated (07c6d8b1845 -> 1b756d62e19)

2023-03-13 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 07c6d8b1845 [MINOR] Use ExecutorFactory in BootstrapHandler (#7808) add 1b756d62e19 [HUDI-5915] Fixed load ckpMe

[GitHub] [hudi] danny0405 merged pull request #8149: [HUDI-5915] Fixed load ckpMeatadata error when using minio

2023-03-13 Thread via GitHub
danny0405 merged PR #8149: URL: https://github.com/apache/hudi/pull/8149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] hudi-bot commented on pull request #8174: [HUDI-5688] Fixing read of an empty table

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8174: URL: https://github.com/apache/hudi/pull/8174#issuecomment-1467201146 ## CI report: * e3014478fed090f1e10e792d8a894349a2a083ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8173: [HUDI-5928] Fixing graceful shut down of deltastreamer with post write termination strategy

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8173: URL: https://github.com/apache/hudi/pull/8173#issuecomment-1467201102 ## CI report: * 287ca14d1d1200bf653eb21d6133ee46a1987622 UNKNOWN * 95618da8be44f9f115f68af5be8f95d1be49a2e9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[hudi] branch master updated: [MINOR] Use ExecutorFactory in BootstrapHandler (#7808)

2023-03-13 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 07c6d8b1845 [MINOR] Use ExecutorFactory in Boot

[GitHub] [hudi] danny0405 merged pull request #7808: [MINOR] use ExecutorFactory in BootstrapHandler

2023-03-13 Thread via GitHub
danny0405 merged PR #7808: URL: https://github.com/apache/hudi/pull/7808 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] danny0405 commented on a diff in pull request #8165: [HUDI-5922] Reuse IMetaStoreClient between HoodieHiveSyncClient and DDLExecutor

2023-03-13 Thread via GitHub
danny0405 commented on code in PR #8165: URL: https://github.com/apache/hudi/pull/8165#discussion_r1134768808 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientUtil.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] [hudi] danny0405 commented on a diff in pull request #8128: [HUDI-5782] Tweak defaults and remove unnecessary configs after config review

2023-03-13 Thread via GitHub
danny0405 commented on code in PR #8128: URL: https://github.com/apache/hudi/pull/8128#discussion_r1134765337 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java: ## @@ -297,21 +295,15 @@ private FlinkOptions() { .booleanType(

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1467124146 ## CI report: * d1eee370f542d5160632f4fdb5a7017e733f2cd7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8171: [HUDI-5926] Improve cleaner parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8171: URL: https://github.com/apache/hudi/pull/8171#issuecomment-1467111032 ## CI report: * 90d6399ae140f157972f46f2901366f79ed4b1bc UNKNOWN * 9b6d4e1a92d727aadb765ef0e35d037786f8a7b4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1467072461 ## CI report: * 10777af559d8be0a0c421ebfb98f001501638aa5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1563

[jira] [Updated] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5929: Story Points: 2 (was: 1) > Automatically infer key generator type > --

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1467063374 ## CI report: * 10777af559d8be0a0c421ebfb98f001501638aa5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1563

[GitHub] [hudi] hudi-bot commented on pull request #8170: [HUDI-5925] Improve bootstrap parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8170: URL: https://github.com/apache/hudi/pull/8170#issuecomment-1467052167 ## CI report: * 1d584799281db4dde73542df33ffd348488fe7bf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1467051791 ## CI report: * 10777af559d8be0a0c421ebfb98f001501638aa5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1563

[GitHub] [hudi] danny0405 commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-13 Thread via GitHub
danny0405 commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1467041684 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[jira] [Updated] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5929: Epic Link: HUDI-5738 > Automatically infer key generator type > -- > >

[jira] [Updated] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5929: Summary: Automatically infer key generator type (was: Auto infer key generator type) > Automatically infer

[jira] [Created] (HUDI-5929) Auto infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5929: --- Summary: Auto infer key generator type Key: HUDI-5929 URL: https://issues.apache.org/jira/browse/HUDI-5929 Project: Apache Hudi Issue Type: Improvement Rep

[jira] [Updated] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5929: Story Points: 1 > Automatically infer key generator type > -- > >

[jira] [Assigned] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5929: --- Assignee: Ethan Guo > Automatically infer key generator type > --

[jira] [Updated] (HUDI-5929) Automatically infer key generator type

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5929: Fix Version/s: 0.14.0 > Automatically infer key generator type > -- > >

[jira] [Updated] (HUDI-5713) Add advanced property for configs

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5713: Description: Add advanced property to configs to indicate that new users don't need to care about them first

[jira] [Closed] (HUDI-5713) Add advanced property for configs

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-5713. --- Resolution: Fixed > Add advanced property for configs > - > >

[jira] [Updated] (HUDI-5713) Add advanced property for configs

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5713: Summary: Add advanced property for configs (was: Add essential property to configs) > Add advanced propert

[GitHub] [hudi] hudi-bot commented on pull request #8169: [HUDI-5924] Fixing cli clean command to trim down a subset based on start and end

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8169: URL: https://github.com/apache/hudi/pull/8169#issuecomment-1466987381 ## CI report: * d9c4e12c3215cc02f35f66809fdf22aef2d84890 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8174: [HUDI-5688] Fixing read of an empty table

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8174: URL: https://github.com/apache/hudi/pull/8174#issuecomment-1466930717 ## CI report: * e3014478fed090f1e10e792d8a894349a2a083ed Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8174: [HUDI-5688] Fixing read of an empty table

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8174: URL: https://github.com/apache/hudi/pull/8174#issuecomment-1466920946 ## CI report: * e3014478fed090f1e10e792d8a894349a2a083ed UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8173: [HUDI-5928] Fixing graceful shut down of deltastreamer with post write termination strategy

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8173: URL: https://github.com/apache/hudi/pull/8173#issuecomment-1466920895 ## CI report: * 287ca14d1d1200bf653eb21d6133ee46a1987622 UNKNOWN * 95618da8be44f9f115f68af5be8f95d1be49a2e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8173: [HUDI-5928] Fixing graceful shut down of deltastreamer with post write termination strategy

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8173: URL: https://github.com/apache/hudi/pull/8173#issuecomment-1466911600 ## CI report: * 287ca14d1d1200bf653eb21d6133ee46a1987622 UNKNOWN * 95618da8be44f9f115f68af5be8f95d1be49a2e9 UNKNOWN Bot commands @hudi-bot supports the following

[GitHub] [hudi] nsivabalan opened a new pull request, #8174: [HUDI-5688] Fixing read of an empty table

2023-03-13 Thread via GitHub
nsivabalan opened a new pull request, #8174: URL: https://github.com/apache/hudi/pull/8174 ### Change Logs Read of an empty table on some occasions might result in NPE. Fixing it in this patch. Hudi will return an EmptyRelation with null StructType. ### Impact Improves

[GitHub] [hudi] hudi-bot commented on pull request #8173: [HUDI-5928] Fixing graceful shut down of deltastreamer with post write termination strategy

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8173: URL: https://github.com/apache/hudi/pull/8173#issuecomment-1466902709 ## CI report: * 287ca14d1d1200bf653eb21d6133ee46a1987622 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] nsivabalan commented on issue #8146: [SUPPORT] Running `--continuous` mode with MultiTable and PostWriteTerminationStrategy seems to leave Spark job hanging

2023-03-13 Thread via GitHub
nsivabalan commented on issue #8146: URL: https://github.com/apache/hudi/issues/8146#issuecomment-1466857916 hey @sydneyhoran : https://github.com/apache/hudi/pull/8173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[jira] [Updated] (HUDI-5928) PostWriteTermination Strategy does not shutdown the spark job completely

2023-03-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5928: - Labels: pull-request-available (was: ) > PostWriteTermination Strategy does not shutdown the spar

[GitHub] [hudi] nsivabalan opened a new pull request, #8173: [HUDI-5928] Fixing graceful shut down of deltastreamer with post write termination strategy

2023-03-13 Thread via GitHub
nsivabalan opened a new pull request, #8173: URL: https://github.com/apache/hudi/pull/8173 ### Change Logs Looks like table services are not propoerly shut down when post write termination kicks in. This patch fixes the flow, where in regular ingestion and any table services are shut

[jira] [Assigned] (HUDI-5928) PostWriteTermination Strategy does not shutdown the spark job completely

2023-03-13 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-5928: - Assignee: sivabalan narayanan > PostWriteTermination Strategy does not shutdown t

[jira] [Created] (HUDI-5928) PostWriteTermination Strategy does not shutdown the spark job completely

2023-03-13 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-5928: - Summary: PostWriteTermination Strategy does not shutdown the spark job completely Key: HUDI-5928 URL: https://issues.apache.org/jira/browse/HUDI-5928 Projec

[jira] [Updated] (HUDI-5928) PostWriteTermination Strategy does not shutdown the spark job completely

2023-03-13 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5928: -- Fix Version/s: 0.13.1 > PostWriteTermination Strategy does not shutdown the spark job co

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1466816853 ## CI report: * d1eee370f542d5160632f4fdb5a7017e733f2cd7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8171: [HUDI-5926] Improve cleaner parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8171: URL: https://github.com/apache/hudi/pull/8171#issuecomment-1466816788 ## CI report: * 90d6399ae140f157972f46f2901366f79ed4b1bc UNKNOWN * 9b6d4e1a92d727aadb765ef0e35d037786f8a7b4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8168: [MINOR] Ignoring warn msg for timeline server for mdt

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8168: URL: https://github.com/apache/hudi/pull/8168#issuecomment-1466816662 ## CI report: * 2bf836fd518847ab01bbcda2db89396220b80f30 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[GitHub] [hudi] hudi-bot commented on pull request #8171: [HUDI-5926] Improve cleaner parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8171: URL: https://github.com/apache/hudi/pull/8171#issuecomment-1466738495 ## CI report: * 90d6399ae140f157972f46f2901366f79ed4b1bc UNKNOWN * 9b6d4e1a92d727aadb765ef0e35d037786f8a7b4 UNKNOWN Bot commands @hudi-bot supports the following

[GitHub] [hudi] hudi-bot commented on pull request #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8172: URL: https://github.com/apache/hudi/pull/8172#issuecomment-1466738671 ## CI report: * d1eee370f542d5160632f4fdb5a7017e733f2cd7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8170: [HUDI-5925] Improve bootstrap parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8170: URL: https://github.com/apache/hudi/pull/8170#issuecomment-1466738368 ## CI report: * 1d584799281db4dde73542df33ffd348488fe7bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1569

[jira] [Updated] (HUDI-5927) Improve parallelism of deleting invalid files

2023-03-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5927: - Labels: pull-request-available (was: ) > Improve parallelism of deleting invalid files >

[GitHub] [hudi] hudi-bot commented on pull request #8171: [HUDI-5926] Improve cleaner parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8171: URL: https://github.com/apache/hudi/pull/8171#issuecomment-1466715369 ## CI report: * 90d6399ae140f157972f46f2901366f79ed4b1bc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8170: [HUDI-5925] Improve bootstrap parallelism

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8170: URL: https://github.com/apache/hudi/pull/8170#issuecomment-1466715259 ## CI report: * 1d584799281db4dde73542df33ffd348488fe7bf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] yihua opened a new pull request, #8172: [HUDI-5927] Improve parallelism of deleting invalid files

2023-03-13 Thread via GitHub
yihua opened a new pull request, #8172: URL: https://github.com/apache/hudi/pull/8172 ### Change Logs This PR improves the parallelism of deleting invalid files when finalizing the write, so that the file deletion is parallelized at the file level instead of the partition level.

[jira] [Updated] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5926: - Labels: pull-request-available (was: ) > Improve cleaner parallelism > --

[GitHub] [hudi] yihua opened a new pull request, #8171: [HUDI-5926] Improve cleaner parallelism

2023-03-13 Thread via GitHub
yihua opened a new pull request, #8171: URL: https://github.com/apache/hudi/pull/8171 ### Change Logs This PR fixes the cleaner execution to be parallelized at the file level instead of the partition level according to the documentation. ### Impact Improves the paralleli

[jira] [Updated] (HUDI-5925) Improve bootstrap parallelism

2023-03-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5925: - Labels: pull-request-available (was: ) > Improve bootstrap parallelism >

[GitHub] [hudi] yihua opened a new pull request, #8170: [HUDI-5925] Improve bootstrap parallelism

2023-03-13 Thread via GitHub
yihua opened a new pull request, #8170: URL: https://github.com/apache/hudi/pull/8170 ### Change Logs This PR improves the bootstrap parallelism to take the minimum of number of bootstrap paths and the configured parallelism. ### Impact Limits the bootstrap parallelism w

[GitHub] [hudi] hudi-bot commented on pull request #8167: [HUDI-5923] Fixing deletion of metadata table via cli

2023-03-13 Thread via GitHub
hudi-bot commented on PR #8167: URL: https://github.com/apache/hudi/pull/8167#issuecomment-1466676091 ## CI report: * 5d7bc0394e6a54643315ea93646c9ebab5e9ef09 UNKNOWN * c9c16294117ca99782f6dc41cdcd860669d317d8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Updated] (HUDI-5927) Improve parallelism of deleting invalid files

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5927: Epic Link: HUDI-5738 Story Points: 0.5 > Improve parallelism of deleting invalid files >

[jira] [Assigned] (HUDI-5927) Improve parallelism of deleting invalid files

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5927: --- Assignee: Ethan Guo > Improve parallelism of deleting invalid files > ---

[jira] [Updated] (HUDI-5927) Improve parallelism of deleting invalid files

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5927: Fix Version/s: 0.14.0 > Improve parallelism of deleting invalid files >

[jira] [Created] (HUDI-5927) Improve parallelism of deleting invalid files

2023-03-13 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5927: --- Summary: Improve parallelism of deleting invalid files Key: HUDI-5927 URL: https://issues.apache.org/jira/browse/HUDI-5927 Project: Apache Hudi Issue Type: Improvement

[jira] [Assigned] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5926: --- Assignee: Ethan Guo > Improve cleaner parallelism > --- > > K

[jira] [Updated] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5926: Story Points: 0.5 > Improve cleaner parallelism > --- > > Key: HUDI-

[jira] [Updated] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5926: Fix Version/s: 0.14.0 > Improve cleaner parallelism > --- > > Key: H

[jira] [Updated] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5926: Epic Link: HUDI-5738 > Improve cleaner parallelism > --- > > Key: HU

[jira] [Updated] (HUDI-5925) Improve bootstrap parallelism

2023-03-13 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5925: Epic Link: HUDI-5738 > Improve bootstrap parallelism > - > > Key

[jira] [Created] (HUDI-5926) Improve cleaner parallelism

2023-03-13 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5926: --- Summary: Improve cleaner parallelism Key: HUDI-5926 URL: https://issues.apache.org/jira/browse/HUDI-5926 Project: Apache Hudi Issue Type: Improvement Repor

  1   2   >