[GitHub] [hudi] leesf commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-07-24 Thread via GitHub
leesf commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1273032213 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/ConsistentBucketIndexBulkInsertPartitionerWithRows.java: ## @@ -0,0 +1,154 @@ +/* + *

[hudi] branch master updated: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable. (#9262)

2023-07-24 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 42799c0956f [HUDI-6438] Config parameter

[GitHub] [hudi] nsivabalan merged pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
nsivabalan merged PR #9262: URL: https://github.com/apache/hudi/pull/9262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9274: URL: https://github.com/apache/hudi/pull/9274#issuecomment-1649134459 ## CI report: * 94d9dbcb05d1505d4a1d5e82dca8a8ba946f47da Azure:

[GitHub] [hudi] KnightChess commented on pull request #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
KnightChess commented on PR #9274: URL: https://github.com/apache/hudi/pull/9274#issuecomment-1649131501 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[hudi] branch master updated: [HUDI-3636] Disabling embedded timeline server with spark streaming sink (#9266)

2023-07-24 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e466daf6f4b [HUDI-3636] Disabling embedded

[GitHub] [hudi] yihua closed pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink

2023-07-24 Thread via GitHub
yihua closed pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink URL: https://github.com/apache/hudi/pull/5269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] yihua merged pull request #9266: [HUDI-3636] Disabling embedded timeline server with spark streaming sink

2023-07-24 Thread via GitHub
yihua merged PR #9266: URL: https://github.com/apache/hudi/pull/9266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #8967: [HUDI-6378] allow to delete twice for an empty table

2023-07-24 Thread via GitHub
hudi-bot commented on PR #8967: URL: https://github.com/apache/hudi/pull/8967#issuecomment-1649074473 ## CI report: * 96b14a14446288bae5070db221f8d0ea04e98d8f UNKNOWN * fdbbec8d0c523cbdce7cbb1f4d3c79f136a3f0e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8967: [HUDI-6378] allow to delete twice for an empty table

2023-07-24 Thread via GitHub
hudi-bot commented on PR #8967: URL: https://github.com/apache/hudi/pull/8967#issuecomment-1649060101 ## CI report: * 96b14a14446288bae5070db221f8d0ea04e98d8f UNKNOWN * fdbbec8d0c523cbdce7cbb1f4d3c79f136a3f0e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9278: [HUDI-6312] Rename enum values of `HollowCommitHandling`

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9278: URL: https://github.com/apache/hudi/pull/9278#issuecomment-1649047783 ## CI report: * 52e1fa26d55c6b82562c802690dc4f42ace14783 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9278: [HUDI-6312] Rename enum values of `HollowCommitHandling`

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9278: URL: https://github.com/apache/hudi/pull/9278#issuecomment-1648966927 ## CI report: * 52e1fa26d55c6b82562c802690dc4f42ace14783 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Created] (HUDI-6587) Handle hollow commit for time travel query

2023-07-24 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-6587: Summary: Handle hollow commit for time travel query Key: HUDI-6587 URL: https://issues.apache.org/jira/browse/HUDI-6587 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] xushiyan opened a new pull request, #9278: [HUDI-6312] Rename enum values of `HollowCommitHandling`

2023-07-24 Thread via GitHub
xushiyan opened a new pull request, #9278: URL: https://github.com/apache/hudi/pull/9278 ### Change Logs - Rename `HollowCommitHandling#EXCEPTION` to `HollowCommitHandling#FAIL` - Rename `HollowCommitHandling#USE_STATE_TRANSITION_TIME` to `HollowCommitHandling#USE_TRANSITION_TIME`

[GitHub] [hudi] zhuanshenbsj1 commented on a diff in pull request #7159: [HUDI-5173] Skip if there is only one file in clusteringGroup

2023-07-24 Thread via GitHub
zhuanshenbsj1 commented on code in PR #7159: URL: https://github.com/apache/hudi/pull/7159#discussion_r1025474961 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/cluster/strategy/TestSparkClusteringPlanPartitionFilter.java: ## @@ -53,9 +53,9 @@

[GitHub] [hudi] nylqd commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir

2023-07-24 Thread via GitHub
nylqd commented on issue #9269: URL: https://github.com/apache/hudi/issues/9269#issuecomment-1648923651 > > hdfs path, since the code running on yarn and my hive-site.xml is in a local dir > > I guess you are right, the hive conf dir is only valid for the catalog itself, not the

[GitHub] [hudi] ksmou commented on a diff in pull request #9229: [HUDI-6565] Spark offline compaction add failed retry mechanism

2023-07-24 Thread via GitHub
ksmou commented on code in PR #9229: URL: https://github.com/apache/hudi/pull/9229#discussion_r1272927559 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java: ## @@ -101,6 +104,12 @@ public static class Config implements Serializable { public

[GitHub] [hudi] ksmou commented on a diff in pull request #9229: [HUDI-6565] Spark offline compaction add failed retry mechanism

2023-07-24 Thread via GitHub
ksmou commented on code in PR #9229: URL: https://github.com/apache/hudi/pull/9229#discussion_r1272927559 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java: ## @@ -101,6 +104,12 @@ public static class Config implements Serializable { public

[GitHub] [hudi] weimingdiit commented on a diff in pull request #9252: [HUDI-6500] Fix bug when Using the RuntimeReplaceable function in the…

2023-07-24 Thread via GitHub
weimingdiit commented on code in PR #9252: URL: https://github.com/apache/hudi/pull/9252#discussion_r1272924805 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala: ## @@ -391,63 +392,65 @@ case class

[GitHub] [hudi] danny0405 commented on a diff in pull request #9229: [HUDI-6565] Spark offline compaction add failed retry mechanism

2023-07-24 Thread via GitHub
danny0405 commented on code in PR #9229: URL: https://github.com/apache/hudi/pull/9229#discussion_r1272922937 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java: ## @@ -101,6 +104,12 @@ public static class Config implements Serializable { public

[GitHub] [hudi] jlloh commented on issue #9256: [SUPPORT] How to do schema evolution for Flink Hudi table registered on Hive

2023-07-24 Thread via GitHub
jlloh commented on issue #9256: URL: https://github.com/apache/hudi/issues/9256#issuecomment-1648874648 Sure let me get back to you sometime later this week when I get some time to test your suggestions. Thanks for the quick support. -- This is an automated message from the Apache Git

[jira] [Updated] (HUDI-6582) Table create schema's name should be set accordingly

2023-07-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6582: - Fix Version/s: 0.14.0 > Table create schema's name should be set accordingly >

[jira] [Closed] (HUDI-6582) Table create schema's name should be set accordingly

2023-07-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6582. Resolution: Fixed Fixed via master branch: f93028a90a53a1c31d44463c7eac4dc27daac599 > Table create

[hudi] branch master updated (a32edd47c95 -> f93028a90a5)

2023-07-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from a32edd47c95 [MINOR] Optimize error display information (#9270) add f93028a90a5 [HUDI-6582] Fix the table schema

[GitHub] [hudi] danny0405 merged pull request #9272: [HUDI-6582] Ensure that default recordName/namespace is not used when…

2023-07-24 Thread via GitHub
danny0405 merged PR #9272: URL: https://github.com/apache/hudi/pull/9272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #9277: [HUDI-6558] support SQL update for no-precombine field tables

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9277: URL: https://github.com/apache/hudi/pull/9277#issuecomment-1648848964 ## CI report: * 4d363f192f951fb54799602270fb0ca16ce19d39 Azure:

[GitHub] [hudi] ksmou commented on a diff in pull request #9229: [HUDI-6565] Spark offline compaction add failed retry mechanism

2023-07-24 Thread via GitHub
ksmou commented on code in PR #9229: URL: https://github.com/apache/hudi/pull/9229#discussion_r1272908002 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java: ## @@ -101,6 +104,12 @@ public static class Config implements Serializable { public

[hudi] branch master updated (f5daa6d789d -> a32edd47c95)

2023-07-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from f5daa6d789d [MINOR] Fix CleanPlanActionExecutor logger print class (#9268) add a32edd47c95 [MINOR] Optimize

[GitHub] [hudi] danny0405 merged pull request #9270: [MINOR] Optimize error display information

2023-07-24 Thread via GitHub
danny0405 merged PR #9270: URL: https://github.com/apache/hudi/pull/9270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [MINOR] Fix CleanPlanActionExecutor logger print class (#9268)

2023-07-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f5daa6d789d [MINOR] Fix

[GitHub] [hudi] danny0405 merged pull request #9268: [MINOR]fix the log print class

2023-07-24 Thread via GitHub
danny0405 merged PR #9268: URL: https://github.com/apache/hudi/pull/9268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #9277: [HUDI-6558] support SQL update for no-precombine field tables

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9277: URL: https://github.com/apache/hudi/pull/9277#issuecomment-1648742175 ## CI report: * 4d363f192f951fb54799602270fb0ca16ce19d39 Azure:

[jira] [Created] (HUDI-6586) Add Incremental scan support to dbt

2023-07-24 Thread Vinoth Govindarajan (Jira)
Vinoth Govindarajan created HUDI-6586: - Summary: Add Incremental scan support to dbt Key: HUDI-6586 URL: https://issues.apache.org/jira/browse/HUDI-6586 Project: Apache Hudi Issue Type:

[GitHub] [hudi] hudi-bot commented on pull request #9277: [HUDI-6558] support SQL update for no-precombine field tables

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9277: URL: https://github.com/apache/hudi/pull/9277#issuecomment-1648736039 ## CI report: * 4d363f192f951fb54799602270fb0ca16ce19d39 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread kazdy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kazdy updated HUDI-6558: Description: Support SQL updates without precombine field (for COW only) is already supported in MERGE INTO (was:

[jira] [Updated] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread kazdy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kazdy updated HUDI-6558: Description: Support SQL updates without precombine field (for COW only), is already supported in MERGE INTO (was:

[jira] [Assigned] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread kazdy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kazdy reassigned HUDI-6558: --- Assignee: kazdy > Support SQL Update for CoW when no precombine field is defined >

[jira] [Updated] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread kazdy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kazdy updated HUDI-6558: Status: In Progress (was: Open) > Support SQL Update for CoW when no precombine field is defined >

[jira] [Updated] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread kazdy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kazdy updated HUDI-6558: Status: Patch Available (was: In Progress) > Support SQL Update for CoW when no precombine field is defined >

[jira] [Updated] (HUDI-6558) Support SQL Update for CoW when no precombine field is defined

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6558: - Labels: pull-request-available (was: ) > Support SQL Update for CoW when no precombine field is

[GitHub] [hudi] kazdy opened a new pull request, #9277: [HUDI-6558] support SQL update for no-precombine field tables

2023-07-24 Thread via GitHub
kazdy opened a new pull request, #9277: URL: https://github.com/apache/hudi/pull/9277 ### Change Logs Support SQL update for no-precombine field tables, improves user experience and makes it easier to start with Hudi, this is now (current master, 0.14) supported in MERGE INTO and

[GitHub] [hudi] hudi-bot commented on pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9262: URL: https://github.com/apache/hudi/pull/9262#issuecomment-1648677021 ## CI report: * 3558e69a749c890f7ddfe2d8e7719d17bea10a74 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9136: URL: https://github.com/apache/hudi/pull/9136#issuecomment-1648676701 ## CI report: * a0e7207fb19738237d56fa0060c91cb7865ae9c0 UNKNOWN * cda1e7724e6267ec471d8c318cd22703a2ecb69f UNKNOWN * 6b33d37bc57d2b5be3649590fee6767f34cccea3 Azure:

[GitHub] [hudi] Armelabdelkbir commented on issue #9213: [SUPPORT] org.apache.hudi.exception.HoodieRollbackException: Failed to rollback

2023-07-24 Thread via GitHub
Armelabdelkbir commented on issue #9213: URL: https://github.com/apache/hudi/issues/9213#issuecomment-1648661942 @ad1happy2go i added some configuration to handle multiple writers, i have multiple micro batchs with spark structured streaming, and I have this issue only when my job crashes

[GitHub] [hudi] hudi-bot commented on pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9136: URL: https://github.com/apache/hudi/pull/9136#issuecomment-1648519963 ## CI report: * a0e7207fb19738237d56fa0060c91cb7865ae9c0 UNKNOWN * cda1e7724e6267ec471d8c318cd22703a2ecb69f UNKNOWN * 91c5a055a0cc71d99aa6a3af513011667118f7b1 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9275: URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648468317 ## CI report: * 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9262: URL: https://github.com/apache/hudi/pull/9262#issuecomment-1648468178 ## CI report: * f66f9d88f825a896b5da3f081bd584dd1d0d9b96 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9136: URL: https://github.com/apache/hudi/pull/9136#issuecomment-1648467682 ## CI report: * a0e7207fb19738237d56fa0060c91cb7865ae9c0 UNKNOWN * cda1e7724e6267ec471d8c318cd22703a2ecb69f UNKNOWN * 9bc507287e35e2c2005bcd6c72f50f73a8ae96cd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9262: URL: https://github.com/apache/hudi/pull/9262#issuecomment-1648457513 ## CI report: * f66f9d88f825a896b5da3f081bd584dd1d0d9b96 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9136: URL: https://github.com/apache/hudi/pull/9136#issuecomment-1648456986 ## CI report: * a0e7207fb19738237d56fa0060c91cb7865ae9c0 UNKNOWN * cda1e7724e6267ec471d8c318cd22703a2ecb69f UNKNOWN * 9bc507287e35e2c2005bcd6c72f50f73a8ae96cd Azure:

[GitHub] [hudi] CTTY commented on a diff in pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
CTTY commented on code in PR #9136: URL: https://github.com/apache/hudi/pull/9136#discussion_r1272640363 ## hudi-common/pom.xml: ## @@ -248,6 +248,13 @@ + + org.apache.spark + spark-streaming-kafka-0-10_${scala.binary.version} + test +

[GitHub] [hudi] hudi-bot commented on pull request #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9274: URL: https://github.com/apache/hudi/pull/9274#issuecomment-1648447286 ## CI report: * 94d9dbcb05d1505d4a1d5e82dca8a8ba946f47da Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9136: [HUDI-6509] Add GitHub CI for Java 17

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9136: URL: https://github.com/apache/hudi/pull/9136#issuecomment-1648446789 ## CI report: * a0e7207fb19738237d56fa0060c91cb7865ae9c0 UNKNOWN * cda1e7724e6267ec471d8c318cd22703a2ecb69f UNKNOWN * 9bc507287e35e2c2005bcd6c72f50f73a8ae96cd Azure:

[GitHub] [hudi] amrishlal commented on a diff in pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
amrishlal commented on code in PR #9262: URL: https://github.com/apache/hudi/pull/9262#discussion_r1272632308 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala: ## @@ -1538,7 +1539,52 @@ class TestCOWDataSource extends

[GitHub] [hudi] ennox108 commented on issue #9070: [SUPPORT] Hudi Sink Connector shows broker disconnected

2023-07-24 Thread via GitHub
ennox108 commented on issue #9070: URL: https://github.com/apache/hudi/issues/9070#issuecomment-1648382447 @ad1happy2go the connector was able to create a folder. However, its still not writing any data to the folder. I am seeing the below error message

[GitHub] [hudi] hudi-bot commented on pull request #9276: Mor perf spark33

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1648371876 ## CI report: * 37d3b9365a38e8f266c1c486e9d18c9ef34be2a0 Azure:

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9262: [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' to allow for marking a newly created column as nullable.

2023-07-24 Thread via GitHub
nsivabalan commented on code in PR #9262: URL: https://github.com/apache/hudi/pull/9262#discussion_r1272587654 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala: ## @@ -1538,7 +1539,52 @@ class TestCOWDataSource extends

[GitHub] [hudi] hudi-bot commented on pull request #9276: Mor perf spark33

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1648361875 ## CI report: * 37d3b9365a38e8f266c1c486e9d18c9ef34be2a0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Commented] (HUDI-6438) Fix issue while inserting non-nullable array columns to nullable columns

2023-07-24 Thread Amrish Lal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746559#comment-17746559 ] Amrish Lal commented on HUDI-6438: -- This issue has been fixed. > Fix issue while inserting non-nullable

[jira] [Commented] (HUDI-6346) Allow duplicates by default for insert operation type

2023-07-24 Thread Amrish Lal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746556#comment-17746556 ] Amrish Lal commented on HUDI-6346: -- [~adityagoenka] I am wondering if you have a set of steps (or a link

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
nsivabalan commented on code in PR #9273: URL: https://github.com/apache/hudi/pull/9273#discussion_r1272554388 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala: ## @@ -115,14 +112,16 @@ class

[GitHub] [hudi] SamarthRaval commented on issue #8925: Upsert taking too long to finish

2023-07-24 Thread via GitHub
SamarthRaval commented on issue #8925: URL: https://github.com/apache/hudi/issues/8925#issuecomment-1648317412 Hello guys, I got the chance to experiment with latest hudi 0.13.1 and enabled all metadata related config to enhance the performance. "hoodie.metadata.enable"

[GitHub] [hudi] jonvex opened a new pull request, #9276: Mor perf spark33

2023-07-24 Thread via GitHub
jonvex opened a new pull request, #9276: URL: https://github.com/apache/hudi/pull/9276 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[GitHub] [hudi] hudi-bot commented on pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9273: URL: https://github.com/apache/hudi/pull/9273#issuecomment-1648265809 ## CI report: * ccdc272f4c7c61263f15174d9b468301800b25c9 UNKNOWN * 7cd21f8fb35c94bd600cbcc75638ffdafe632625 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1648265426 ## CI report: * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure:

[jira] [Created] (HUDI-6585) Certify DedupeSparkJob for both table types

2023-07-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6585: - Summary: Certify DedupeSparkJob for both table types Key: HUDI-6585 URL: https://issues.apache.org/jira/browse/HUDI-6585 Project: Apache Hudi Issue Type: Task

[GitHub] [hudi] yihua commented on a diff in pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
yihua commented on code in PR #9273: URL: https://github.com/apache/hudi/pull/9273#discussion_r1272478660 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala: ## @@ -941,6 +942,70 @@ class TestCOWDataSource extends

[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9275: URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648204721 ## CI report: * 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9275: URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648189728 ## CI report: * 5ff8958366e3b682552dc1a21f04bcf24333c84b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9274: URL: https://github.com/apache/hudi/pull/9274#issuecomment-1648189672 ## CI report: * 94d9dbcb05d1505d4a1d5e82dca8a8ba946f47da Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9274: URL: https://github.com/apache/hudi/pull/9274#issuecomment-1648176390 ## CI report: * 94d9dbcb05d1505d4a1d5e82dca8a8ba946f47da UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor

2023-07-24 Thread via GitHub
Zouxxyy commented on code in PR #9275: URL: https://github.com/apache/hudi/pull/9275#discussion_r1272430983 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java: ## @@ -223,47 +221,11 @@ protected void

[GitHub] [hudi] Zouxxyy opened a new pull request, #9275: [HUDI-6584] Abstract commit in CommitActionExecutor

2023-07-24 Thread via GitHub
Zouxxyy opened a new pull request, #9275: URL: https://github.com/apache/hudi/pull/9275 ### Change Logs Abstract commit in CommitActionExecutor ### Impact Abstract commit in CommitActionExecutor ### Risk level (write none, low medium or high below) low

[jira] [Updated] (HUDI-6584) Abstract commit in CommitActionExecutor

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6584: - Labels: pull-request-available (was: ) > Abstract commit in CommitActionExecutor >

[jira] [Created] (HUDI-6584) Abstract commit in CommitActionExecutor

2023-07-24 Thread zouxxyy (Jira)
zouxxyy created HUDI-6584: - Summary: Abstract commit in CommitActionExecutor Key: HUDI-6584 URL: https://issues.apache.org/jira/browse/HUDI-6584 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] hudi-bot commented on pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9273: URL: https://github.com/apache/hudi/pull/9273#issuecomment-1648088921 ## CI report: * ccdc272f4c7c61263f15174d9b468301800b25c9 UNKNOWN * 7cd21f8fb35c94bd600cbcc75638ffdafe632625 Azure:

[GitHub] [hudi] KnightChess opened a new pull request, #9274: [MINOR] fix millis append format error

2023-07-24 Thread via GitHub
KnightChess opened a new pull request, #9274: URL: https://github.com/apache/hudi/pull/9274 ### Change Logs MILLIS_GRANULARITY_DATE_FORMAT is `-MM-dd HH:mm:ss.SSS` ### Impact None ### Risk level (write none, low medium or high below) None ###

[GitHub] [hudi] hudi-bot commented on pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9273: URL: https://github.com/apache/hudi/pull/9273#issuecomment-1648074941 ## CI report: * ccdc272f4c7c61263f15174d9b468301800b25c9 UNKNOWN * 7cd21f8fb35c94bd600cbcc75638ffdafe632625 UNKNOWN Bot commands @hudi-bot supports the

[GitHub] [hudi] hudi-bot commented on pull request #9272: [HUDI-6582] Ensure that default recordName/namespace is not used when…

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9272: URL: https://github.com/apache/hudi/pull/9272#issuecomment-1648074844 ## CI report: * 8aeaf2013c950b50e02c3820d5942ea52c84c1cc Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9273: URL: https://github.com/apache/hudi/pull/9273#issuecomment-1647999629 ## CI report: * ccdc272f4c7c61263f15174d9b468301800b25c9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6320) Spark read inferring the data type incorrectly when partition path contains slash

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6320: - Labels: pull-request-available (was: ) > Spark read inferring the data type incorrectly when

[GitHub] [hudi] codope opened a new pull request, #9273: [HUDI-6320] Fix partition parsing in Spark file index for custom keygen

2023-07-24 Thread via GitHub
codope opened a new pull request, #9273: URL: https://github.com/apache/hudi/pull/9273 ### Change Logs Whem using custom key generator with timestamp field with type `EPOCHMILLISECONDS` and output date format as `/MM/dd`, partition parsing fails because partition column is

[GitHub] [hudi] hudi-bot commented on pull request #9270: [MINOR] Optimize error display information

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9270: URL: https://github.com/apache/hudi/pull/9270#issuecomment-1647966302 ## CI report: * c4188514ca4e4f80d4b8ad263f7cf329cd0480f7 Azure:

[GitHub] [hudi] adityaverma1997 commented on issue #9257: [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning

2023-07-24 Thread via GitHub
adityaverma1997 commented on issue #9257: URL: https://github.com/apache/hudi/issues/9257#issuecomment-1647925326 @danny0405 I haven't tried the other cleaning strategies, not even default one because I don't want to run cleaner after every commit. Also, in my case I want only last 2-3

[GitHub] [hudi] hudi-bot commented on pull request #9268: [MINOR]fix the log print class

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9268: URL: https://github.com/apache/hudi/pull/9268#issuecomment-1647857893 ## CI report: * 8d99486574bd2cfb7bd0d2b6af3d24bfac0d2aaf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647786249 ## CI report: * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647773627 ## CI report: * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9255: [HUDI-6503] Make TableServiceClient's txnManager consistent with Writ…

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9255: URL: https://github.com/apache/hudi/pull/9255#issuecomment-1647762907 ## CI report: * 257b18bc9faffdf7d063fb153e5ee1b53d57 Azure:

[GitHub] [hudi] ad1happy2go commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

2023-07-24 Thread via GitHub
ad1happy2go commented on issue #9271: URL: https://github.com/apache/hudi/issues/9271#issuecomment-1647758062 @jspaine When you were running insert, is your source containing duplicates? If yes, can you try using hoodie.datasource.write.operation as upsert only for the first case

[GitHub] [hudi] danny0405 commented on issue #9269: [SUPPORT] Hudi HMS Catalog hive_sync.conf.dir

2023-07-24 Thread via GitHub
danny0405 commented on issue #9269: URL: https://github.com/apache/hudi/issues/9269#issuecomment-1647728384 > hdfs path, since the code running on yarn and my hive-site.xml is in a local dir I guess you are right, the hive conf dir is only valid for the catalog itself, not the job,

[GitHub] [hudi] danny0405 commented on a diff in pull request #9252: [HUDI-6500] Fix bug when Using the RuntimeReplaceable function in the…

2023-07-24 Thread via GitHub
danny0405 commented on code in PR #9252: URL: https://github.com/apache/hudi/pull/9252#discussion_r1272124257 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala: ## @@ -391,63 +392,65 @@ case class

[GitHub] [hudi] danny0405 commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

2023-07-24 Thread via GitHub
danny0405 commented on issue #9271: URL: https://github.com/apache/hudi/issues/9271#issuecomment-1647714129 cc @ad1happy2go for the help ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Updated] (HUDI-6583) Fix HoodieWriteClient confs in HoodieCLIUtils

2023-07-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6583: - Labels: pull-request-available (was: ) > Fix HoodieWriteClient confs in HoodieCLIUtils >

[hudi] branch master updated (306f2bb03e6 -> 3515c8ed9fd)

2023-07-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 306f2bb03e6 [HUDI-6077] Add more partition push down filters (#8452) add 3515c8ed9fd [HUDI-6583] Fix

[jira] [Closed] (HUDI-6583) Fix HoodieWriteClient confs in HoodieCLIUtils

2023-07-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6583. Resolution: Fixed Fixed via master branch: 3515c8ed9fd9b015fdfce595910ff36e458d9116 > Fix

[jira] [Updated] (HUDI-6583) Fix HoodieWriteClient confs in HoodieCLIUtils

2023-07-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6583: - Fix Version/s: 0.14.0 > Fix HoodieWriteClient confs in HoodieCLIUtils >

[GitHub] [hudi] danny0405 merged pull request #9185: [HUDI-6583] Fix HoodieWriteClient confs in HoodieCLIUtils

2023-07-24 Thread via GitHub
danny0405 merged PR #9185: URL: https://github.com/apache/hudi/pull/9185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Created] (HUDI-6583) Fix HoodieWriteClient confs in HoodieCLIUtils

2023-07-24 Thread Danny Chen (Jira)
Danny Chen created HUDI-6583: Summary: Fix HoodieWriteClient confs in HoodieCLIUtils Key: HUDI-6583 URL: https://issues.apache.org/jira/browse/HUDI-6583 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] hudi-bot commented on pull request #9272: [HUDI-6582] Ensure that default recordName/namespace is not used when…

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9272: URL: https://github.com/apache/hudi/pull/9272#issuecomment-1647702619 ## CI report: * 8aeaf2013c950b50e02c3820d5942ea52c84c1cc Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9272: [HUDI-6582] Ensure that default recordName/namespace is not used when…

2023-07-24 Thread via GitHub
hudi-bot commented on PR #9272: URL: https://github.com/apache/hudi/pull/9272#issuecomment-1647690438 ## CI report: * 8aeaf2013c950b50e02c3820d5942ea52c84c1cc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

  1   2   >