[GitHub] [hudi] ksoullpwk commented on issue #9440: [SUPPORT] Trino cannot read when there is replacecommit metadata

2023-08-14 Thread via GitHub
ksoullpwk commented on issue #9440: URL: https://github.com/apache/hudi/issues/9440#issuecomment-1678460113 Yes, it works. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] hudi-bot commented on pull request #9434: Dummy commit to trigger CI

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9434: URL: https://github.com/apache/hudi/pull/9434#issuecomment-1678442027 ## CI report: * e895bfb27350f497100c3cd50246badcba99f27d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9434: Dummy commit to trigger CI

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9434: URL: https://github.com/apache/hudi/pull/9434#issuecomment-1678437811 ## CI report: * e895bfb27350f497100c3cd50246badcba99f27d Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments

2023-08-14 Thread via GitHub
danny0405 commented on code in PR #8683: URL: https://github.com/apache/hudi/pull/8683#discussion_r1294178078 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/TableOptionProperties.java: ## @@ -184,7 +184,9 @@ public static Map

[GitHub] [hudi] Riddle4045 commented on issue #9435: [SUPPORT] Trino can't read tables created by Flink Hudi conector

2023-08-14 Thread via GitHub
Riddle4045 commented on issue #9435: URL: https://github.com/apache/hudi/issues/9435#issuecomment-1678406201 > > HMS props for the Hudi table creating using Flink SQL > > You are using the Flink Hive catalog, the table are actually created by the hive catalog. Actually we have a

[hudi] branch asf-site updated: [DOCS] Updated image paths for blogs (#9446)

2023-08-14 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new b1b1b524bbd [DOCS] Updated image paths for

[GitHub] [hudi] yihua merged pull request #9446: [DOCS] Updated image paths for blogs

2023-08-14 Thread via GitHub
yihua merged PR #9446: URL: https://github.com/apache/hudi/pull/9446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-6654) Add new log block header type to store record positions

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6654: Epic Link: HUDI-6242 > Add new log block header type to store record positions >

[jira] [Updated] (HUDI-6654) Add new log block header type to store record positions

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6654: Fix Version/s: 1.0.0 > Add new log block header type to store record positions >

[jira] [Closed] (HUDI-6654) Add new log block header type to store record positions

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-6654. --- Resolution: Fixed > Add new log block header type to store record positions >

[GitHub] [hudi] danny0405 commented on issue #9435: [SUPPORT] Trino can't read tables created by Flink Hudi conector

2023-08-14 Thread via GitHub
danny0405 commented on issue #9435: URL: https://github.com/apache/hudi/issues/9435#issuecomment-1678355424 > HMS props for the Hudi table creating using Flink SQL You are using the Flink Hive catalog, the table are actually created by the hive catalog. Actually we have a separate

[GitHub] [hudi] danny0405 commented on issue #8848: [SUPPORT] Hive Sync tool fails to sync Hoodi table written using Flink 1.16 to HMS

2023-08-14 Thread via GitHub
danny0405 commented on issue #8848: URL: https://github.com/apache/hudi/issues/8848#issuecomment-1678352784 In principle, we do not package any hadoop related jars into the bundle jar, the classpath of the runtime env should include it. -- This is an automated message from the Apache Git

[hudi] branch asf-site updated: [HUDI-6685] Fix code typo in pyspark 'Insert Overwrite' section of Quick Start Guide. (#9432)

2023-08-14 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new b0e57453d3a [HUDI-6685] Fix code typo in

[GitHub] [hudi] nsivabalan merged pull request #9432: [HUDI-6685] Fix code typo in pyspark 'Insert Overwrite' section of Quick Start Guide.

2023-08-14 Thread via GitHub
nsivabalan merged PR #9432: URL: https://github.com/apache/hudi/pull/9432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9437: URL: https://github.com/apache/hudi/pull/9437#issuecomment-1678342431 ## CI report: * b25b5402c1e3e14264c6bbfd38910f4b93b8a871 Azure:

[GitHub] [hudi] yihua commented on a diff in pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
yihua commented on code in PR #9437: URL: https://github.com/apache/hudi/pull/9437#discussion_r1294125375 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java: ## @@ -741,6 +791,116 @@ private void validateBloomFilters(

[GitHub] [hudi] boneanxs commented on a diff in pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-14 Thread via GitHub
boneanxs commented on code in PR #9408: URL: https://github.com/apache/hudi/pull/9408#discussion_r1294125769 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableAddPartitionCommand.scala: ## @@ -0,0 +1,88 @@ +/* + *

[GitHub] [hudi] yihua commented on a diff in pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
yihua commented on code in PR #9437: URL: https://github.com/apache/hudi/pull/9437#discussion_r1294125576 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java: ## @@ -741,6 +791,116 @@ private void validateBloomFilters(

[GitHub] [hudi] codope commented on issue #9440: [SUPPORT] Trino cannot read when there is replacecommit metadata

2023-08-14 Thread via GitHub
codope commented on issue #9440: URL: https://github.com/apache/hudi/issues/9440#issuecomment-1678338350 @ksoullpwk Thanks for the diagnosis. Could you check if this fix helps you? https://github.com/trinodb/trino/pull/18213 -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] hudi-bot commented on pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9437: URL: https://github.com/apache/hudi/pull/9437#issuecomment-1678336975 ## CI report: * b25b5402c1e3e14264c6bbfd38910f4b93b8a871 Azure:

[GitHub] [hudi] danny0405 commented on pull request #9444: [HUDI-6692] Do not allow switching from Primary keyed table to primary key less table

2023-08-14 Thread via GitHub
danny0405 commented on PR #9444: URL: https://github.com/apache/hudi/pull/9444#issuecomment-1678331086 > If a write to a table with a pk was missing the recordkey field in options it would think it was a pkless write. now it fails I'm confused, if we already know it is a table with

[hudi] branch asf-site updated: [HUDI-6676][DOCS] Add command for CreateHoodieTableLike (#9441)

2023-08-14 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new f138422fcb9 [HUDI-6676][DOCS] Add command

[GitHub] [hudi] danny0405 merged pull request #9441: [HUDI-6676][DOCS] Add command for CreateHoodieTableLike

2023-08-14 Thread via GitHub
danny0405 merged PR #9441: URL: https://github.com/apache/hudi/pull/9441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Closed] (HUDI-6683) Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-14 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6683. Resolution: Fixed Fixed via master branch: 4099e1d18b78583d739fdb252f85b58d991d2fb0 > Added kafka key as

[hudi] branch master updated: [HUDI-6683] Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource (#9403)

2023-08-14 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 4099e1d18b7 [HUDI-6683] Added kafka key as

[GitHub] [hudi] danny0405 merged pull request #9403: [HUDI-6683] Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-14 Thread via GitHub
danny0405 merged PR #9403: URL: https://github.com/apache/hudi/pull/9403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] danny0405 commented on pull request #9403: [HUDI-6683] Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-14 Thread via GitHub
danny0405 commented on PR #9403: URL: https://github.com/apache/hudi/pull/9403#issuecomment-1678327812 Thanks for the nice feedback @hussein-awala , maybe you can fire a separate PR to address it. -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Updated] (HUDI-6683) Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6683: - Fix Version/s: 1.1.0 (was: 1.0.0) > Added kafka key as part of hudi

[jira] [Updated] (HUDI-6585) Certify DedupeSparkJob for both table types

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6585: - Fix Version/s: 1.1.0 0.15.0 (was: 1.0.0) > Certify

[jira] [Updated] (HUDI-6586) Add Incremental scan support to dbt

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6586: - Fix Version/s: 0.15.0 0.14.1 > Add Incremental scan support to dbt >

[jira] [Commented] (HUDI-6490) Implement support for applying updates as deletes + inserts

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754362#comment-17754362 ] Vinoth Chandar commented on HUDI-6490: -- [~tim.brown] Do you want to take this work up? This can be

[jira] [Updated] (HUDI-6490) Implement support for applying updates as deletes + inserts

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6490: - Fix Version/s: 0.15.0 0.14.1 > Implement support for applying updates as

[jira] [Assigned] (HUDI-6490) Implement support for applying updates as deletes + inserts

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-6490: Assignee: Timothy Brown > Implement support for applying updates as deletes + inserts >

[jira] [Updated] (HUDI-6296) Add Scala 2.13 build profile to support scala 2.13

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6296: - Fix Version/s: 1.1.0 (was: 1.0.0) > Add Scala 2.13 build profile to

[jira] [Commented] (HUDI-6640) Non-blocking concurrency control

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754361#comment-17754361 ] Vinoth Chandar commented on HUDI-6640: -- This is a duplicate of HUDI-5672 > Non-blocking concurrency

[jira] [Updated] (HUDI-1045) Support updates during clustering

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1045: - Epic Link: HUDI-5672 (was: HUDI-1042) > Support updates during clustering >

[jira] [Updated] (HUDI-5672) Non-blocking multi writer support

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-5672: - Summary: Non-blocking multi writer support (was: Lockless multi writer support) > Non-blocking

[jira] [Updated] (HUDI-1238) [UMBRELLA] Perf test env

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1238: - Fix Version/s: 1.1.0 (was: 1.0.0) > [UMBRELLA] Perf test env >

[jira] [Updated] (HUDI-2519) [UMBRELLA] Seamless meta sync

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2519: - Fix Version/s: 1.0.0 (was: 1.1.0) > [UMBRELLA] Seamless meta sync >

[jira] [Updated] (HUDI-2519) [UMBRELLA] Seamless meta sync

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2519: - Fix Version/s: 1.1.0 (was: 1.0.0) > [UMBRELLA] Seamless meta sync >

[GitHub] [hudi] hudi-bot commented on pull request #9444: [HUDI-6692] Do not allow switching from Primary keyed table to primary key less table

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9444: URL: https://github.com/apache/hudi/pull/9444#issuecomment-167822 ## CI report: * c7e99fd19a00469c0e181b6c64b63aa9cfb7ed4e Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #9425: When invalidate the table in the spark sql query cache, verify if theā€¦

2023-08-14 Thread via GitHub
danny0405 commented on code in PR #9425: URL: https://github.com/apache/hudi/pull/9425#discussion_r1294098528 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala: ## @@ -965,8 +965,9 @@ object HoodieSparkSqlWriter { // we

[jira] [Updated] (HUDI-2638) Rewrite tests around Hudi index

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2638: - Fix Version/s: (was: 1.0.0) > Rewrite tests around Hudi index >

[jira] [Updated] (HUDI-3121) Spark datasource with bucket index unit test reuse

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-3121: - Fix Version/s: 1.1.0 (was: 1.0.0) > Spark datasource with bucket index

[jira] [Updated] (HUDI-2638) Rewrite tests around Hudi index

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2638: - Fix Version/s: 1.1.0 > Rewrite tests around Hudi index > --- > >

[jira] [Updated] (HUDI-1916) Create a matrix of datatypes across spark, hive, presto, Avro, parquet.

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1916: - Fix Version/s: 1.1.0 (was: 1.0.0) > Create a matrix of datatypes across

[jira] [Updated] (HUDI-2375) Create common SchemaProvider and RecordPayloads for spark, flink etc.

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2375: - Fix Version/s: 1.1.0 (was: 1.0.0) > Create common SchemaProvider and

[jira] [Assigned] (HUDI-309) General Redesign of Archived Timeline for efficient scan and management

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-309: --- Assignee: Danny Chen (was: Balaji Varadarajan) > General Redesign of Archived Timeline for

[GitHub] [hudi] danny0405 commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-14 Thread via GitHub
danny0405 commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1678284808 @prashantwason You can cherry pick https://github.com/apache/hudi/pull/9401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Updated] (HUDI-6676) Add command for CreateHoodieTableLike

2023-08-14 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6676: - Fix Version/s: 1.0.0 > Add command for CreateHoodieTableLike > - > >

[jira] [Closed] (HUDI-6676) Add command for CreateHoodieTableLike

2023-08-14 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6676. Resolution: Fixed Fixed via master branch: 8220d23be19af4783a9a776dfffa48167975a6a2 > Add command for

[jira] [Updated] (HUDI-3529) Improve dependency management and bundling

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-3529: - Fix Version/s: 1.1.0 (was: 1.0.0) > Improve dependency management and

[GitHub] [hudi] danny0405 merged pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike

2023-08-14 Thread via GitHub
danny0405 merged PR #9412: URL: https://github.com/apache/hudi/pull/9412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [HUDI-6676] Add command for CreateHoodieTableLike (#9412)

2023-08-14 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8220d23be19 [HUDI-6676] Add command for

[jira] [Updated] (HUDI-2871) Decouple metrics dependencies from hudi-client-common

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2871: - Fix Version/s: 1.1.0 (was: 1.0.0) > Decouple metrics dependencies from

[jira] [Updated] (HUDI-6483) MERGE INTO should support schema evolution for partial updates.

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6483: - Fix Version/s: 1.1.0 > MERGE INTO should support schema evolution for partial updates. >

[jira] [Updated] (HUDI-2687) [UMBRELLA] A new Trino connector for Hudi

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2687: - Fix Version/s: 1.0.0 (was: 1.1.0) > [UMBRELLA] A new Trino connector for

[jira] [Updated] (HUDI-1574) Trim existing unit tests to finish in much shorter amount of time

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1574: - Fix Version/s: 1.1.0 (was: 1.0.0) > Trim existing unit tests to finish in

[jira] [Commented] (HUDI-1457) Add multi writing to Hudi tables using DFS based locking (only HDFS atomic renames)

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754349#comment-17754349 ] Vinoth Chandar commented on HUDI-1457: -- this does not work on cloud storage, since we cannot rely

[jira] [Updated] (HUDI-3057) Instants should be generated strictly under locks

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-3057: - Fix Version/s: 1.1.0 > Instants should be generated strictly under locks >

[jira] [Updated] (HUDI-1457) Add multi writing to Hudi tables using DFS based locking (only HDFS atomic renames)

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1457: - Fix Version/s: (was: 0.15.0) > Add multi writing to Hudi tables using DFS based locking (only

[jira] [Updated] (HUDI-4068) Add Cosmos based lock provider for Azure

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-4068: - Fix Version/s: 1.1.0 (was: 1.0.0) > Add Cosmos based lock provider for

[jira] [Updated] (HUDI-4067) Add Spanner based lock provider for GCP

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-4067: - Fix Version/s: 1.1.0 (was: 1.0.0) > Add Spanner based lock provider for

[jira] [Updated] (HUDI-2173) Enhancing DynamoDB based LockProvider

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2173: - Fix Version/s: 1.1.0 > Enhancing DynamoDB based LockProvider >

[jira] [Updated] (HUDI-2687) [UMBRELLA] A new Trino connector for Hudi

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2687: - Fix Version/s: 1.1.0 (was: 1.0.0) > [UMBRELLA] A new Trino connector for

[GitHub] [hudi] hussein-awala commented on a diff in pull request #9441: [HUDI-6676][DOCS] Add command for CreateHoodieTableLike

2023-08-14 Thread via GitHub
hussein-awala commented on code in PR #9441: URL: https://github.com/apache/hudi/pull/9441#discussion_r1294085403 ## website/docs/quick-start-guide.md: ## @@ -384,6 +384,68 @@ create table hudi_ctas_cow_pt_tbl2 using hudi location 'file:/tmp/hudi/hudi_tbl/ partitioned by

[jira] [Updated] (HUDI-4141) [RFC-64] Table Format APIs

2023-08-14 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-4141: - Start Date: 4/Sep/23 Due Date: 4/Oct/23 > [RFC-64] Table Format APIs >

[GitHub] [hudi] hussein-awala commented on a diff in pull request #9444: [HUDI-6692] Do not allow switching from Primary keyed table to primary key less table

2023-08-14 Thread via GitHub
hussein-awala commented on code in PR #9444: URL: https://github.com/apache/hudi/pull/9444#discussion_r1294078912 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala: ## @@ -179,9 +179,11 @@ object HoodieWriterUtils { if

[GitHub] [hudi] hudi-bot commented on pull request #9445: [HUDI-6694] Fix log file CLI around command blocks

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9445: URL: https://github.com/apache/hudi/pull/9445#issuecomment-1678258723 ## CI report: * 06d72d5563b9cd26e131c3907dcc653e59a2b8be Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9445: [HUDI-6694] Fix log file CLI around command blocks

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9445: URL: https://github.com/apache/hudi/pull/9445#issuecomment-1678253584 ## CI report: * 06d72d5563b9cd26e131c3907dcc653e59a2b8be UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9437: URL: https://github.com/apache/hudi/pull/9437#issuecomment-1678253520 ## CI report: * b25b5402c1e3e14264c6bbfd38910f4b93b8a871 Azure:

[GitHub] [hudi] hussein-awala commented on a diff in pull request #9403: [HUDI-6683] Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-14 Thread via GitHub
hussein-awala commented on code in PR #9403: URL: https://github.com/apache/hudi/pull/9403#discussion_r1294069377 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java: ## @@ -175,9 +176,11 @@ public GenericRecord

[GitHub] [hudi] yihua opened a new pull request, #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink

2023-08-14 Thread via GitHub
yihua opened a new pull request, #5269: URL: https://github.com/apache/hudi/pull/5269 ## What is the purpose of the pull request - In Deltastreamer, we re-instantiate WriteClient whenever schema changes. Same write client is used by all async table services as well. This poses an

[GitHub] [hudi] nfarah86 opened a new pull request, #9446: updated image path for /blog

2023-08-14 Thread via GitHub
nfarah86 opened a new pull request, #9446: URL: https://github.com/apache/hudi/pull/9446 ### Change Logs fixed broken images https://github.com/apache/hudi/assets/5392555/055efb07-c4bc-4727-a4e4-bdb81fdbf546;> @nsivabalan please review -- This is an automated message from

[jira] [Updated] (HUDI-6694) Fix log file CLI around command blocks

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6694: - Labels: pull-request-available (was: ) > Fix log file CLI around command blocks >

[GitHub] [hudi] yihua opened a new pull request, #9445: [HUDI-6694] Fix log file CLI around command blocks

2023-08-14 Thread via GitHub
yihua opened a new pull request, #9445: URL: https://github.com/apache/hudi/pull/9445 ### Change Logs This PR fixes the log file CLI commands when the log file contains command blocks like rollback commands. The tests are adjusted to consider such a scenario. Without the fix,

[jira] [Updated] (HUDI-6694) Fix log file CLI around command blocks

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6694: Description: When there are rollback command blocks in the log files, the log file command throws NPE:

[jira] [Updated] (HUDI-6694) Fix log file CLI around command blocks

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6694: Description: When there are rollback command blocks in the log files, the > Fix log file CLI around

[jira] [Created] (HUDI-6694) Fix log file CLI around command blocks

2023-08-14 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6694: --- Summary: Fix log file CLI around command blocks Key: HUDI-6694 URL: https://issues.apache.org/jira/browse/HUDI-6694 Project: Apache Hudi Issue Type: Bug

[jira] [Assigned] (HUDI-6694) Fix log file CLI around command blocks

2023-08-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-6694: --- Assignee: Ethan Guo > Fix log file CLI around command blocks >

[GitHub] [hudi] yihua commented on a diff in pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
yihua commented on code in PR #9437: URL: https://github.com/apache/hudi/pull/9437#discussion_r1294049901 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java: ## @@ -741,6 +791,116 @@ private void validateBloomFilters(

[GitHub] [hudi] hudi-bot commented on pull request #9444: [HUDI-6692] Do not allow switching from Primary keyed table to primary key less table

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9444: URL: https://github.com/apache/hudi/pull/9444#issuecomment-1678162678 ## CI report: * c7e99fd19a00469c0e181b6c64b63aa9cfb7ed4e Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9444: [HUDI-6692] Do not allow switching from Primary keyed table to primary key less table

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9444: URL: https://github.com/apache/hudi/pull/9444#issuecomment-1678153770 ## CI report: * c7e99fd19a00469c0e181b6c64b63aa9cfb7ed4e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
nsivabalan commented on code in PR #9437: URL: https://github.com/apache/hudi/pull/9437#discussion_r1294013795 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java: ## @@ -741,6 +791,116 @@ private void validateBloomFilters(

[GitHub] [hudi] prashantwason commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-14 Thread via GitHub
prashantwason commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1678132749 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] prashantwason commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-14 Thread via GitHub
prashantwason commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1678132491 @stream2000 The build is failing due to a test failure due to this commit. Can you please check?

[jira] [Created] (HUDI-6693) Streaming writes fail in quick start w/ 0.14.0

2023-08-14 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-6693: - Summary: Streaming writes fail in quick start w/ 0.14.0 Key: HUDI-6693 URL: https://issues.apache.org/jira/browse/HUDI-6693 Project: Apache Hudi

[jira] [Updated] (HUDI-6692) If table with recordkey doesn't have recordkey in spark ds write, it will bulk insert by default

2023-08-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6692: - Labels: pull-request-available (was: ) > If table with recordkey doesn't have recordkey in spark

[GitHub] [hudi] jonvex opened a new pull request, #9444: [HUDI-6692] If pk table has no recordkey in write, it should fail

2023-08-14 Thread via GitHub
jonvex opened a new pull request, #9444: URL: https://github.com/apache/hudi/pull/9444 ### Change Logs if the write was missing the recordkey it would think it was a pkless write. now it fails ### Impact prevent unexpected behavior ### Risk level (write none, low

[jira] [Created] (HUDI-6692) If table with recordkey doesn't have recordkey in spark ds write, it will bulk insert by default

2023-08-14 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-6692: - Summary: If table with recordkey doesn't have recordkey in spark ds write, it will bulk insert by default Key: HUDI-6692 URL: https://issues.apache.org/jira/browse/HUDI-6692

[GitHub] [hudi] Riddle4045 commented on issue #9435: [SUPPORT] Trino can't read tables created by Flink Hudi conector

2023-08-14 Thread via GitHub
Riddle4045 commented on issue #9435: URL: https://github.com/apache/hudi/issues/9435#issuecomment-1678089145 @danny0405 I checked the Table props in metastore of a table that's synced using Hudi HMS sync tool vs the Flink table I mentioned below. I see very different properties here

[GitHub] [hudi] hudi-bot commented on pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9437: URL: https://github.com/apache/hudi/pull/9437#issuecomment-1678003863 ## CI report: * 699793358327fe0caf4df52a0ee199a9c54ab58d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9434: Dummy commit to trigger CI

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9434: URL: https://github.com/apache/hudi/pull/9434#issuecomment-1677992887 ## CI report: * e895bfb27350f497100c3cd50246badcba99f27d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9437: [HUDI-6689] Add record index validation in MDT validator

2023-08-14 Thread via GitHub
hudi-bot commented on PR #9437: URL: https://github.com/apache/hudi/pull/9437#issuecomment-1677992987 ## CI report: * f0f16cb886023e9f9eba3073583a786803944e9b Azure:

[GitHub] [hudi] voonhous commented on issue #9443: Getting `Caused by: java.lang.UnsupportedOperationException` when running a previously successful Spark Job on writing to Hudi

2023-08-14 Thread via GitHub
voonhous commented on issue #9443: URL: https://github.com/apache/hudi/issues/9443#issuecomment-1677986967 Erm, stack trace looks similar to that described in: https://issues.apache.org/jira/browse/HUDI-5992 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments

2023-08-14 Thread via GitHub
parisni commented on code in PR #8683: URL: https://github.com/apache/hudi/pull/8683#discussion_r1293892201 ## hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java: ## @@ -133,7 +154,7 @@ private static String

[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments

2023-08-14 Thread via GitHub
parisni commented on code in PR #8683: URL: https://github.com/apache/hudi/pull/8683#discussion_r1293890701 ## hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java: ## @@ -19,40 +19,61 @@ package

[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments

2023-08-14 Thread via GitHub
parisni commented on code in PR #8683: URL: https://github.com/apache/hudi/pull/8683#discussion_r1293890017 ## hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java: ## @@ -19,40 +19,61 @@ package

[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments

2023-08-14 Thread via GitHub
parisni commented on code in PR #8683: URL: https://github.com/apache/hudi/pull/8683#discussion_r1293889135 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/JDBCExecutor.java: ## @@ -65,7 +65,7 @@ public void runSQL(String s) { try { stmt =

[GitHub] [hudi] parisni commented on pull request #9071: [HUDI-6453] Cascade Glue schema changes to partitions

2023-08-14 Thread via GitHub
parisni commented on PR #9071: URL: https://github.com/apache/hudi/pull/9071#issuecomment-1677939952 > If we want to support unordered addition of fields then let's restrict cascading to that case? Unlike hive metastore, glue supports unordered addition of columns. It does not

  1   2   >