[GitHub] [hudi] hudi-bot commented on pull request #9362: [HUDI-6644] Flink append mode use auto key generator

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9362: URL: https://github.com/apache/hudi/pull/9362#issuecomment-1665035180 ## CI report: * be8d22a885fedffd0baa991470d0e04862b3c380 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1665035137 ## CI report: * 7308946c4344ca04736b2c83b505d3a159146541 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1665035080 ## CI report: * 5c1391571fbed3ce391399b7848f33b629455941 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1665034674 ## CI report: * 884a71af797b71a7f5818472884e45f39f758328 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9362: [HUDI-6644] Flink append mode use auto key generator

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9362: URL: https://github.com/apache/hudi/pull/9362#issuecomment-1664960288 ## CI report: * be8d22a885fedffd0baa991470d0e04862b3c380 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1664960255 ## CI report: * 1f04f2b8ec4d73b4cb96229dd5381650152ee1dd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664960199 ## CI report: * 5c1391571fbed3ce391399b7848f33b629455941 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1664955794 ## CI report: * 1f04f2b8ec4d73b4cb96229dd5381650152ee1dd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664955745 ## CI report: * 5c1391571fbed3ce391399b7848f33b629455941 Azure:

[jira] [Updated] (HUDI-6644) Flink append mode use auto key generator

2023-08-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6644: - Labels: pull-request-available (was: ) > Flink append mode use auto key generator >

[GitHub] [hudi] hbgstc123 opened a new pull request, #9362: [HUDI-6644] Flink append mode use auto key generator

2023-08-03 Thread via GitHub
hbgstc123 opened a new pull request, #9362: URL: https://github.com/apache/hudi/pull/9362 ### Change Logs Support use auto key generator in flink append mode when user don't provide primary key to align with spark. Add a new class AutoRowDataKeyGen to used in

[GitHub] [hudi] kepplertreet opened a new issue, #9361: [SUPPORT] Hudi Merge On Read Tables don't write Delta Log Files

2023-08-03 Thread via GitHub
kepplertreet opened a new issue, #9361: URL: https://github.com/apache/hudi/issues/9361 Hi. I'm using a Spark Structured Streaming Application running on EMR-6.11.0 to Write into a Hudi MOR Table. Hudi Version : 0.13.0 Spark Version : 3.3.2 ``` 'hoodie.table.name':

[GitHub] [hudi] mansipp commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
mansipp commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664944883 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] hudi-bot commented on pull request #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1664931647 ## CI report: * 1f04f2b8ec4d73b4cb96229dd5381650152ee1dd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9337: [HUDI-6628] Rely on methods in HoodieBaseFile and HoodieLogFile instead of FSUtils when possible

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9337: URL: https://github.com/apache/hudi/pull/9337#issuecomment-1664931559 ## CI report: * 306b6c94e2f4793f91ae9b6ffa3f102c8bc2a18e Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9324: [HUDI-6619] Fix hudi-integ-test-bundle dependency on jackson jsk310 package

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9324: URL: https://github.com/apache/hudi/pull/9324#issuecomment-1664931493 ## CI report: * 98e49fad21b4c7b1151e96c7a72b18caf5014a7f Azure:

[GitHub] [hudi] stream2000 commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
stream2000 commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1283899831 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BucketBulkInsertDataInternalWriterHelper.java: ## @@ -65,7 +71,6 @@ public void

[GitHub] [hudi] hudi-bot commented on pull request #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1664927344 ## CI report: * 1f04f2b8ec4d73b4cb96229dd5381650152ee1dd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9324: [HUDI-6619] Fix hudi-integ-test-bundle dependency on jackson jsk310 package

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9324: URL: https://github.com/apache/hudi/pull/9324#issuecomment-1664927190 ## CI report: * 98e49fad21b4c7b1151e96c7a72b18caf5014a7f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9349: [MINOR] JSR dependency not used in spark3.3 version

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9349: URL: https://github.com/apache/hudi/pull/9349#issuecomment-1664927305 ## CI report: * 7c3142bdb0e1b1c677e61495e42c81e44916e1a0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9337: [HUDI-6628] Rely on methods in HoodieBaseFile and HoodieLogFile instead of FSUtils when possible

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9337: URL: https://github.com/apache/hudi/pull/9337#issuecomment-1664927277 ## CI report: * 306b6c94e2f4793f91ae9b6ffa3f102c8bc2a18e Azure:

[GitHub] [hudi] xushiyan commented on a diff in pull request #9278: [HUDI-6312] Rename enum values of `HollowCommitHandling`

2023-08-03 Thread via GitHub
xushiyan commented on code in PR #9278: URL: https://github.com/apache/hudi/pull/9278#discussion_r1283941770 ## hudi-common/src/main/java/org/apache/hudi/common/config/HoodieCommonConfig.java: ## @@ -78,18 +78,18 @@ public class HoodieCommonConfig extends HoodieConfig {

[GitHub] [hudi] hudi-bot commented on pull request #9261: [HUDI-6579] Adding support for upsert and deletes with spark datasource for pk less table

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9261: URL: https://github.com/apache/hudi/pull/9261#issuecomment-1664922934 ## CI report: * 7e7efc78003b0e8ef5c2d809276796b7b987a35b Azure:

[GitHub] [hudi] xushiyan commented on a diff in pull request #9278: [HUDI-6312] Rename enum values of `HollowCommitHandling`

2023-08-03 Thread via GitHub
xushiyan commented on code in PR #9278: URL: https://github.com/apache/hudi/pull/9278#discussion_r1283941125 ## hudi-common/src/main/java/org/apache/hudi/common/config/HoodieCommonConfig.java: ## @@ -78,18 +78,18 @@ public class HoodieCommonConfig extends HoodieConfig {

[jira] [Created] (HUDI-6644) Flink append mode use auto key generator

2023-08-03 Thread HBG (Jira)
HBG created HUDI-6644: - Summary: Flink append mode use auto key generator Key: HUDI-6644 URL: https://issues.apache.org/jira/browse/HUDI-6644 Project: Apache Hudi Issue Type: Task Reporter:

[GitHub] [hudi] xuzifu666 commented on pull request #9349: [MINOR] JSR dependency not used in spark3.3 version

2023-08-03 Thread via GitHub
xuzifu666 commented on PR #9349: URL: https://github.com/apache/hudi/pull/9349#issuecomment-1664917538 > @xuzifu666 why close this? sorry,want to run ci @xushiyan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] xushiyan commented on pull request #9349: [MINOR] JSR dependency not used in spark3.3 version

2023-08-03 Thread via GitHub
xushiyan commented on PR #9349: URL: https://github.com/apache/hudi/pull/9349#issuecomment-1664916791 @xuzifu666 why close this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] wecharyu commented on a diff in pull request #9343: Revert "[HUDI-6476] Improve the performance of getAllPartitionPaths (#9121)"

2023-08-03 Thread via GitHub
wecharyu commented on code in PR #9343: URL: https://github.com/apache/hudi/pull/9343#discussion_r1283930796 ## hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java: ## @@ -168,57 +167,66 @@ private List

[GitHub] [hudi] eric9204 opened a new pull request, #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-08-03 Thread via GitHub
eric9204 opened a new pull request, #9360: URL: https://github.com/apache/hudi/pull/9360 ### Change Logs Upgrade thrift's version from 0.12.0 to 0.13.0. There is an issue in thrift-0.12.0: https://issues.apache.org/jira/browse/THRIFT-4805 Will cause the following

[GitHub] [hudi] hudi-bot commented on pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9357: URL: https://github.com/apache/hudi/pull/9357#issuecomment-1664898540 ## CI report: * d8e159b823f516f584802bd3dacdaa782f185854 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9357: URL: https://github.com/apache/hudi/pull/9357#issuecomment-1664893935 ## CI report: * d8e159b823f516f584802bd3dacdaa782f185854 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9330: [HUDI-6622] Reuse the table config from HoodieTableMetaClient in the …

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9330: URL: https://github.com/apache/hudi/pull/9330#issuecomment-1664893863 ## CI report: * 38aec912160b7531914cd4c07ea8317606f34616 UNKNOWN * d6d32a693c455830a31b883915e9940fa309c77f Azure:

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9308: [HUDI-6606] Use record level index with SQL equality queries

2023-08-03 Thread via GitHub
nsivabalan commented on code in PR #9308: URL: https://github.com/apache/hudi/pull/9308#discussion_r1283913819 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] hudi-bot commented on pull request #9330: [HUDI-6622] Reuse the table config from HoodieTableMetaClient in the …

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9330: URL: https://github.com/apache/hudi/pull/9330#issuecomment-1664888349 ## CI report: * 38aec912160b7531914cd4c07ea8317606f34616 UNKNOWN * d6d32a693c455830a31b883915e9940fa309c77f Azure:

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9308: [HUDI-6606] Use record level index with SQL equality queries

2023-08-03 Thread via GitHub
nsivabalan commented on code in PR #9308: URL: https://github.com/apache/hudi/pull/9308#discussion_r1283907863 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala: ## @@ -223,10 +225,13 @@ case class HoodieFileIndex(spark:

[jira] [Assigned] (HUDI-6640) Non-blocking concurrency control

2023-08-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned HUDI-6640: Assignee: Jing Zhang > Non-blocking concurrency control > > >

[jira] [Created] (HUDI-6643) Make the compaction non-serial (plan schedule and execution)

2023-08-03 Thread Danny Chen (Jira)
Danny Chen created HUDI-6643: Summary: Make the compaction non-serial (plan schedule and execution) Key: HUDI-6643 URL: https://issues.apache.org/jira/browse/HUDI-6643 Project: Apache Hudi

[jira] [Updated] (HUDI-6480) Flink lockless multi-writer

2023-08-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6480: - Parent: HUDI-6640 Issue Type: Sub-task (was: New Feature) > Flink lockless multi-writer >

[jira] [Created] (HUDI-6642) Use completion time for file slicing

2023-08-03 Thread Danny Chen (Jira)
Danny Chen created HUDI-6642: Summary: Use completion time for file slicing Key: HUDI-6642 URL: https://issues.apache.org/jira/browse/HUDI-6642 Project: Apache Hudi Issue Type: Sub-task

[jira] [Created] (HUDI-6641) Remove the log append and always uses the current instant time in file name

2023-08-03 Thread Danny Chen (Jira)
Danny Chen created HUDI-6641: Summary: Remove the log append and always uses the current instant time in file name Key: HUDI-6641 URL: https://issues.apache.org/jira/browse/HUDI-6641 Project: Apache Hudi

[jira] [Created] (HUDI-6640) Non-blocking concurrency control

2023-08-03 Thread Danny Chen (Jira)
Danny Chen created HUDI-6640: Summary: Non-blocking concurrency control Key: HUDI-6640 URL: https://issues.apache.org/jira/browse/HUDI-6640 Project: Apache Hudi Issue Type: Epic

[GitHub] [hudi] voonhous commented on a diff in pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
voonhous commented on code in PR #9357: URL: https://github.com/apache/hudi/pull/9357#discussion_r1283900588 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteFunction.java: ## @@ -448,7 +448,8 @@ private boolean flushBucket(DataBucket bucket) {

[jira] [Reopened] (HUDI-2141) Integration flink metric in flink stream

2023-08-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reopened HUDI-2141: -- > Integration flink metric in flink stream > > >

[jira] [Closed] (HUDI-2141) Integration flink metric in flink stream

2023-08-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-2141. Resolution: Fixed Fixed via master branch: ade9d0bcb9d7ad7adabfaeb5ff2f42bc0585fdb1 > Integration flink

[hudi] branch master updated (bc583b41586 -> ade9d0bcb9d)

2023-08-03 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from bc583b41586 [HUDI-6609] Reverting multi writer checkpointing with HoodieStreamer (#9312) add ade9d0bcb9d

[GitHub] [hudi] danny0405 merged pull request #9350: [HUDI-2141] Support flink read metrics

2023-08-03 Thread via GitHub
danny0405 merged PR #9350: URL: https://github.com/apache/hudi/pull/9350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] danny0405 commented on a diff in pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
danny0405 commented on code in PR #9357: URL: https://github.com/apache/hudi/pull/9357#discussion_r1283890740 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/bucket/BucketStreamWriteFunctionWithFailOverTest.java: ## @@ -0,0 +1,202 @@ +/* + * Licensed to

[GitHub] [hudi] danny0405 commented on a diff in pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
danny0405 commented on code in PR #9357: URL: https://github.com/apache/hudi/pull/9357#discussion_r1283889835 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteFunction.java: ## @@ -448,7 +448,8 @@ private boolean flushBucket(DataBucket bucket) {

[GitHub] [hudi] danny0405 commented on a diff in pull request #9343: Revert "[HUDI-6476] Improve the performance of getAllPartitionPaths (#9121)"

2023-08-03 Thread via GitHub
danny0405 commented on code in PR #9343: URL: https://github.com/apache/hudi/pull/9343#discussion_r1283888081 ## hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java: ## @@ -168,57 +167,66 @@ private List

[GitHub] [hudi] danny0405 commented on pull request #9327: [HUDI-6617] make HoodieRecordDelegate implement KryoSerializable

2023-08-03 Thread via GitHub
danny0405 commented on PR #9327: URL: https://github.com/apache/hudi/pull/9327#issuecomment-1664854084 Test have passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=19024=results -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664852578 ## CI report: * 3dd8c31863ff6b5dc918a23ff0d13041cbc60bd3 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664847379 ## CI report: * e8163de397fcdbf1b3194bcf696435be9d28c171 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1664847144 ## CI report: * 662f3b320ab6ea06462bad9a4448add1ec2f380a UNKNOWN * ef8eaadd4f817aa08253b938e19ab3fa61d27b5c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8697: [HUDI-5514] Improving usability/performance with out of box default for append only use-cases

2023-08-03 Thread via GitHub
hudi-bot commented on PR #8697: URL: https://github.com/apache/hudi/pull/8697#issuecomment-1664846409 ## CI report: * 214bc9b6f9c0d522f4196cca12daf4756dc96439 Azure:

[GitHub] [hudi] mansipp commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
mansipp commented on code in PR #9347: URL: https://github.com/apache/hudi/pull/9347#discussion_r1283879950 ## hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java: ## @@ -143,31 +142,32 @@ public void addPartitionsToTable(String tableName, List

[GitHub] [hudi] mansipp commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
mansipp commented on code in PR #9347: URL: https://github.com/apache/hudi/pull/9347#discussion_r1283877492 ## hudi-aws/src/main/java/org/apache/hudi/aws/transaction/lock/DynamoDBBasedLockProvider.java: ## @@ -153,45 +153,53 @@ public LockItem getLock() { return lock; }

[GitHub] [hudi] yihua commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
yihua commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664836797 @mansipp could you also check the Azure CI failure? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] yihua commented on a diff in pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to V2

2023-08-03 Thread via GitHub
yihua commented on code in PR #9347: URL: https://github.com/apache/hudi/pull/9347#discussion_r1283849039 ## pom.xml: ## @@ -130,6 +130,8 @@ 1.5.6 0.16 0.8.0 +4.5.13 +4.4.13 Review Comment: Any reason we pick different versions here? ##

[GitHub] [hudi] leesf commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
leesf commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1283870818 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/ConsistentBucketBulkInsertDataInternalWriterHelper.java: ## @@ -0,0 +1,120 @@ +/* + *

[GitHub] [hudi] leesf commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
leesf commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1283870066 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BucketBulkInsertDataInternalWriterHelper.java: ## @@ -65,7 +71,6 @@ public void

[GitHub] [hudi] leesf commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
leesf commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1283867267 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/ConsistentBucketIndexBulkInsertPartitionerWithRows.java: ## @@ -0,0 +1,154 @@ +/* + *

[GitHub] [hudi] leesf commented on a diff in pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
leesf commented on code in PR #9199: URL: https://github.com/apache/hudi/pull/9199#discussion_r1283865112 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java: ## @@ -79,15 +94,19 @@

[GitHub] [hudi] amrishlal commented on a diff in pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
amrishlal commented on code in PR #9359: URL: https://github.com/apache/hudi/pull/9359#discussion_r1283857986 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala: ## @@ -194,9 +194,9 @@ trait ProvidesHoodieConfig

[GitHub] [hudi] hudi-bot commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1664818603 ## CI report: * abf8721378220e0d669aee49599a231ceff34e19 Azure:

[GitHub] [hudi] nsivabalan commented on pull request #9327: [HUDI-6617] make HoodieRecordDelegate implement KryoSerializable

2023-08-03 Thread via GitHub
nsivabalan commented on PR #9327: URL: https://github.com/apache/hudi/pull/9327#issuecomment-1664806952 @prashantwason can you review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664806676 ## CI report: * e8163de397fcdbf1b3194bcf696435be9d28c171 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9199: [HUDI-6534]Support consistent hashing row writer

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9199: URL: https://github.com/apache/hudi/pull/9199#issuecomment-1664805693 ## CI report: * abf8721378220e0d669aee49599a231ceff34e19 Azure:

[GitHub] [hudi] yihua commented on a diff in pull request #8441: Upgrade aws java sdk to v2

2023-08-03 Thread via GitHub
yihua commented on code in PR #8441: URL: https://github.com/apache/hudi/pull/8441#discussion_r1283850979 ## hudi-aws/src/main/java/org/apache/hudi/aws/credentials/HoodieAWSCredentialsProviderFactory.java: ## @@ -30,16 +30,23 @@ * Factory class for Hoodie

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664798077 ## CI report: * e8163de397fcdbf1b3194bcf696435be9d28c171 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9308: [HUDI-6606] Use record level index with SQL equality queries

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9308: URL: https://github.com/apache/hudi/pull/9308#issuecomment-1664797937 ## CI report: * ae0002b81c71f77e0c19aeb3e5872b0eb399ddb6 Azure:

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
nsivabalan commented on code in PR #9359: URL: https://github.com/apache/hudi/pull/9359#discussion_r1283833724 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala: ## @@ -194,9 +194,9 @@ trait ProvidesHoodieConfig

[GitHub] [hudi] amrishlal commented on a diff in pull request #9324: [HUDI-6619] [WIP] Fix hudi-integ-test-bundle dependency on jackson jsk310 package.

2023-08-03 Thread via GitHub
amrishlal commented on code in PR #9324: URL: https://github.com/apache/hudi/pull/9324#discussion_r1283805598 ## pom.xml: ## @@ -98,8 +98,6 @@ ${fasterxml.spark3.version} ${fasterxml.spark3.version} ${fasterxml.spark3.version} - - Review Comment:

[GitHub] [hudi] hudi-bot commented on pull request #9261: [HUDI-6579] Adding support for upsert and deletes with spark datasource for pk less table

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9261: URL: https://github.com/apache/hudi/pull/9261#issuecomment-1664733692 ## CI report: * 7d85ec69d154b5b02c81737212f415fc1aeded91 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9261: [HUDI-6579] Adding support for upsert and deletes with spark datasource for pk less table

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9261: URL: https://github.com/apache/hudi/pull/9261#issuecomment-1664728409 ## CI report: * 7d85ec69d154b5b02c81737212f415fc1aeded91 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664723220 ## CI report: * 5c1391571fbed3ce391399b7848f33b629455941 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1664686919 ## CI report: * 662f3b320ab6ea06462bad9a4448add1ec2f380a UNKNOWN * 1875a19fd05f413373eb1f2400f390706d62725e Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1664679688 ## CI report: * 662f3b320ab6ea06462bad9a4448add1ec2f380a UNKNOWN * af768285ff85e68a361482806a5de1e0b9c272a9 Azure:

[GitHub] [hudi] mansipp commented on pull request #9347: [HUDI-6638] Upgrade AWS Java SDK to v2

2023-08-03 Thread via GitHub
mansipp commented on PR #9347: URL: https://github.com/apache/hudi/pull/9347#issuecomment-1664649658 Manually tested s3a path using EMR cluster. ```scala spark-shell \ --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \ --conf

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664628275 ## CI report: * e8163de397fcdbf1b3194bcf696435be9d28c171 Azure:

[GitHub] [hudi] jonvex commented on a diff in pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
jonvex commented on code in PR #9276: URL: https://github.com/apache/hudi/pull/9276#discussion_r1283702970 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBootstrapMORRelation.scala: ## @@ -58,10 +59,13 @@ case class

[GitHub] [hudi] xushiyan commented on a diff in pull request #9324: [HUDI-6619] [WIP] Fix hudi-integ-test-bundle dependency on jackson jsk310 package.

2023-08-03 Thread via GitHub
xushiyan commented on code in PR #9324: URL: https://github.com/apache/hudi/pull/9324#discussion_r1283702886 ## pom.xml: ## @@ -98,8 +98,6 @@ ${fasterxml.spark3.version} ${fasterxml.spark3.version} ${fasterxml.spark3.version} - - Review Comment: as

[GitHub] [hudi] hudi-bot commented on pull request #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9359: URL: https://github.com/apache/hudi/pull/9359#issuecomment-1664617802 ## CI report: * e8163de397fcdbf1b3194bcf696435be9d28c171 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9357: [HUDI-6588] Fix duplicate fileId on TM failover and recovery

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9357: URL: https://github.com/apache/hudi/pull/9357#issuecomment-1664608150 ## CI report: * d8e159b823f516f584802bd3dacdaa782f185854 Azure:

[GitHub] [hudi] jonvex commented on a diff in pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
jonvex commented on code in PR #9276: URL: https://github.com/apache/hudi/pull/9276#discussion_r1283684686 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala: ## @@ -247,11 +247,23 @@ object DefaultSource { Option(schema) }

[jira] [Updated] (HUDI-6639) Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6639: - Labels: pull-request-available (was: ) > Rename hoodie.sql.write.operation to

[GitHub] [hudi] amrishlal opened a new pull request, #9359: [HUDI-6639] Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread via GitHub
amrishlal opened a new pull request, #9359: URL: https://github.com/apache/hudi/pull/9359 Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation ### Change Logs Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation ### Impact

[jira] [Created] (HUDI-6639) Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation

2023-08-03 Thread Amrish Lal (Jira)
Amrish Lal created HUDI-6639: Summary: Rename hoodie.sql.write.operation to hoodie.spark.sql.insert.into.operation Key: HUDI-6639 URL: https://issues.apache.org/jira/browse/HUDI-6639 Project: Apache Hudi

[jira] [Comment Edited] (HUDI-6596) Propose rollback implementation changes to guard against concurrent jobs

2023-08-03 Thread Krishen Bhan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750879#comment-17750879 ] Krishen Bhan edited comment on HUDI-6596 at 8/3/23 8:15 PM: Thanks for the

[jira] [Commented] (HUDI-6596) Propose rollback implementation changes to guard against concurrent jobs

2023-08-03 Thread Krishen Bhan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750879#comment-17750879 ] Krishen Bhan commented on HUDI-6596: Thanks for the reply! > The table lock could become a bottleneck,

[GitHub] [hudi] yihua commented on a diff in pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
yihua commented on code in PR #9276: URL: https://github.com/apache/hudi/pull/9276#discussion_r1283625541 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala: ## @@ -247,11 +247,23 @@ object DefaultSource { Option(schema) }

[GitHub] [hudi] hudi-bot commented on pull request #8697: [HUDI-5514] Improving usability/performance with out of box default for append only use-cases

2023-08-03 Thread via GitHub
hudi-bot commented on PR #8697: URL: https://github.com/apache/hudi/pull/8697#issuecomment-1664552015 ## CI report: * 7a0e3786173a124177688946c37011577fe97478 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8697: [HUDI-5514] Improving usability/performance with out of box default for append only use-cases

2023-08-03 Thread via GitHub
hudi-bot commented on PR #8697: URL: https://github.com/apache/hudi/pull/8697#issuecomment-1664533286 ## CI report: * 7a0e3786173a124177688946c37011577fe97478 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9337: [HUDI-6628] Rely on methods in HoodieBaseFile and HoodieLogFile instead of FSUtils when possible

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9337: URL: https://github.com/apache/hudi/pull/9337#issuecomment-1664517346 ## CI report: * 306b6c94e2f4793f91ae9b6ffa3f102c8bc2a18e Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9276: [HUDI-6635] Hudi Spark Integration Redesign MOR and Bootstrap reading

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9276: URL: https://github.com/apache/hudi/pull/9276#issuecomment-1664516647 ## CI report: * 662f3b320ab6ea06462bad9a4448add1ec2f380a UNKNOWN * af768285ff85e68a361482806a5de1e0b9c272a9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9308: [HUDI-6606] Use record level index with SQL equality queries

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9308: URL: https://github.com/apache/hudi/pull/9308#issuecomment-1664516962 ## CI report: * 5195761aea8a8d9cfc046978601d10a246507da8 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9261: [HUDI-6579] Adding support for upsert and deletes with spark datasource for pk less table

2023-08-03 Thread via GitHub
hudi-bot commented on PR #9261: URL: https://github.com/apache/hudi/pull/9261#issuecomment-1664516511 ## CI report: * 7d85ec69d154b5b02c81737212f415fc1aeded91 Azure:

[hudi] branch asf-site updated: [DOCS] Update bootstrap page (#9338)

2023-08-03 Thread bhavanisudha
This is an automated email from the ASF dual-hosted git repository. bhavanisudha pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 8cd33fa3ad8 [DOCS] Update bootstrap

[GitHub] [hudi] bhasudha merged pull request #9338: [DOCS] Update bootstrap page

2023-08-03 Thread via GitHub
bhasudha merged PR #9338: URL: https://github.com/apache/hudi/pull/9338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] amrishlal commented on a diff in pull request #9324: [HUDI-6619] [WIP] Fix hudi-integ-test-bundle dependency on jackson jsk310 package.

2023-08-03 Thread via GitHub
amrishlal commented on code in PR #9324: URL: https://github.com/apache/hudi/pull/9324#discussion_r1283598675 ## pom.xml: ## @@ -98,8 +98,6 @@ ${fasterxml.spark3.version} ${fasterxml.spark3.version} ${fasterxml.spark3.version} - - Review Comment:

[GitHub] [hudi] bhasudha commented on pull request #9338: [DOCS] Update bootstrap page

2023-08-03 Thread via GitHub
bhasudha commented on PR #9338: URL: https://github.com/apache/hudi/pull/9338#issuecomment-1664460516 ![Screenshot 2023-08-03 at 11 40 19 AM](https://github.com/apache/hudi/assets/2179254/2ef881a4-6537-4465-9a03-9e4fe7cf99f8) ![Screenshot 2023-08-03 at 11 40 28

  1   2   3   >