[hudi] branch master updated (051eb0e930e -> 98d956fd845)

2023-10-24 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 051eb0e930e [MINOR] Add tests on combine parallelism (#9731) add 98d956fd845 [HUDI-6977] Upgrade hadoop version

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
yihua merged PR #9914: URL: https://github.com/apache/hudi/pull/9914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6973] Instantiate HoodieFileGroupRecordBuffer inside new file group reader [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9910: URL: https://github.com/apache/hudi/pull/9910#issuecomment-1778561669 ## CI report: * f158692bc1611582566b3bbd76e49d07a290e802 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [I] [SUPPORT] Compaction error [hudi]

2023-10-24 Thread via GitHub
codope closed issue #9885: [SUPPORT] Compaction error URL: https://github.com/apache/hudi/issues/9885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [SUPPORT] AWS Glue Sync fails on a Hudi table with > 25 partitions [hudi]

2023-10-24 Thread via GitHub
ad1happy2go commented on issue #9806: URL: https://github.com/apache/hudi/issues/9806#issuecomment-1778552572 @buiducsinh34 @noahtaite Closing this out as PR is merged. Thanks Everybody. Feel free to reopen if you still see the issue. -- This is an automated message from the Apache Git

Re: [I] [SUPPORT] Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(Lscala/PartialFunction;)Lorg/apache/spark/sql/catalyst/p

2023-10-24 Thread via GitHub
pushpavanthar commented on issue #8614: URL: https://github.com/apache/hudi/issues/8614#issuecomment-1778547455 we tried running this on emr-6.7.0 and few other higher labels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1371173652 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala: ## @@ -261,7 +262,8 @@ class TestMergeIntoTable extends

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1371170669 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala: ## @@ -261,7 +262,8 @@ class TestMergeIntoTable extends

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1371157540 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala: ## @@ -261,7 +262,8 @@ class TestMergeIntoTable extends

Re: [I] [SUPPORT] Control file sizing during FULL_RECORD bootstrap mode [hudi]

2023-10-24 Thread via GitHub
ad1happy2go commented on issue #9915: URL: https://github.com/apache/hudi/issues/9915#issuecomment-1778505679 @fenil25 bulk-insert operation doesn't handle the small file handling, that is why you see the file sizes equal to split size. Sp the total number of partitions is calculated as

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778503193 ## CI report: * 3672dea3c9d2512071dc27b99e24dfb3922a3b38 UNKNOWN * bfdb36f31ef0b8670c82c308494f9af2f7ef1272 Azure:

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1371143452 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -65,8 +65,11 @@ case class BaseFileOnlyRelation(override val

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1371142864 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala: ## @@ -220,7 +220,9 @@ abstract class HoodieBaseRelation(val

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1371141863 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

Re: [PR] [HUDI-6975] Optimize the code of DayBasedCompactionStrategy [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9911: URL: https://github.com/apache/hudi/pull/9911#discussion_r1371139632 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/DayBasedCompactionStrategy.java: ## @@ -63,21 +60,9 @@ public Comparator

Re: [PR] [MINOR] Add table name and range msg for streaming reads logs [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9912: URL: https://github.com/apache/hudi/pull/9912#discussion_r1371137154 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java: ## @@ -226,9 +226,9 @@ public void

Re: [PR] [MINOR] Add table name and range msg for streaming reads logs [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9912: URL: https://github.com/apache/hudi/pull/9912#discussion_r1371135529 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/InstantRange.java: ## @@ -57,6 +59,15 @@ public String getEndInstant() { public abstract boolean

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9892: URL: https://github.com/apache/hudi/pull/9892#discussion_r1371133664 ## hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java: ## @@ -86,30 +86,26 @@ public Option getInsertValue(Schema schema,

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9892: URL: https://github.com/apache/hudi/pull/9892#discussion_r1371132703 ## hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java: ## @@ -45,12 +45,12 @@ public class DefaultHoodieRecordPayload extends

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9892: URL: https://github.com/apache/hudi/pull/9892#discussion_r1371109726 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/PayloadCreation.java: ## @@ -43,14 +44,17 @@ public class PayloadCreation implements

Re: [PR] [MINOR] Add table name and range msg for streaming reads logs [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9912: URL: https://github.com/apache/hudi/pull/9912#issuecomment-1778466955 ## CI report: * 7f6535290896455bb3312e7203f2eafa69109f05 Azure:

Re: [PR] [MINOR] Add table name and range msg for streaming reads logs [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9912: URL: https://github.com/apache/hudi/pull/9912#issuecomment-1778462077 ## CI report: * 7f6535290896455bb3312e7203f2eafa69109f05 Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778461940 ## CI report: * 3672dea3c9d2512071dc27b99e24dfb3922a3b38 UNKNOWN * d96a7423b1c1bae13148744547726ed95ee5c6b7 Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1371105049 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala: ## @@ -261,7 +262,8 @@ class TestMergeIntoTable extends

Re: [PR] [MINOR] Add table name and range msg for streaming reads logs [hudi]

2023-10-24 Thread via GitHub
zhuanshenbsj1 commented on code in PR #9912: URL: https://github.com/apache/hudi/pull/9912#discussion_r1371102567 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/InstantRange.java: ## @@ -34,10 +34,12 @@ public abstract class InstantRange implements Serializable {

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1778455957 ## CI report: * 985e9f099aff341d7d0cec4384ef82b7dcdd4de8 Azure:

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
wecharyu commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1371097371 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

[jira] [Closed] (HUDI-6900) TestInsertTable "Test Bulk Insert Into Consistent Hashing Bucket Index Table" is failing continuously

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6900. Fix Version/s: 1.0.0 Resolution: Fixed Fixed via master branch:

[hudi] branch master updated: [MINOR] Add tests on combine parallelism (#9731)

2023-10-24 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 051eb0e930e [MINOR] Add tests on combine

Re: [PR] [MINOR] Add tests on combine parallelism [hudi]

2023-10-24 Thread via GitHub
nsivabalan merged PR #9731: URL: https://github.com/apache/hudi/pull/9731 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-24 Thread via GitHub
nsivabalan commented on code in PR #9892: URL: https://github.com/apache/hudi/pull/9892#discussion_r1371085714 ## hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroPayload.java: ## @@ -39,11 +42,19 @@ public class HoodieAvroPayload implements HoodieRecordPayload

Re: [PR] [HUDI-6877] Fix avro read issue after ALTER TABLE RENAME DDL on Spark3_1 [hudi]

2023-10-24 Thread via GitHub
voonhous commented on code in PR #9752: URL: https://github.com/apache/hudi/pull/9752#discussion_r1371085479 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java: ## @@ -115,6 +114,35 @@ public byte[] getContentBytes() throws IOException {

Re: [PR] [HUDI-6975] Optimize the code of DayBasedCompactionStrategy [hudi]

2023-10-24 Thread via GitHub
ksmou commented on code in PR #9911: URL: https://github.com/apache/hudi/pull/9911#discussion_r1371082887 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/DayBasedCompactionStrategy.java: ## @@ -63,21 +60,9 @@ public Comparator

[jira] [Created] (HUDI-6979) support EventTimeBasedCompactionStrategy

2023-10-24 Thread Kong Wei (Jira)
Kong Wei created HUDI-6979: -- Summary: support EventTimeBasedCompactionStrategy Key: HUDI-6979 URL: https://issues.apache.org/jira/browse/HUDI-6979 Project: Apache Hudi Issue Type: New Feature

Re: [PR] [HUDI-6969] Add speed limit for stream read [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9904: URL: https://github.com/apache/hudi/pull/9904#discussion_r137106 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/IncrementalInputSplits.java: ## @@ -269,6 +269,9 @@ public Result inputSplits( Result

[jira] [Commented] (HUDI-6968) remove block logical in BulkInsertWriteFunction#open

2023-10-24 Thread Jing Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779295#comment-17779295 ] Jing Zhang commented on HUDI-6968: -- Fixed via master branch: f05b5fc9db38e0bc4ccc2941cccf049991b67db2 >

[jira] [Closed] (HUDI-6968) remove block logical in BulkInsertWriteFunction#open

2023-10-24 Thread Jing Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhang closed HUDI-6968. Fix Version/s: 1.0.0 Resolution: Fixed > remove block logical in BulkInsertWriteFunction#open >

Re: [PR] [HUDI-6975] Optimize the code of DayBasedCompactionStrategy [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9911: URL: https://github.com/apache/hudi/pull/9911#discussion_r1371010223 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/DayBasedCompactionStrategy.java: ## @@ -63,21 +60,9 @@ public Comparator

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1371009429 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1368068891 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778351985 ## CI report: * b8bc65dc87cfd1305634bf16f96a97944ce85816 Azure:

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1778352024 ## CI report: * c140ff462f58b649d45c782ce072b683cd908c1c Azure:

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
boneanxs commented on PR #9887: URL: https://github.com/apache/hudi/pull/9887#issuecomment-1778351293 @danny0405 Yea, sure, will raise the pr soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on PR #9887: URL: https://github.com/apache/hudi/pull/9887#issuecomment-1778350430 @stream2000 @boneanxs Merge it first because it looks like a bug fix. Can you finalize it with following up PRs with more tests or probable the correct fix with `#abort`. -- This is an

Re: [I] [SUPPORT]flink-sql write hudi use TIMESTAMP, when hive query, it get time+8h question, use TIMESTAMP_LTZ, the hive schema is bigint but timestamp [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on issue #9864: URL: https://github.com/apache/hudi/issues/9864#issuecomment-1778351080 > but TIMESTAMP cannot be changed to long What do you mean by changed to long? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
boneanxs commented on PR #9887: URL: https://github.com/apache/hudi/pull/9887#issuecomment-1778346251 > we can confirm that datasource v2 won't waiting for all subtasks to be canceled before calling `org.apache.hudi.table.action.commit.BulkInsertDataInternalWriterHelper#abort`

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1778343885 ## CI report: * c140ff462f58b649d45c782ce072b683cd908c1c Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778343787 ## CI report: * b8bc65dc87cfd1305634bf16f96a97944ce85816 Azure:

Re: [PR] [HUDI-6975] Optimize the implementation of DayBasedCompactionStrategy [hudi]

2023-10-24 Thread via GitHub
ksmou commented on code in PR #9911: URL: https://github.com/apache/hudi/pull/9911#discussion_r1370998924 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/DayBasedCompactionStrategy.java: ## @@ -63,21 +60,9 @@ public Comparator

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
stream2000 commented on code in PR #9887: URL: https://github.com/apache/hudi/pull/9887#discussion_r1370998616 ## hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java: ## @@ -97,7 +97,6 @@ public void commit(List

[jira] [Closed] (HUDI-6959) Do not rollback current instant when bulk insert as row failed

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6959. Resolution: Fixed Fixed via master branch: 65dd645b487a61fbca7e4e4b849d1f2f1ec143f9 > Do not rollback

[jira] [Updated] (HUDI-6959) Do not rollback current instant when bulk insert as row failed

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6959: - Fix Version/s: 1.0.0 0.14.1 > Do not rollback current instant when bulk insert as row

[hudi] branch master updated: [HUDI-6959] Bulk insert as row do not rollback failed instant on abort (#9887)

2023-10-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 65dd645b487 [HUDI-6959] Bulk insert as row do

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
danny0405 merged PR #9887: URL: https://github.com/apache/hudi/pull/9887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778336139 ## CI report: * b8bc65dc87cfd1305634bf16f96a97944ce85816 Azure:

Re: [PR] [MINOR] Add tests on combine parallelism [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1778335971 ## CI report: * 047941b66ee52a99f626fd0dadb72581d9855385 Azure:

Re: [PR] [HUDI-6975] Optimize the implementation of DayBasedCompactionStrategy [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on code in PR #9911: URL: https://github.com/apache/hudi/pull/9911#discussion_r1370996694 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/DayBasedCompactionStrategy.java: ## @@ -63,21 +60,9 @@ public Comparator

[jira] [Closed] (HUDI-6929) Lazy loading dynamically for CompletionTimeQueryView

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6929. Resolution: Fixed Fixed via master branch: bb8fc3e9f632a1fc3647fda63d482849355df2b7 > Lazy loading

[jira] [Updated] (HUDI-6962) Correct the behavior of bulk insert for NB-CC

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6962: - Fix Version/s: 1.0.0 > Correct the behavior of bulk insert for NB-CC >

[jira] [Closed] (HUDI-6962) Correct the behavior of bulk insert for NB-CC

2023-10-24 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6962. Resolution: Fixed Fixed via master branch: f05b5fc9db38e0bc4ccc2941cccf049991b67db2 > Correct the behavior

[hudi] branch master updated: [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC (#9896)

2023-10-24 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f05b5fc9db3 [HUDI-6962] Fix the conflicts

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-24 Thread via GitHub
danny0405 merged PR #9896: URL: https://github.com/apache/hudi/pull/9896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-24 Thread via GitHub
danny0405 commented on PR #9896: URL: https://github.com/apache/hudi/pull/9896#issuecomment-1778327433 The failed test is known to be flaky: `TestHoodieLogFormat.testAvroLogRecordReaderWithMixedInsertsCorruptsRollbackAndMergedLogBlock` :

Re: [I] [SUPPORT]flink 写hudi 同步hive后,timestamp字段为什么是bigint类型,如何才能让同步到hive的字段保持timestamp类型 [hudi]

2023-10-24 Thread via GitHub
linrongjun-l commented on issue #9766: URL: https://github.com/apache/hudi/issues/9766#issuecomment-1778312506 > > Before release 0.14.0, there is a sync param `hive_sync.support_timestamp`, when enabled, the `Timestamp(6)` type would be synced as `TIMESTAMP` in hive, since release 0.14.0,

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9914: URL: https://github.com/apache/hudi/pull/9914#issuecomment-1778296527 ## CI report: * 6aa578288e31414d8f13c37525ed4e2b7d9a6521 Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778296319 ## CI report: * b8bc65dc87cfd1305634bf16f96a97944ce85816 Azure:

Re: [PR] [MINOR] Add tests on combine parallelism [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1778296103 ## CI report: * 047941b66ee52a99f626fd0dadb72581d9855385 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [MINOR] Add tests on combine parallelism [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1778294686 CI is green. https://github.com/apache/hudi/assets/2497195/b14e4414-fbb5-4f1b-a3e0-5a2d8335775d;> -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9914: URL: https://github.com/apache/hudi/pull/9914#issuecomment-1778289302 ## CI report: * 6aa578288e31414d8f13c37525ed4e2b7d9a6521 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6978) Fix TestMergeIntoTable2 test

2023-10-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6978: Description: For the test TestMergeIntoTable2@"Test only insert for source table in dup key without

[jira] [Updated] (HUDI-6978) Fix TestMergeIntoTable2 test

2023-10-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6978: Description: For the test "Test only insert for source table in dup key without preCombineField" @"Test

[jira] [Updated] (HUDI-6978) Fix TestMergeIntoTable2 test

2023-10-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6978: Description: For the test TestMergeIntoTable2@"Test only insert for source table in dup key without

[jira] [Updated] (HUDI-6978) Fix TestMergeIntoTable2 test

2023-10-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6978: Description: For @"Test only insert for source table in dup key without preCombineField" > Fix

[jira] [Created] (HUDI-6978) Fix TestMergeIntoTable2 test

2023-10-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6978: --- Summary: Fix TestMergeIntoTable2 test Key: HUDI-6978 URL: https://issues.apache.org/jira/browse/HUDI-6978 Project: Apache Hudi Issue Type: Improvement

[I] [SUPPORT] Control file sizing during FULL_RECORD bootstrap mode [hudi]

2023-10-24 Thread via GitHub
fenil25 opened a new issue, #9915: URL: https://github.com/apache/hudi/issues/9915 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? Yes - Join the mailing list to engage in conversations and get faster support at

Re: [PR] Test ci [hudi]

2023-10-24 Thread via GitHub
kkalanda-score closed pull request #9095: Test ci URL: https://github.com/apache/hudi/pull/9095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6551] A new slashed month partition value extractor [hudi]

2023-10-24 Thread via GitHub
yihua closed pull request #9184: [HUDI-6551] A new slashed month partition value extractor URL: https://github.com/apache/hudi/pull/9184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1778089960 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * 7c353cd134d555bf0adfb50a64f012b609e75308 Azure:

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9914: URL: https://github.com/apache/hudi/pull/9914#issuecomment-1778090503 ## CI report: * 6aa578288e31414d8f13c37525ed4e2b7d9a6521 Azure:

Re: [PR] [HUDI-6551] A new slashed month partition value extractor [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9184: URL: https://github.com/apache/hudi/pull/9184#issuecomment-1778090063 Closing this PR now. @banank1989 feel free to reopen it you need additional functionality. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Test ci [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9095: URL: https://github.com/apache/hudi/pull/9095#issuecomment-1778088836 @kkalanda-score do you still need this PR? If not, the PR should be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [HUDI-6898] Medatawriter closing in tests, update logging [hudi]

2023-10-24 Thread via GitHub
yihua merged PR #9768: URL: https://github.com/apache/hudi/pull/9768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1778076328 I discussed the comments with @danny0405 offline. Two things to address in this PR: (1) Instead of putting both partial and full schemas in the log block header, when partial updates

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1370828538 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala: ## @@ -411,10 +414,14 @@ object ExpressionPayload {

Re: [PR] [HUDI-6836] Shutting down deltastreamer in tests and shutting down metrics for write client [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9667: URL: https://github.com/apache/hudi/pull/9667#issuecomment-1778007738 @pratyakshsharma are you good with the changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [HUDI-6877] Fix avro read issue after ALTER TABLE RENAME DDL on Spark3_1 [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9752: URL: https://github.com/apache/hudi/pull/9752#discussion_r1370788996 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java: ## @@ -115,6 +114,35 @@ public byte[] getContentBytes() throws IOException {

Re: [PR] [HUDI-6898] Medatawriter closing in tests, update logging [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9768: URL: https://github.com/apache/hudi/pull/9768#issuecomment-1778003218 ## CI report: * 55beb62d168b2c9b9d99f0c3765637d441f58b5f Azure:

Re: [PR] [HUDI-6877] Fix avro read issue after ALTER TABLE RENAME DDL on Spark3_1 [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9752: URL: https://github.com/apache/hudi/pull/9752#issuecomment-1777997252 > Seems we have plan to mograte to avro above 1.8.2 right cc @yihua ~ The Avro dependency version is tied to Spark version and Avro 1.8.2 is tied to Spark 3.1. -- This is an automated

Re: [PR] [HUDI-6895][WIP] Change default timeline timezone from local to UTC [hudi]

2023-10-24 Thread via GitHub
yihua commented on PR #9794: URL: https://github.com/apache/hudi/pull/9794#issuecomment-1777989084 @codope do we still plan to land this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [HUDI-6959] Bulk insert V2 do not rollback failed instant on abort [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9887: URL: https://github.com/apache/hudi/pull/9887#discussion_r1370778292 ## hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java: ## @@ -97,7 +97,6 @@ public void commit(List

Re: [PR] [HUDI-6790] Support incremental/CDC queries using HadoopFsRelation [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9888: URL: https://github.com/apache/hudi/pull/9888#discussion_r1370761534 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionCDCFileGroupMapping.scala: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache

Re: [PR] [HUDI-6790] Support incremental/CDC queries using HadoopFsRelation [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9888: URL: https://github.com/apache/hudi/pull/9888#discussion_r1370759016 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala: ## @@

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1777922386 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * 0fe4d74eb04601d878a44c6d8892168e1e321d1a Azure:

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9914: URL: https://github.com/apache/hudi/pull/9914#issuecomment-1777909468 ## CI report: * 6aa578288e31414d8f13c37525ed4e2b7d9a6521 Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1777908868 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * 0fe4d74eb04601d878a44c6d8892168e1e321d1a Azure:

Re: [PR] [HUDI-6977] Upgrade hadoop version from 2.10.1 to 2.10.2 [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9914: URL: https://github.com/apache/hudi/pull/9914#issuecomment-1777840859 ## CI report: * 6aa578288e31414d8f13c37525ed4e2b7d9a6521 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6898] Medatawriter closing in tests, update logging [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9768: URL: https://github.com/apache/hudi/pull/9768#issuecomment-1777840295 ## CI report: * 55beb62d168b2c9b9d99f0c3765637d441f58b5f Azure:

Re: [PR] [HUDI-6898] Medatawriter closing in tests, update logging [hudi]

2023-10-24 Thread via GitHub
hudi-bot commented on PR #9768: URL: https://github.com/apache/hudi/pull/9768#issuecomment-1777825883 ## CI report: * 55beb62d168b2c9b9d99f0c3765637d441f58b5f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6898] Medatawriter closing in tests, update logging [hudi]

2023-10-24 Thread via GitHub
yihua commented on code in PR #9768: URL: https://github.com/apache/hudi/pull/9768#discussion_r1370671655 ## pom.xml: ## @@ -115,7 +115,7 @@ 2.17.2 1.7.36 2.9.9 -2.10.1 +2.10.2 Review Comment: Separate upgrade PR here: #9914 -- This is an

[jira] [Updated] (HUDI-6977) Upgrade hadoop version from 2.10.1 to 2.10.2

2023-10-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6977: - Labels: pull-request-available (was: ) > Upgrade hadoop version from 2.10.1 to 2.10.2 >

  1   2   >