[GitHub] [hudi] aiwenmo commented on issue #5513: [SUPPORT] Sync realtime whole mysql database to hudi failed when using flink datastream api

2022-05-05 Thread GitBox
aiwenmo commented on issue #5513: URL: https://github.com/apache/hudi/issues/5513#issuecomment-1119280273 > You need to set up the key generator clazz correctly. thx. Your method is also OK. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] hudi-bot commented on pull request #5402: [WIP] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default

2022-05-05 Thread GitBox
hudi-bot commented on PR #5402: URL: https://github.com/apache/hudi/pull/5402#issuecomment-1119276503 ## CI report: * 8c6f6e19940ce7ac04dfcfce52da3ccdaf3a8b0f UNKNOWN * b7ed5a5b237814ee2f0266b0ec1345f23d69d94a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5402: [WIP] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default

2022-05-05 Thread GitBox
hudi-bot commented on PR #5402: URL: https://github.com/apache/hudi/pull/5402#issuecomment-1119275155 ## CI report: * 8c6f6e19940ce7ac04dfcfce52da3ccdaf3a8b0f UNKNOWN * b7ed5a5b237814ee2f0266b0ec1345f23d69d94a Azure:

[GitHub] [hudi] danny0405 commented on issue #5513: [SUPPORT] Sync realtime whole mysql database to hudi failed when using flink datastream api

2022-05-05 Thread GitBox
danny0405 commented on issue #5513: URL: https://github.com/apache/hudi/issues/5513#issuecomment-1119274099 You need to set up the key generator clazz correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] yihua commented on pull request #5402: [WIP] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default

2022-05-05 Thread GitBox
yihua commented on PR #5402: URL: https://github.com/apache/hudi/pull/5402#issuecomment-1119273974 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] danny0405 commented on issue #5382: [SUPPORT] org.apache.hudi.hadoop.hive.HoodieCombineRealtimeFileSplit cannot be cast to org.apache.hadoop.hive.shims.HadoopShimsSecure$InputSplitShim

2022-05-05 Thread GitBox
danny0405 commented on issue #5382: URL: https://github.com/apache/hudi/issues/5382#issuecomment-1119273063 Can you ask for help in the dingTalk group, some people might solve this problem already. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] danny0405 commented on a diff in pull request #4739: [HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS

2022-05-05 Thread GitBox
danny0405 commented on code in PR #4739: URL: https://github.com/apache/hudi/pull/4739#discussion_r866499815 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1301,4 +1359,33 @@ public void close() {

[GitHub] [hudi] parisni commented on issue #5484: [SUPPORT] Hive Sync + AWS Data Catalog failling with Hudi 0.11.0

2022-05-05 Thread GitBox
parisni commented on issue #5484: URL: https://github.com/apache/hudi/issues/5484#issuecomment-1119271257 what about removing that `'hoodie.meta.sync.client.tool.class': 'org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool'` ? This will work as 0.10 with the regular hive sync connector, which

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1119255595 ## CI report: * 85be21c2ec99670eb9e0e697259e407d78bf4524 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1119253350 ## CI report: * 99e207f779a258ff32c57c9d1b962d772213c081 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1119252227 ## CI report: * 99e207f779a258ff32c57c9d1b962d772213c081 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5073: [HUDI-3675] Adding post write termination strategy to deltastreamer continuous mode

2022-05-05 Thread GitBox
hudi-bot commented on PR #5073: URL: https://github.com/apache/hudi/pull/5073#issuecomment-1119208823 ## CI report: * a1322fbeb11fe5bb71cd5d70f13147bf8a036996 Azure:

[GitHub] [hudi] aiwenmo opened a new issue, #5513: [SUPPORT] Sync realtime whole mysql database to hudi failed when using flink datastream api

2022-05-05 Thread GitBox
aiwenmo opened a new issue, #5513: URL: https://github.com/apache/hudi/issues/5513 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at

[GitHub] [hudi] hudi-bot commented on pull request #5073: [HUDI-3675] Adding post write termination strategy to deltastreamer continuous mode

2022-05-05 Thread GitBox
hudi-bot commented on PR #5073: URL: https://github.com/apache/hudi/pull/5073#issuecomment-1119186216 ## CI report: * 7042acc09b38acd8741c89ca77e99bdacaa6 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5073: [HUDI-3675] Adding post write termination strategy to deltastreamer continuous mode

2022-05-05 Thread GitBox
hudi-bot commented on PR #5073: URL: https://github.com/apache/hudi/pull/5073#issuecomment-1119184975 ## CI report: * 7042acc09b38acd8741c89ca77e99bdacaa6 Azure:

[GitHub] [hudi] lanyuanxiaoyao commented on a diff in pull request #5473: [HUDI-4003] Try to read all the log file to parse schema

2022-05-05 Thread GitBox
lanyuanxiaoyao commented on code in PR #5473: URL: https://github.com/apache/hudi/pull/5473#discussion_r866425592 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -109,13 +110,18 @@ private MessageType getTableParquetSchemaFromDataFile()

[jira] [Updated] (HUDI-4048) Upgrade Hudi version in presto-hive

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4048: -- Status: In Progress (was: Open) > Upgrade Hudi version in presto-hive >

[jira] [Updated] (HUDI-4048) Upgrade Hudi version in presto-hive

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4048: -- Status: Patch Available (was: In Progress) > Upgrade Hudi version in presto-hive >

[jira] [Updated] (HUDI-3960) Update HudiRealtimeSplitConverter to correctly instantiate HoodieRealtimeFileSplit

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3960: -- Status: Patch Available (was: In Progress) > Update HudiRealtimeSplitConverter to

[jira] [Updated] (HUDI-3960) Update HudiRealtimeSplitConverter to correctly instantiate HoodieRealtimeFileSplit

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3960: -- Status: In Progress (was: Open) > Update HudiRealtimeSplitConverter to correctly

[jira] [Resolved] (HUDI-4031) Avoid clustering update handling when clustering is disabled

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-4031. --- > Avoid clustering update handling when clustering is disabled >

[jira] [Updated] (HUDI-3960) Update HudiRealtimeSplitConverter to correctly instantiate HoodieRealtimeFileSplit

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-3960: -- Story Points: 1 > Update HudiRealtimeSplitConverter to correctly instantiate > HoodieRealtimeFileSplit

[jira] [Updated] (HUDI-4050) Upgrade Hudi version to 0.11.0 in the connector

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-4050: -- Sprint: 2022/05/02 > Upgrade Hudi version to 0.11.0 in the connector >

[jira] [Created] (HUDI-4050) Upgrade Hudi version to 0.11.0 in the connector

2022-05-05 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-4050: - Summary: Upgrade Hudi version to 0.11.0 in the connector Key: HUDI-4050 URL: https://issues.apache.org/jira/browse/HUDI-4050 Project: Apache Hudi Issue Type: Task

[jira] [Updated] (HUDI-4049) Upgrade Hudi version in the connector

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-4049: -- Sprint: 2022/05/02 > Upgrade Hudi version in the connector > - > >

[jira] [Updated] (HUDI-4048) Upgrade Hudi version in presto-hive

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-4048: -- Sprint: 2022/05/02 > Upgrade Hudi version in presto-hive > --- > >

[jira] [Updated] (HUDI-3960) Update HudiRealtimeSplitConverter to correctly instantiate HoodieRealtimeFileSplit

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-3960: -- Sprint: 2022/05/02 > Update HudiRealtimeSplitConverter to correctly instantiate >

[jira] [Created] (HUDI-4048) Upgrade Hudi version in presto-hive

2022-05-05 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-4048: - Summary: Upgrade Hudi version in presto-hive Key: HUDI-4048 URL: https://issues.apache.org/jira/browse/HUDI-4048 Project: Apache Hudi Issue Type: Task

[jira] [Created] (HUDI-4049) Upgrade Hudi version in the connector

2022-05-05 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-4049: - Summary: Upgrade Hudi version in the connector Key: HUDI-4049 URL: https://issues.apache.org/jira/browse/HUDI-4049 Project: Apache Hudi Issue Type: Task

[jira] [Assigned] (HUDI-3960) Update HudiRealtimeSplitConverter to correctly instantiate HoodieRealtimeFileSplit

2022-05-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-3960: - Assignee: Sagar Sumit > Update HudiRealtimeSplitConverter to correctly instantiate >

[GitHub] [hudi] hudi-bot commented on pull request #5073: [HUDI-3675] Adding post write termination strategy to deltastreamer continuous mode

2022-05-05 Thread GitBox
hudi-bot commented on PR #5073: URL: https://github.com/apache/hudi/pull/5073#issuecomment-1119162871 ## CI report: * 2e170509cffd77d4124ecbe337cd018c96a621fd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1119161967 ## CI report: * c7324b8703ec68a4fd57195028d7977ab6862ac5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1119161930 ## CI report: * 6f9f0539ab0102ff502e7985a453c4dddea6193a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5073: [HUDI-3675] Adding post write termination strategy to deltastreamer continuous mode

2022-05-05 Thread GitBox
hudi-bot commented on PR #5073: URL: https://github.com/apache/hudi/pull/5073#issuecomment-1119161676 ## CI report: * 2e170509cffd77d4124ecbe337cd018c96a621fd Azure:

[jira] [Updated] (HUDI-3995) Bulk insert row writer perf improvements

2022-05-05 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3995: -- Description: *EDIT* ** While investigating, perf hits in the Bulk Insert a few issues were

[jira] [Updated] (HUDI-3995) Bulk insert row writer perf improvements

2022-05-05 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3995: -- Description: *EDIT* *-* While investigating, perf hits in the Bulk Insert a few issues

[jira] [Updated] (HUDI-4036) Investigate whether meta fields could be omitted completely

2022-05-05 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-4036: -- Epic Link: HUDI-3249 > Investigate whether meta fields could be omitted completely >

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #4480: [HUDI-3123] consistent hashing index: basic write path (upsert/insert)

2022-05-05 Thread GitBox
alexeykudinkin commented on code in PR #4480: URL: https://github.com/apache/hudi/pull/4480#discussion_r866314882 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/BucketIdentifier.java: ## @@ -22,41 +22,50 @@ import

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1119147087 ## CI report: * 2acc8007cc153d7d4a228e126ef706e5bb25cfbb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1119145841 ## CI report: * 2acc8007cc153d7d4a228e126ef706e5bb25cfbb Azure:

[jira] [Updated] (HUDI-4018) Prepare minimal set of yamls to be tested against any write mode and against any query engine

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4018: -- Reviewers: Raymond Xu > Prepare minimal set of yamls to be tested against any write

[jira] [Updated] (HUDI-4018) Prepare minimal set of yamls to be tested against any write mode and against any query engine

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4018: -- Reviewers: Raymond Xu (was: Raymond Xu) > Prepare minimal set of yamls to be tested

[jira] [Updated] (HUDI-3873) 0.11 release blog

2022-05-05 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3873: Status: Patch Available (was: In Progress) > 0.11 release blog > - > >

[jira] [Commented] (HUDI-4018) Prepare minimal set of yamls to be tested against any write mode and against any query engine

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532554#comment-17532554 ] sivabalan narayanan commented on HUDI-4018: --- # simple sanity (1 insert, 1 upsert, 1

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1119111963 ## CI report: * 8b22298c933375b9af687093cecc68603d7e3c3d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1119110521 ## CI report: * 8b22298c933375b9af687093cecc68603d7e3c3d Azure:

[GitHub] [hudi] vicuna96 commented on issue #4700: [SUPPORT] Adding new column to table is not propagated to Hive via HMS sync mode

2022-05-05 Thread GitBox
vicuna96 commented on issue #4700: URL: https://github.com/apache/hudi/issues/4700#issuecomment-1119074796 Hi @xiarixiaoyao , @nsivabalan , is there any known workaround for this? It does seem like the problem is that it's trying to use the implementation from

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5393: [MINOR] follow up HUDI-3921, address all comments

2022-05-05 Thread GitBox
alexeykudinkin commented on code in PR #5393: URL: https://github.com/apache/hudi/pull/5393#discussion_r866304797 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -840,13 +840,9 @@ private static Object rewriteRecordWithNewSchema(Object oldRecord,

[hudi] branch master updated (d794f4fbf9 -> abb4893b25)

2022-05-05 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from d794f4fbf9 [MINOR] Optimize code logic (#5499) add abb4893b25 [HUDI-2875] Make HoodieParquetWriter Thread safe and

[GitHub] [hudi] yihua merged pull request #4264: [HUDI-2875] Make HoodieParquetWriter Thread safe and memory executor …

2022-05-05 Thread GitBox
yihua merged PR #4264: URL: https://github.com/apache/hudi/pull/4264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer

2022-05-05 Thread GitBox
alexeykudinkin commented on code in PR #5269: URL: https://github.com/apache/hudi/pull/5269#discussion_r866299432 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java: ## @@ -31,16 +33,30 @@ private static final long serialVersionUID =

[GitHub] [hudi] alexeykudinkin commented on pull request #4264: [HUDI-2875] Make HoodieParquetWriter Thread safe and memory executor …

2022-05-05 Thread GitBox
alexeykudinkin commented on PR #4264: URL: https://github.com/apache/hudi/pull/4264#issuecomment-1119021233 LGTM, @nsivabalan @yihua can you please help land that one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1118874858 ## CI report: * 2acc8007cc153d7d4a228e126ef706e5bb25cfbb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1118859963 ## CI report: * 8b22298c933375b9af687093cecc68603d7e3c3d Azure:

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5473: [HUDI-4003] Try to read all the log file to parse schema

2022-05-05 Thread GitBox
alexeykudinkin commented on code in PR #5473: URL: https://github.com/apache/hudi/pull/5473#discussion_r866120175 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -109,13 +110,18 @@ private MessageType getTableParquetSchemaFromDataFile()

[GitHub] [hudi] hudi-bot commented on pull request #4676: [HUDI-3304] support partial update on mor table

2022-05-05 Thread GitBox
hudi-bot commented on PR #4676: URL: https://github.com/apache/hudi/pull/4676#issuecomment-1118817192 ## CI report: * 0e3f7e76cfdf18b34aafabf2a7949f6b1e62bddc Azure:

[GitHub] [hudi] alexeykudinkin commented on pull request #5287: [HUDI-3849] AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration

2022-05-05 Thread GitBox
alexeykudinkin commented on PR #5287: URL: https://github.com/apache/hudi/pull/5287#issuecomment-1118812053 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #4739: [HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS

2022-05-05 Thread GitBox
alexeykudinkin commented on code in PR #4739: URL: https://github.com/apache/hudi/pull/4739#discussion_r866116760 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1301,4 +1359,33 @@ public void close() {

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1118804996 ## CI report: * 2acc8007cc153d7d4a228e126ef706e5bb25cfbb Azure:

[jira] [Updated] (HUDI-4017) Spark sql tests as part of github actions for diff spark versions

2022-05-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4017: - Labels: pull-request-available (was: ) > Spark sql tests as part of github actions for diff

[GitHub] [hudi] hudi-bot commented on pull request #5512: [HUDI-4017] Improve spark sql coverage

2022-05-05 Thread GitBox
hudi-bot commented on PR #5512: URL: https://github.com/apache/hudi/pull/5512#issuecomment-1118796395 ## CI report: * 2acc8007cc153d7d4a228e126ef706e5bb25cfbb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #4676: [HUDI-3304] support partial update on mor table

2022-05-05 Thread GitBox
hudi-bot commented on PR #4676: URL: https://github.com/apache/hudi/pull/4676#issuecomment-1118784042 ## CI report: * c9ee1edc0285fb17a9455cc5ca52072854d66a91 Azure:

[GitHub] [hudi] yihua merged pull request #5499: [MINOR] Optimize code logic

2022-05-05 Thread GitBox
yihua merged PR #5499: URL: https://github.com/apache/hudi/pull/5499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [MINOR] Optimize code logic (#5499)

2022-05-05 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d794f4fbf9 [MINOR] Optimize code logic (#5499)

[GitHub] [hudi] hudi-bot commented on pull request #4676: [HUDI-3304] support partial update on mor table

2022-05-05 Thread GitBox
hudi-bot commented on PR #4676: URL: https://github.com/apache/hudi/pull/4676#issuecomment-1118775741 ## CI report: * c9ee1edc0285fb17a9455cc5ca52072854d66a91 Azure:

[jira] [Updated] (HUDI-4047) hoodie.avro.schema.validate error message refact

2022-05-05 Thread Istvan Darvas (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Darvas updated HUDI-4047: Description: Hi Guys!   I have just used the schema validation and works as a charm, but :)   A

[jira] [Updated] (HUDI-4047) hoodie.avro.schema.validate error message refact

2022-05-05 Thread Istvan Darvas (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Darvas updated HUDI-4047: Priority: Minor (was: Major) > hoodie.avro.schema.validate error message refact >

[jira] [Created] (HUDI-4047) hoodie.avro.schema.validate error message refact

2022-05-05 Thread Istvan Darvas (Jira)
Istvan Darvas created HUDI-4047: --- Summary: hoodie.avro.schema.validate error message refact Key: HUDI-4047 URL: https://issues.apache.org/jira/browse/HUDI-4047 Project: Apache Hudi Issue Type:

[GitHub] [hudi] fengjian428 commented on a diff in pull request #4676: [HUDI-3304] support partial update on mor table

2022-05-05 Thread GitBox
fengjian428 commented on code in PR #4676: URL: https://github.com/apache/hudi/pull/4676#discussion_r866079366 ## hudi-common/src/test/java/org/apache/hudi/common/model/TestPartialUpdateAvroPayload.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [hudi] fengjian428 commented on a diff in pull request #4676: [HUDI-3304] support partial update on mor table

2022-05-05 Thread GitBox
fengjian428 commented on code in PR #4676: URL: https://github.com/apache/hudi/pull/4676#discussion_r866078762 ## hudi-common/src/test/java/org/apache/hudi/common/model/TestPartialUpdateAvroPayload.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [hudi] hudi-bot commented on pull request #5510: [minor] fix the flacky test ITTestHoodieDataSource#testStreamWriteBat…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5510: URL: https://github.com/apache/hudi/pull/5510#issuecomment-1118719037 ## CI report: * ec40e1e3c0495c301a964a1fa1a740dc6a1d0e00 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118698058 ## CI report: * ad175e8b93bc54e10a846cbcb8caad988ce8280b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118651795 ## CI report: * 28c50bbaacaa1b70811f02c3a1e02138dfd09e15 Azure:

[GitHub] [hudi] VitoMakarevich opened a new issue, #5511: [SUPPORT] Inremental query from the beginning of time

2022-05-05 Thread GitBox
VitoMakarevich opened a new issue, #5511: URL: https://github.com/apache/hudi/issues/5511 **Describe the problem you faced** Incremental query with `begin.instanttime` less than the first commit time is different, depending on how many commits added. **To Reproduce**

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1118638660 ## CI report: * 2d627024cd13ca2389008a649b3defd9fba3b04c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1118634615 ## CI report: * 99e207f779a258ff32c57c9d1b962d772213c081 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5501: [HUDI-4018][HUDI-4027] Adding integ test yamls for immutable use-cases. Added delete partition support to integ tests

2022-05-05 Thread GitBox
hudi-bot commented on PR #5501: URL: https://github.com/apache/hudi/pull/5501#issuecomment-1118634531 ## CI report: * 2d627024cd13ca2389008a649b3defd9fba3b04c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5510: [minor] fix the flacky test ITTestHoodieDataSource#testStreamWriteBat…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5510: URL: https://github.com/apache/hudi/pull/5510#issuecomment-1118629985 ## CI report: * ec40e1e3c0495c301a964a1fa1a740dc6a1d0e00 Azure:

[jira] [Updated] (HUDI-3957) Evaluate Support for spark2 and scala12

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3957: -- Status: Patch Available (was: In Progress) > Evaluate Support for spark2 and scala12

[jira] [Updated] (HUDI-4018) Prepare minimal set of yamls to be tested against any write mode and against any query engine

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4018: -- Status: Patch Available (was: In Progress) > Prepare minimal set of yamls to be tested

[jira] [Updated] (HUDI-4027) add support to test non-core write operations (insert overwrite, delete partitions) to integ test framework

2022-05-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-4027: -- Status: Patch Available (was: In Progress) > add support to test non-core write

[GitHub] [hudi] hudi-bot commented on pull request #5510: [minor] fix the flacky test ITTestHoodieDataSource#testStreamWriteBat…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5510: URL: https://github.com/apache/hudi/pull/5510#issuecomment-1118625639 ## CI report: * ec40e1e3c0495c301a964a1fa1a740dc6a1d0e00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118625554 ## CI report: * 28c50bbaacaa1b70811f02c3a1e02138dfd09e15 Azure:

[GitHub] [hudi] andykrk commented on issue #4604: [SUPPORT] Archive functionality fails

2022-05-05 Thread GitBox
andykrk commented on issue #4604: URL: https://github.com/apache/hudi/issues/4604#issuecomment-1118619225 @nsivabalan We need to park this item temporarily. We may get some additional resources to work on that on our side after this. I will keep you posted on that. -- This is an

[GitHub] [hudi] danny0405 opened a new pull request, #5510: [minor] fix the flacky test ITTestHoodieDataSource#testStreamWriteBat…

2022-05-05 Thread GitBox
danny0405 opened a new pull request, #5510: URL: https://github.com/apache/hudi/pull/5510 …chReadOptimized ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*

[GitHub] [hudi] hudi-bot commented on pull request #5506: [HUDI-4042] Support truncate-partition for Spark-3.2

2022-05-05 Thread GitBox
hudi-bot commented on PR #5506: URL: https://github.com/apache/hudi/pull/5506#issuecomment-1118579317 ## CI report: * 6f6ffdafc1e6ade28a7d340024905374353032af Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118579254 ## CI report: * 28c50bbaacaa1b70811f02c3a1e02138dfd09e15 Azure:

[jira] [Closed] (HUDI-4043) Clean the marker files for compaction rollback

2022-05-05 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-4043. Resolution: Won't Fix > Clean the marker files for compaction rollback >

[GitHub] [hudi] danny0405 closed pull request #5508: [HUDI-4043] Clean the marker files for compaction rollback

2022-05-05 Thread GitBox
danny0405 closed pull request #5508: [HUDI-4043] Clean the marker files for compaction rollback URL: https://github.com/apache/hudi/pull/5508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] danny0405 commented on pull request #5508: [HUDI-4043] Clean the marker files for compaction rollback

2022-05-05 Thread GitBox
danny0405 commented on PR #5508: URL: https://github.com/apache/hudi/pull/5508#issuecomment-1118567331 Close because `BaseRollbackActionExecutor.runRollback` already did that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] hudi-bot commented on pull request #5445: [HUDI-3953]Flink Hudi module should support low-level source and sink…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5445: URL: https://github.com/apache/hudi/pull/5445#issuecomment-1118566015 ## CI report: * 1e9b3ac4c34f97f5ccf3a639cc74b7081eeaab37 UNKNOWN * a5669a78b314a5dc4166bcc4d41d2a377653da75 UNKNOWN * 6426727bb88fce863d7aa50ef04b2cdac7acb2e2 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1118561927 ## CI report: * 99e207f779a258ff32c57c9d1b962d772213c081 Azure:

[jira] [Updated] (HUDI-4046) spark.read.load API

2022-05-05 Thread Istvan Darvas (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Darvas updated HUDI-4046: Description: Hi Guys! I would like to controll the number of partions which will be read by HUDI.

[jira] [Updated] (HUDI-4046) spark.read.load API

2022-05-05 Thread Istvan Darvas (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Darvas updated HUDI-4046: Description: Hi Guys! I would like to controll the number of partions which will be read by HUDI.

[GitHub] [hudi] hudi-bot commented on pull request #5509: [HUDI-4041] compact with precombineKey in RealtimeCompactedRecordRead…

2022-05-05 Thread GitBox
hudi-bot commented on PR #5509: URL: https://github.com/apache/hudi/pull/5509#issuecomment-1118557081 ## CI report: * 99e207f779a258ff32c57c9d1b962d772213c081 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118556987 ## CI report: * 688503b426fd812bde2e053795048d6509ccf7ec Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5505: [HUDI-3687] Enable spark32 in GH actions

2022-05-05 Thread GitBox
hudi-bot commented on PR #5505: URL: https://github.com/apache/hudi/pull/5505#issuecomment-1118551715 ## CI report: * 688503b426fd812bde2e053795048d6509ccf7ec Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5506: [HUDI-4042] Support truncate-partition for Spark-3.2

2022-05-05 Thread GitBox
hudi-bot commented on PR #5506: URL: https://github.com/apache/hudi/pull/5506#issuecomment-1118551765 ## CI report: * 00114df1c0184fc769bcbf2595014935aeff3281 Azure:

[jira] [Commented] (HUDI-4045) DynamoDB billing_mode property is incorrectly documented

2022-05-05 Thread Rajesh (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532244#comment-17532244 ] Rajesh commented on HUDI-4045: -- Thank you for providing the detail, Atharva. Looks like the billingmode

[jira] [Updated] (HUDI-4041) Support compact according to precombinekey in the RealtimeCompactedRecordReader class

2022-05-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4041: - Labels: pull-request-available (was: ) > Support compact according to precombinekey in the >

  1   2   3   >