[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1314929406 ## CI report: * 8ca8f81c5cdf02f111de83d287c085991f46597f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] eric9204 closed pull request #7167: [HUDI-5094] Remove partition fields before transform bytes to avro,if enable DROP_PARTITION_COLUMNS.

2022-11-15 Thread GitBox
eric9204 closed pull request #7167: [HUDI-5094] Remove partition fields before transform bytes to avro,if enable DROP_PARTITION_COLUMNS. URL: https://github.com/apache/hudi/pull/7167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [hudi] trushev commented on pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1314934901 > What do you mean for the rest cases ? Isn't either schema evolution enabled or disabled ? Even though schema evolution enabled there is might not an `internalSchema`. So we use read

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1314937282 ## CI report: * 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1314937708 ## CI report: * 8ca8f81c5cdf02f111de83d287c085991f46597f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] YannByron commented on a diff in pull request #7196: [MINOR] move logic for deleting active instant to HoodieActiveTimeline

2022-11-15 Thread GitBox
YannByron commented on code in PR #7196: URL: https://github.com/apache/hudi/pull/7196#discussion_r1022461347 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java: ## @@ -257,46 +257,36 @@ public static void deleteInstantFile(FileSystem fs

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1314944564 ## CI report: * 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1314944892 ## CI report: * 8ca8f81c5cdf02f111de83d287c085991f46597f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] danny0405 commented on a diff in pull request #7196: [MINOR] move logic for deleting active instant to HoodieActiveTimeline

2022-11-15 Thread GitBox
danny0405 commented on code in PR #7196: URL: https://github.com/apache/hudi/pull/7196#discussion_r1022466692 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java: ## @@ -257,46 +257,36 @@ public static void deleteInstantFile(FileSystem fs

[GitHub] [hudi] lucasberlang commented on issue #7185: [SUPPORT] Error to write in s3 bucket with K8s

2022-11-15 Thread GitBox
lucasberlang commented on issue #7185: URL: https://github.com/apache/hudi/issues/7185#issuecomment-1314962293 Yes, I have the s3 credentials in the core-site.xml and in the flink-conf.yaml in the jobmanager and the tasmanager: ```xml fs.s3.awsAccessKeyId

[GitHub] [hudi] YannByron commented on a diff in pull request #7196: [MINOR] move logic for deleting active instant to HoodieActiveTimeline

2022-11-15 Thread GitBox
YannByron commented on code in PR #7196: URL: https://github.com/apache/hudi/pull/7196#discussion_r1022483387 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java: ## @@ -257,46 +257,36 @@ public static void deleteInstantFile(FileSystem fs

[GitHub] [hudi] danny0405 commented on a diff in pull request #7196: [MINOR] move logic for deleting active instant to HoodieActiveTimeline

2022-11-15 Thread GitBox
danny0405 commented on code in PR #7196: URL: https://github.com/apache/hudi/pull/7196#discussion_r1022491991 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -780,6 +780,22 @@ public static Map parallelizeFilesProcess( return result; } + pu

[GitHub] [hudi] eric9204 opened a new pull request, #7205: [HUDI-5094] modify avro schema for drop partition fields case

2022-11-15 Thread GitBox
eric9204 opened a new pull request, #7205: URL: https://github.com/apache/hudi/pull/7205 ### Change Logs NONE ### Impact NONE ### Risk level (write none, low medium or high below) NONE ### Documentation Update _Describe any necessary documentat

[GitHub] [hudi] danny0405 commented on a diff in pull request #7196: [MINOR] move logic for deleting active instant to HoodieActiveTimeline

2022-11-15 Thread GitBox
danny0405 commented on code in PR #7196: URL: https://github.com/apache/hudi/pull/7196#discussion_r1022493356 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java: ## @@ -257,46 +257,36 @@ public static void deleteInstantFile(FileSystem fs

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022504653 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java: ## @@ -130,4 +145,48 @@ protected Void getResult() {

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022510833 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatReader.java: ## @@ -52,9 +53,9 @@ public class HoodieLogFormatReader implements HoodieLo

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022515240 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java: ## @@ -130,4 +145,48 @@ protected Void getResult() { retur

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022515240 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java: ## @@ -130,4 +145,48 @@ protected Void getResult() { retur

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022517049 ## hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java: ## @@ -66,6 +77,11 @@ public InternalSchema(Field... columns) { this(DEFAULT_VE

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022515240 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java: ## @@ -130,4 +145,48 @@ protected Void getResult() { retur

[GitHub] [hudi] onlywangyh opened a new pull request, #7206: Hudi 5220

2022-11-15 Thread GitBox
onlywangyh opened a new pull request, #7206: URL: https://github.com/apache/hudi/pull/7206 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performan

[GitHub] [hudi] onlywangyh closed issue #7204: failed to snapshot query in hive when query a empty partition

2022-11-15 Thread GitBox
onlywangyh closed issue #7204: failed to snapshot query in hive when query a empty partition URL: https://github.com/apache/hudi/issues/7204 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022520427 ## hudi-common/src/main/java/org/apache/hudi/internal/schema/convert/AvroInternalSchemaConverter.java: ## @@ -91,6 +91,11 @@ public static InternalSchema convert(Sche

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022522069 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java: ## @@ -491,7 +492,7 @@ public Pair getLogRecordScanner(List< .withFile

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022523171 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordReader.java: ## @@ -54,15 +54,15 @@ public class HoodieMetadataMergedLogRecordReade

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022524997 ## hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java: ## @@ -66,6 +77,11 @@ public InternalSchema(Field... columns) { this(DEFAULT_VERSION

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022526462 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java: ## @@ -491,7 +492,7 @@ public Pair getLogRecordScanner(List< .withFileSyste

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022524997 ## hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java: ## @@ -66,6 +77,11 @@ public InternalSchema(Field... columns) { this(DEFAULT_VERSION

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022530081 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordReader.java: ## @@ -54,15 +54,15 @@ public class HoodieMetadataMergedLogRecordReader ext

[GitHub] [hudi] xiarixiaoyao commented on pull request #5830: [HUDI-3981][WIP][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
xiarixiaoyao commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1315020080 > > Either we have some tool for fetching the right avro schema in evolution use cases > > `avroSchema` does not support custom ids. We use them to build mergedIntern

[GitHub] [hudi] trushev commented on a diff in pull request #5830: [HUDI-3981][WIP][RFC-33] Flink engine support for comprehensive schema evolution

2022-11-15 Thread GitBox
trushev commented on code in PR #5830: URL: https://github.com/apache/hudi/pull/5830#discussion_r1022524997 ## hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java: ## @@ -66,6 +77,11 @@ public InternalSchema(Field... columns) { this(DEFAULT_VERSION

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315029694 ## CI report: * 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1272

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1315030061 ## CI report: * 8ca8f81c5cdf02f111de83d287c085991f46597f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7205: [HUDI-5094] modify avro schema for drop partition fields case

2022-11-15 Thread GitBox
hudi-bot commented on PR #7205: URL: https://github.com/apache/hudi/pull/7205#issuecomment-1315030115 ## CI report: * 615902b4794dfae394c64191a1f2a8ea1e4176b1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7206: [HUDI-5220] fix hive snapshot query add non hoodie paths file status

2022-11-15 Thread GitBox
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1315030185 ## CI report: * f2435ef886c015ef73bb728cdf104cfc65864551 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-5220) failed to snapshot query in hive when query a empty partition

2022-11-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5220: - Labels: pull-request-available (was: ) > failed to snapshot query in hive when query a empty part

[jira] [Created] (HUDI-5221) Make the decision for flink sql bucket index case-insensitive

2022-11-15 Thread Danny Chen (Jira)
Danny Chen created HUDI-5221: Summary: Make the decision for flink sql bucket index case-insensitive Key: HUDI-5221 URL: https://issues.apache.org/jira/browse/HUDI-5221 Project: Apache Hudi Issu

[GitHub] [hudi] xushiyan commented on a diff in pull request #6680: [HUDI-4812] Lazy fetching partition path & file slice for HoodieFileIndex

2022-11-15 Thread GitBox
xushiyan commented on code in PR #6680: URL: https://github.com/apache/hudi/pull/6680#discussion_r1022548772 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala: ## @@ -237,70 +246,64 @@ class SparkHoodieTableFileIndex(spark:

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315036789 ## CI report: * 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=130

[GitHub] [hudi] danny0405 opened a new pull request, #7207: [HUDI-5221] Make the decision for flink sql bucket index case-insensi…

2022-11-15 Thread GitBox
danny0405 opened a new pull request, #7207: URL: https://github.com/apache/hudi/pull/7207 …tive ### Change Logs make the bucket index decision case insensitive ### Impact No ### Risk level (write none, low medium or high below) none ### Documen

[GitHub] [hudi] xushiyan commented on a diff in pull request #6680: [HUDI-4812] Lazy fetching partition path & file slice for HoodieFileIndex

2022-11-15 Thread GitBox
xushiyan commented on code in PR #6680: URL: https://github.com/apache/hudi/pull/6680#discussion_r1022550825 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UTF8StringPartitionPathFormatter.java: ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software F

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1315037189 ## CI report: * 796ca489fa819e622bf4c0798cdc03f07da33e87 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=130

[jira] [Updated] (HUDI-5221) Make the decision for flink sql bucket index case-insensitive

2022-11-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5221: - Labels: pull-request-available (was: ) > Make the decision for flink sql bucket index case-insens

[GitHub] [hudi] hudi-bot commented on pull request #7205: [HUDI-5094] modify avro schema for drop partition fields case

2022-11-15 Thread GitBox
hudi-bot commented on PR #7205: URL: https://github.com/apache/hudi/pull/7205#issuecomment-1315037233 ## CI report: * 615902b4794dfae394c64191a1f2a8ea1e4176b1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7206: [HUDI-5220] fix hive snapshot query add non hoodie paths file status

2022-11-15 Thread GitBox
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1315037289 ## CI report: * f2435ef886c015ef73bb728cdf104cfc65864551 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7207: [HUDI-5221] Make the decision for flink sql bucket index case-insensi…

2022-11-15 Thread GitBox
hudi-bot commented on PR #7207: URL: https://github.com/apache/hudi/pull/7207#issuecomment-1315044405 ## CI report: * eea7e731fc3d7b4f9b5b9440be10ca7a6fafa27d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7207: [HUDI-5221] Make the decision for flink sql bucket index case-insensi…

2022-11-15 Thread GitBox
hudi-bot commented on PR #7207: URL: https://github.com/apache/hudi/pull/7207#issuecomment-1315051617 ## CI report: * eea7e731fc3d7b4f9b5b9440be10ca7a6fafa27d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] idrismike commented on issue #6787: [SUPPORT] hoodie.metadata.enable and DEFAULT_METADATA_ENABLE_FOR_READERS conflict

2022-11-15 Thread GitBox
idrismike commented on issue #6787: URL: https://github.com/apache/hudi/issues/6787#issuecomment-1315108738 Interesting but at the same time curious to know that from `hudi 0.11` hive metastore is not used anymore for getting partition. If I understand correctly, since metastore does not do

[jira] [Created] (HUDI-5222) Flink job may read wrong timline service address when enable offline compaction

2022-11-15 Thread Jian Feng (Jira)
Jian Feng created HUDI-5222: --- Summary: Flink job may read wrong timline service address when enable offline compaction Key: HUDI-5222 URL: https://issues.apache.org/jira/browse/HUDI-5222 Project: Apache Hud

[jira] [Updated] (HUDI-5222) Flink job may read wrong timline service address when enable offline compaction

2022-11-15 Thread Jian Feng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Feng updated HUDI-5222: Description: For now, Flink job is forced to use embed timeline service, and will create new .hoodie/.aux/v

[jira] [Updated] (HUDI-5222) Flink job may read wrong timline service address when enable offline compaction

2022-11-15 Thread Jian Feng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Feng updated HUDI-5222: Description: For now, Flink job is forced to use embed timeline service, and will create new .hoodie/.aux/v

[jira] [Updated] (HUDI-5222) Flink job may read wrong timline service address when enable offline compaction

2022-11-15 Thread Jian Feng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Feng updated HUDI-5222: Description: For now, Flink job is forced to use embed timeline service, and will create new .hoodie/.aux/v

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315126939 ## CI report: * 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=130

[GitHub] [hudi] hudi-bot commented on pull request #7003: [minor] add more test for rfc46

2022-11-15 Thread GitBox
hudi-bot commented on PR #7003: URL: https://github.com/apache/hudi/pull/7003#issuecomment-1315133065 ## CI report: * 3fdbb2a733c1f8bcb10113b167f4ce6fda273ffa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1300

[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-15 Thread GitBox
hudi-bot commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315133318 ## CI report: * ace488c3ed6ec5b91594a544932d5b4fb8ceddd5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=130

[GitHub] [hudi] hudi-bot commented on pull request #7003: [minor] add more test for rfc46

2022-11-15 Thread GitBox
hudi-bot commented on PR #7003: URL: https://github.com/apache/hudi/pull/7003#issuecomment-1315139022 ## CI report: * 3fdbb2a733c1f8bcb10113b167f4ce6fda273ffa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1300

[GitHub] [hudi] zhangyue19921010 commented on pull request #6843: [WIP][HUDI-5023] Avoiding using `BoundedInMemoryExecutor` on the hot-path

2022-11-15 Thread GitBox
zhangyue19921010 commented on PR #6843: URL: https://github.com/apache/hudi/pull/6843#issuecomment-1315140013 Hey @alexeykudinkin ! Thanks for your response! > Yeah, this PR was put up purely for experimental purposes even before we finalized previous PR landing disruptor which (considera

[jira] [Resolved] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu resolved HUDI-2913. --- > Disable auto clean in writer task > - > > Key: HUDI-2913

[jira] [Closed] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu closed HUDI-2913. - Resolution: Fixed > Disable auto clean in writer task > - > >

[jira] [Updated] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu updated HUDI-2913: -- Fix Version/s: 0.12.2 > Disable auto clean in writer task > - > >

[jira] [Reopened] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu reopened HUDI-2913: --- > Disable auto clean in writer task > - > > Key: HUDI-2913

[jira] [Closed] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu closed HUDI-2913. - Resolution: Fixed > Disable auto clean in writer task > - > >

[jira] [Resolved] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu resolved HUDI-2913. --- > Disable auto clean in writer task > - > > Key: HUDI-2913

[jira] [Reopened] (HUDI-2913) Disable auto clean in writer task

2022-11-15 Thread Zhaojing Yu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaojing Yu reopened HUDI-2913: --- > Disable auto clean in writer task > - > > Key: HUDI-2913

[jira] [Created] (HUDI-5223) Partial failover for flink

2022-11-15 Thread Danny Chen (Jira)
Danny Chen created HUDI-5223: Summary: Partial failover for flink Key: HUDI-5223 URL: https://issues.apache.org/jira/browse/HUDI-5223 Project: Apache Hudi Issue Type: Task Components: f

[GitHub] [hudi] complone commented on a diff in pull request #7156: [HUDI-5161] add TIMESTAMP_LTZ

2022-11-15 Thread GitBox
complone commented on code in PR #7156: URL: https://github.com/apache/hudi/pull/7156#discussion_r1016421358 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/TypeInfoLogicalTypeVisitor.java: ## @@ -144,6 +145,16 @@ public TypeInfo visit(TimestampTy

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1315215105 ## CI report: * 1eafc33c31ce70b57e402aa6f3ea78601a780c94 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1315224068 ## CI report: * 1eafc33c31ce70b57e402aa6f3ea78601a780c94 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7203: [HUDI-5208] Enable Spark to support OPERATION_METADATA_FIELD

2022-11-15 Thread GitBox
hudi-bot commented on PR #7203: URL: https://github.com/apache/hudi/pull/7203#issuecomment-1315230853 ## CI report: * 1eafc33c31ce70b57e402aa6f3ea78601a780c94 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] danny0405 opened a new pull request, #7208: [HUDI-5223] Partial failover for flink

2022-11-15 Thread GitBox
danny0405 opened a new pull request, #7208: URL: https://github.com/apache/hudi/pull/7208 ### Change Logs Supports partial failover for flink write pipeline. ### Impact No impact. ### Risk level (write none, low medium or high below) none ### Document

[jira] [Updated] (HUDI-5223) Partial failover for flink

2022-11-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5223: - Labels: pull-request-available (was: ) > Partial failover for flink > --

[GitHub] [hudi] koldic opened a new issue, #7209: [SUPPORT]

2022-11-15 Thread GitBox
koldic opened a new issue, #7209: URL: https://github.com/apache/hudi/issues/7209 **Describe the problem you faced** Hudi deltastreamer fails with this exception `Could not deserialize metadata of type class org.apache.hudi.avro.model.HoodieCleanMetadata` **To Reproduce**

[GitHub] [hudi] hudi-bot commented on pull request #7205: [HUDI-5094] modify avro schema for drop partition fields case

2022-11-15 Thread GitBox
hudi-bot commented on PR #7205: URL: https://github.com/apache/hudi/pull/7205#issuecomment-1315306460 ## CI report: * 615902b4794dfae394c64191a1f2a8ea1e4176b1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7206: [HUDI-5220] fix hive snapshot query add non hoodie paths file status

2022-11-15 Thread GitBox
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1315306502 ## CI report: * f2435ef886c015ef73bb728cdf104cfc65864551 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1302

[GitHub] [hudi] hudi-bot commented on pull request #7208: [HUDI-5223] Partial failover for flink

2022-11-15 Thread GitBox
hudi-bot commented on PR #7208: URL: https://github.com/apache/hudi/pull/7208#issuecomment-1315313418 ## CI report: * d5bdf67c594f2193cca59f77daa68e57a296eb09 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7208: [HUDI-5223] Partial failover for flink

2022-11-15 Thread GitBox
hudi-bot commented on PR #7208: URL: https://github.com/apache/hudi/pull/7208#issuecomment-1315320314 ## CI report: * d5bdf67c594f2193cca59f77daa68e57a296eb09 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1303

[jira] [Updated] (HUDI-5136) [Reader] Project schemas in FileSliceReader (FSR)

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5136: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > [Reader] Project schemas in FileSliceReader (FSR) > -

[jira] [Updated] (HUDI-5138) [Reader] Implement FileSliceReader (FSR) APIs

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5138: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > [Reader] Implement FileSliceReader (FSR) APIs > --

[jira] [Updated] (HUDI-4990) Parallelize deduplication in CLI tool

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4990: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Parallelize deduplication in CLI tool > --

[jira] [Updated] (HUDI-5135) Abstract out FileIndex and RecordIndex APIs

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5135: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Abstract out FileIndex and RecordIndex APIs >

[jira] [Updated] (HUDI-5141) [Reader] Integrate metadata files and column_stats partitions in FileIndex

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5141: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > [Reader] Integrate metadata files and column_stats par

[jira] [Updated] (HUDI-3601) Support multi-arch builds in docker setup

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3601: - Sprint: 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/09/05, 2022/09/

[jira] [Updated] (HUDI-4411) Bump Spark version to 3.2.2

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4411: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Bump Spark version to 3.2.2 >

[jira] [Updated] (HUDI-5134) Implement PartitionSnapshot and PartitionDescriptor

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5134: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Implement PartitionSnapshot and PartitionDescriptor >

[jira] [Updated] (HUDI-4967) Improve docs for meta sync with TimestampBasedKeyGenerator

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4967: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Improve docs for meta sync with TimestampBasedKeyGener

[jira] [Updated] (HUDI-3529) Improve dependency management and bundling

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3529: - Sprint: 2022/08/22, 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/08/

[jira] [Updated] (HUDI-5051) Add a functional regression test for Bloom Index followed on w/ Upserts

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5051: - Sprint: 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/10/18, 2022/11/01) > Add a functional regression te

[jira] [Updated] (HUDI-4886) Detect incompatible schema change during deltastreamer ingestion

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4886: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Detect incompatible schema change during deltastreamer

[jira] [Updated] (HUDI-4142) RFC for new Table APIs proposal for query engine integrations

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4142: - Sprint: 2022/05/16, 2022/05/31, 2022/11/01, 2022/11/15 (was: 2022/05/16, 2022/05/31, 2022/11/01) > RFC f

[jira] [Updated] (HUDI-5018) Make user-provided copyOnWriteRecordSizeEstimate first precedence

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5018: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Make user-provided copyOnWriteRecordSizeEstimate first

[jira] [Updated] (HUDI-83) Map Timestamp type in spark to corresponding Timestamp type in Hive during Hive sync

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-83: --- Sprint: Cont' improve - 2021/01/24, Cont' improve - 2021/01/31, 2022/09/05, 2022/10/04, 2022/10/18, 2022/11/01

[jira] [Updated] (HUDI-5137) [Reader] Push down filters in FSR

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5137: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > [Reader] Push down filters in FSR > --

[jira] [Updated] (HUDI-4631) Enhance retries for failed writes w/ write conflicts in a multi writer scenarios

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4631: - Sprint: 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/10/04, 2022/10/18, 2022/11/01) > Enhan

[jira] [Updated] (HUDI-4847) hive sync fails w/ utilities bundle in 0.13-snapshot, but succeeds w/ 0.11

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4847: - Sprint: 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/09/19, 2022/10/04, 2022/10/

[jira] [Updated] (HUDI-4986) Enhance hudi integ test readme for multi-writer tests

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4986: - Sprint: 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/10/04, 2022/10/18, 2022/11/01) > Enhan

[jira] [Updated] (HUDI-1574) Trim existing unit tests to finish in much shorter amount of time

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1574: - Sprint: 2022/08/22, 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/08/

[jira] [Updated] (HUDI-3249) Performance Improvements

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3249: - Sprint: 2022/08/22, 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/15 (was: 2022/08/

[jira] [Updated] (HUDI-4863) Deprecate `hoodie.compaction.payload.class` and re-use hoodie.datasource.write.payload.class

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4863: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Deprecate `hoodie.compaction.payload.class` and re-use

[jira] [Updated] (HUDI-5109) Source all metadata table instability issues

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5109: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Source all metadata table instability issues > ---

[jira] [Updated] (HUDI-2740) Support for snapshot querying on MOR table

2022-11-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2740: - Sprint: 2022/11/01, 2022/11/15 (was: 2022/11/01) > Support for snapshot querying on MOR table > -

  1   2   3   >