[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849141245 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -164,6 +164,18 @@ public static GenericRecord jsonBytesToAvro(byte[] bytes, Schema schema)

[GitHub] [hudi] yihua commented on a diff in pull request #4243: [HUDI-2952] Fixing metadata table for non-partitioned dataset

2022-04-12 Thread GitBox
yihua commented on code in PR #4243: URL: https://github.com/apache/hudi/pull/4243#discussion_r849140057 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -282,20 +284,22 @@ private static void processRollbackMetadata(HoodieActiveTimeline

[GitHub] [hudi] yihua commented on pull request #4243: [HUDI-2952] Fixing metadata table for non-partitioned dataset

2022-04-12 Thread GitBox
yihua commented on PR #4243: URL: https://github.com/apache/hudi/pull/4243#issuecomment-1097626338 Comments addressed in #5312 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [hudi] yihua commented on a diff in pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
yihua commented on code in PR #5308: URL: https://github.com/apache/hudi/pull/5308#discussion_r849137873 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -1453,43 +1453,43 @@ public String getHbaseTableName() { } public

[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849137217 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala: ## @@ -286,8 +319,16 @@ abstract class HoodieBaseRelation(val sqlConte

[GitHub] [hudi] hudi-bot commented on pull request #5312: [MINOR] Code cleanup in test utils

2022-04-12 Thread GitBox
hudi-bot commented on PR #5312: URL: https://github.com/apache/hudi/pull/5312#issuecomment-1097623211 ## CI report: * 2a079343e61c00543c63640bd812949da96527b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8041

[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849136014 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala: ## @@ -209,14 +219,37 @@ abstract class HoodieBaseRelation(val sqlCont

[GitHub] [hudi] hudi-bot commented on pull request #5312: [MINOR] Code cleanup in test utils

2022-04-12 Thread GitBox
hudi-bot commented on PR #5312: URL: https://github.com/apache/hudi/pull/5312#issuecomment-1097620964 ## CI report: * 2a079343e61c00543c63640bd812949da96527b8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849133380 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala: ## @@ -209,14 +219,37 @@ abstract class HoodieBaseRelation(val sqlCont

[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849132592 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -159,23 +167,67 @@ public Schema getTableAvroSchema() throws Exception { * @

[GitHub] [hudi] yihua opened a new pull request, #5312: [MINOR] Code cleanup in test utils

2022-04-12 Thread GitBox
yihua opened a new pull request, #5312: URL: https://github.com/apache/hudi/pull/5312 ## What is the purpose of the pull request As above. ## Verify this pull request This pull request is a trivial rework / code cleanup without any test coverage. ## Committer chec

[GitHub] [hudi] YannByron commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
YannByron commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849128685 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -159,23 +167,67 @@ public Schema getTableAvroSchema() throws Exception { * @

[GitHub] [hudi] danny0405 commented on pull request #5310: [minor] Rework 3454, inline the partition path logic into the builder

2022-04-12 Thread GitBox
danny0405 commented on PR #5310: URL: https://github.com/apache/hudi/pull/5310#issuecomment-1097613242 cc @codejoyan Can you help review this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Resolved] (HUDI-3868) Disable the sort input for flink streaming append mode

2022-04-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-3868. -- > Disable the sort input for flink streaming append mode > -

[jira] [Commented] (HUDI-3868) Disable the sort input for flink streaming append mode

2022-04-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521484#comment-17521484 ] Danny Chen commented on HUDI-3868: -- Fixed via master branch: 43de2b470288aa3914ba712b7d2a

[GitHub] [hudi] danny0405 merged pull request #5309: [HUDI-3868] Disable the sort input for flink streaming append mode

2022-04-12 Thread GitBox
danny0405 merged PR #5309: URL: https://github.com/apache/hudi/pull/5309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] xushiyan commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
xushiyan commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849116151 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -159,23 +167,67 @@ public Schema getTableAvroSchema() throws Exception { * @t

[GitHub] [hudi] hudi-bot commented on pull request #5311: [HUDI-3869] Improve error handling of loading Hudi conf

2022-04-12 Thread GitBox
hudi-bot commented on PR #5311: URL: https://github.com/apache/hudi/pull/5311#issuecomment-1097596582 ## CI report: * dca808f6065b14146ca087526e9b545ecf84eb53 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8039

[GitHub] [hudi] hudi-bot commented on pull request #5309: [HUDI-3868] Disable the sort input for flink streaming append mode

2022-04-12 Thread GitBox
hudi-bot commented on PR #5309: URL: https://github.com/apache/hudi/pull/5309#issuecomment-1097596563 ## CI report: * 10c1260e5cda2ca9e0fb1093cf7310272a453996 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8035

[GitHub] [hudi] xushiyan commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
xushiyan commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849109618 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -196,6 +196,11 @@ public class HoodieTableConfig extends HoodieConfig { .wit

[GitHub] [hudi] hudi-bot commented on pull request #5311: [HUDI-3869] Improve error handling of loading Hudi conf

2022-04-12 Thread GitBox
hudi-bot commented on PR #5311: URL: https://github.com/apache/hudi/pull/5311#issuecomment-1097594711 ## CI report: * dca808f6065b14146ca087526e9b545ecf84eb53 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] yihua commented on a diff in pull request #4167: [HUDI-2880] Fixing loading of props from default dir

2022-04-12 Thread GitBox
yihua commented on code in PR #4167: URL: https://github.com/apache/hudi/pull/4167#discussion_r849107169 ## hudi-common/src/main/java/org/apache/hudi/common/config/DFSPropertiesConfiguration.java: ## @@ -97,11 +99,7 @@ public static TypedProperties loadGlobalProps() { if (d

[jira] [Updated] (HUDI-3869) Address error handling in DFSPropertiesConfiguration

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3869: - Labels: pull-request-available (was: ) > Address error handling in DFSPropertiesConfiguration > -

[GitHub] [hudi] yihua opened a new pull request, #5311: [HUDI-3869] Improve error handling of loading Hudi conf

2022-04-12 Thread GitBox
yihua opened a new pull request, #5311: URL: https://github.com/apache/hudi/pull/5311 ## What is the purpose of the pull request This PR improves error handling of loading Hudi conf. ## Brief change log - Catches exception if there is error loading default conf and uses

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097590666 ## CI report: * 0794eb5a8fc46eeeb3fb321926c9d57432e3ecf7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8034

[hudi] branch master updated: [HUDI-3867] Disable Data Skipping by default (#5306)

2022-04-12 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 434e782b7d [HUDI-3867] Disable Data Skipping by de

[GitHub] [hudi] codope merged pull request #5306: [HUDI-3867] Disable Data Skipping by default

2022-04-12 Thread GitBox
codope merged PR #5306: URL: https://github.com/apache/hudi/pull/5306 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.or

[GitHub] [hudi] easonwood commented on issue #5290: [SUPPORT] Problems in handling column deletions in Hudi

2022-04-12 Thread GitBox
easonwood commented on issue #5290: URL: https://github.com/apache/hudi/issues/5290#issuecomment-1097581271 Found such empty files: 20220411034143.rollback.inflight After deleting this, Job still runs and Data are loaded to Hudi successfully. -- This is an automated message

[GitHub] [hudi] hudi-bot commented on pull request #5310: [minor] Rework 3454, inline the partition path logic into the builder

2022-04-12 Thread GitBox
hudi-bot commented on PR #5310: URL: https://github.com/apache/hudi/pull/5310#issuecomment-1097579337 ## CI report: * 65b45d5f08e67df9217bf77d68e0a9ccbbc3da71 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8037

[GitHub] [hudi] hudi-bot commented on pull request #5310: [minor] Rework 3454, inline the partition path logic into the builder

2022-04-12 Thread GitBox
hudi-bot commented on PR #5310: URL: https://github.com/apache/hudi/pull/5310#issuecomment-109752 ## CI report: * 65b45d5f08e67df9217bf77d68e0a9ccbbc3da71 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Created] (HUDI-3869) Address error handling in DFSPropertiesConfiguration

2022-04-12 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-3869: --- Summary: Address error handling in DFSPropertiesConfiguration Key: HUDI-3869 URL: https://issues.apache.org/jira/browse/HUDI-3869 Project: Apache Hudi Issue Type: Impr

[GitHub] [hudi] danny0405 opened a new pull request, #5310: [minor] Rework 3454, inline the partition path logic into the builder

2022-04-12 Thread GitBox
danny0405 opened a new pull request, #5310: URL: https://github.com/apache/hudi/pull/5310 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpo

[GitHub] [hudi] easonwood commented on issue #5290: [SUPPORT] Problems in handling column deletions in Hudi

2022-04-12 Thread GitBox
easonwood commented on issue #5290: URL: https://github.com/apache/hudi/issues/5290#issuecomment-1097571587 Sometimes, the error continues like this and blocks the job: 22/04/13 03:45:29 ERROR HoodieTimelineArchiveLog: Failed to archive commits, .commit file: 20220411034143.rollback.infli

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097556454 ## CI report: * 477b34930f113b4221c994f5db864222f99be02f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8033

[GitHub] [hudi] hudi-bot commented on pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
hudi-bot commented on PR #5308: URL: https://github.com/apache/hudi/pull/5308#issuecomment-1097551621 ## CI report: * 4833349933c202632b3230f3db85ed3dac3a47cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8031

[GitHub] [hudi] hudi-bot commented on pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
hudi-bot commented on PR #5308: URL: https://github.com/apache/hudi/pull/5308#issuecomment-1097550162 ## CI report: * 4833349933c202632b3230f3db85ed3dac3a47cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8031

[GitHub] [hudi] codejoyan commented on issue #5231: [SUPPORT] Inconsistent query result using GetLatestBaseFiles compared to Snapshot Query

2022-04-12 Thread GitBox
codejoyan commented on issue #5231: URL: https://github.com/apache/hudi/issues/5231#issuecomment-1097544892 Thanks @alexeykudinkin for the solution. I will do some testing and go through the PR. Will let you know if I have further questions. Thank you very much! -- This is an automated

[GitHub] [hudi] hudi-bot commented on pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
hudi-bot commented on PR #5308: URL: https://github.com/apache/hudi/pull/5308#issuecomment-1097543379 ## CI report: * 4833349933c202632b3230f3db85ed3dac3a47cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8031

[GitHub] [hudi] hudi-bot commented on pull request #5309: [HUDI-3868] Disable the sort input for flink streaming append mode

2022-04-12 Thread GitBox
hudi-bot commented on PR #5309: URL: https://github.com/apache/hudi/pull/5309#issuecomment-1097541323 ## CI report: * 10c1260e5cda2ca9e0fb1093cf7310272a453996 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8035

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097541077 ## CI report: * 3e6b234e433ef15742416319799a76504aeba118 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=803

[GitHub] [hudi] hudi-bot commented on pull request #5309: [HUDI-3868] Disable the sort input for flink streaming append mode

2022-04-12 Thread GitBox
hudi-bot commented on PR #5309: URL: https://github.com/apache/hudi/pull/5309#issuecomment-1097539644 ## CI report: * 10c1260e5cda2ca9e0fb1093cf7310272a453996 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-3868) Disable the sort input for flink streaming append mode

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3868: - Labels: pull-request-available (was: ) > Disable the sort input for flink streaming append mode >

[GitHub] [hudi] danny0405 opened a new pull request, #5309: [HUDI-3868] Disable the sort input for flink streaming append mode

2022-04-12 Thread GitBox
danny0405 opened a new pull request, #5309: URL: https://github.com/apache/hudi/pull/5309 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpo

[jira] [Created] (HUDI-3868) Disable the sort input for flink streaming append mode

2022-04-12 Thread Danny Chen (Jira)
Danny Chen created HUDI-3868: Summary: Disable the sort input for flink streaming append mode Key: HUDI-3868 URL: https://issues.apache.org/jira/browse/HUDI-3868 Project: Apache Hudi Issue Type:

[GitHub] [hudi] zhilinli123 commented on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-04-12 Thread GitBox
zhilinli123 commented on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1097523827 > That's weird, what version did you use ? hudi:0.10.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [hudi] danny0405 commented on pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on PR #5201: URL: https://github.com/apache/hudi/pull/5201#issuecomment-1097522037 I would suggest to revert this PR before we reach unified conclusion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [hudi] hudi-bot commented on pull request #5307: [HUDI-3686] Fix inline and async table service check in HoodieWriteConfig

2022-04-12 Thread GitBox
hudi-bot commented on PR #5307: URL: https://github.com/apache/hudi/pull/5307#issuecomment-1097521819 ## CI report: * 2cef586381174a66b5417ca31a7e97ed342ea9a4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8030

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849042547 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -159,23 +167,67 @@ public Schema getTableAvroSchema() throws Exception { * @

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097519401 ## CI report: * 0d6e6af5d236a017bb079e868e08dfe6c59186e0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8029

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849042314 ## hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java: ## @@ -159,23 +167,67 @@ public Schema getTableAvroSchema() throws Exception { * @

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097517836 ## CI report: * 0d6e6af5d236a017bb079e868e08dfe6c59186e0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8029

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097516332 ## CI report: * 3e6b234e433ef15742416319799a76504aeba118 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=803

[GitHub] [hudi] hudi-bot commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-12 Thread GitBox
hudi-bot commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1097514871 ## CI report: Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated messag

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097514836 ## CI report: * defd623805a61c4898b8f2656f356e6f5d5dd863 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8014

[GitHub] [hudi] sekaiga commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-12 Thread GitBox
sekaiga commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1097513727 All checks have passed , please merge , thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849037938 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -196,6 +196,11 @@ public class HoodieTableConfig extends HoodieConfig { .wi

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849037938 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -196,6 +196,11 @@ public class HoodieTableConfig extends HoodieConfig { .wi

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097513082 ## CI report: * 0d6e6af5d236a017bb079e868e08dfe6c59186e0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8029

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097512818 ## CI report: * defd623805a61c4898b8f2656f356e6f5d5dd863 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8014

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849036618 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -324,13 +336,19 @@ public static GenericRecord addOperationToRecord(GenericRecord record, H

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1097510822 ## CI report: * defd623805a61c4898b8f2656f356e6f5d5dd863 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8014

[GitHub] [hudi] danny0405 commented on a diff in pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5201: URL: https://github.com/apache/hudi/pull/5201#discussion_r849035284 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -164,6 +164,18 @@ public static GenericRecord jsonBytesToAvro(byte[] bytes, Schema schema)

[GitHub] [hudi] danny0405 commented on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-04-12 Thread GitBox
danny0405 commented on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1097504364 That's weird, what version did you use ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hudi] danny0405 commented on a diff in pull request #5304: [DOCS] Add faq for async compaction options

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5304: URL: https://github.com/apache/hudi/pull/5304#discussion_r84903 ## website/docs/faq.md: ## @@ -253,6 +253,24 @@ Simplest way to run compaction on MOR dataset is to run the [compaction inline]( That said, for obvious reasons of not

[GitHub] [hudi] danny0405 commented on a diff in pull request #5304: [DOCS] Add faq for async compaction options

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5304: URL: https://github.com/apache/hudi/pull/5304#discussion_r849028469 ## website/docs/compaction.md: ## @@ -74,7 +74,7 @@ To improve ingestion latency, Async Compaction is the default configuration. If immediate read performance of a new

[GitHub] [hudi] hudi-bot commented on pull request #5306: [HUDI-3867] Disable Data Skipping by default

2022-04-12 Thread GitBox
hudi-bot commented on PR #5306: URL: https://github.com/apache/hudi/pull/5306#issuecomment-1097494255 ## CI report: * c216326443ec14bca96e1f9e482b0642e2f888ba Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8028

[jira] [Updated] (HUDI-3867) Disable Data Skipping by default in 0.11

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3867: - Labels: pull-request-available (was: ) > Disable Data Skipping by default in 0.11 > -

[jira] [Updated] (HUDI-3862) Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread xi chaomin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HUDI-3862: - Description: For example GET_BATCH_SIZE, MAX_QPS_PER_REGION_SERVER, QPS_ALLOCATOR_CLASS_NAME, these config

[GitHub] [hudi] danny0405 commented on a diff in pull request #4443: [HUDI-3101] Excluding compaction instants from pending rollback info

2022-04-12 Thread GitBox
danny0405 commented on code in PR #4443: URL: https://github.com/apache/hudi/pull/4443#discussion_r849017626 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java: ## @@ -902,10 +902,14 @@ private HoodieTimeline getInflightTimelin

[GitHub] [hudi] danny0405 commented on a diff in pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
danny0405 commented on code in PR #5296: URL: https://github.com/apache/hudi/pull/5296#discussion_r849017448 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java: ## @@ -370,6 +360,16 @@ public void write(GenericRecord oldRecord) { }

[jira] [Updated] (HUDI-3862) Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread xi chaomin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HUDI-3862: - Component/s: index > Some default configurations in HoodieHBaseIndexConfig do not work > -

[GitHub] [hudi] hudi-bot commented on pull request #5307: [HUDI-3686] Fix inline and async table service check in HoodieWriteConfig

2022-04-12 Thread GitBox
hudi-bot commented on PR #5307: URL: https://github.com/apache/hudi/pull/5307#issuecomment-1097480437 ## CI report: * 2cef586381174a66b5417ca31a7e97ed342ea9a4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8030

[GitHub] [hudi] hudi-bot commented on pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
hudi-bot commented on PR #5308: URL: https://github.com/apache/hudi/pull/5308#issuecomment-1097480449 ## CI report: * 4833349933c202632b3230f3db85ed3dac3a47cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8031

[GitHub] [hudi] hudi-bot commented on pull request #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
hudi-bot commented on PR #5308: URL: https://github.com/apache/hudi/pull/5308#issuecomment-1097478746 ## CI report: * 4833349933c202632b3230f3db85ed3dac3a47cc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5307: [HUDI-3686] Fix inline and async table service check in HoodieWriteConfig

2022-04-12 Thread GitBox
hudi-bot commented on PR #5307: URL: https://github.com/apache/hudi/pull/5307#issuecomment-1097478707 ## CI report: * 2cef586381174a66b5417ca31a7e97ed342ea9a4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-3686) Fix areAnyTableServicesInline() in HoodieWriteConfig

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3686: - Labels: pull-request-available (was: ) > Fix areAnyTableServicesInline() in HoodieWriteConfig > -

[jira] [Updated] (HUDI-3862) Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3862: - Labels: pull-request-available (was: ) > Some default configurations in HoodieHBaseIndexConfig do

[GitHub] [hudi] xicm opened a new pull request, #5308: HUDI-3862 Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread GitBox
xicm opened a new pull request, #5308: URL: https://github.com/apache/hudi/pull/5308 …t work ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is

[GitHub] [hudi] yihua opened a new pull request, #5307: [HUDI-3686] Fix inline and async table service check in HoodieWriteConfig

2022-04-12 Thread GitBox
yihua opened a new pull request, #5307: URL: https://github.com/apache/hudi/pull/5307 ## What is the purpose of the pull request This PR fixes the check of inline and async table services in `HoodieWriteConfig`. ## Brief change log - Adjusts `areAnyTableServicesExecutedI

[jira] [Updated] (HUDI-3867) Disable Data Skipping by default in 0.11

2022-04-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3867: -- Description: Since it nor relies on MT's Column Stats Index which is off by default in 0.11  

[jira] [Created] (HUDI-3867) Disable Data Skipping by default in 0.11

2022-04-12 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created HUDI-3867: - Summary: Disable Data Skipping by default in 0.11 Key: HUDI-3867 URL: https://issues.apache.org/jira/browse/HUDI-3867 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097457010 ## CI report: * 1cb81a90f3053bb03de07c384771723ebaa80cce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7957

[GitHub] [hudi] hudi-bot commented on pull request #5306: [WIP] Disable Data Skipping by default

2022-04-12 Thread GitBox
hudi-bot commented on PR #5306: URL: https://github.com/apache/hudi/pull/5306#issuecomment-1097455427 ## CI report: * c216326443ec14bca96e1f9e482b0642e2f888ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8028

[GitHub] [hudi] hudi-bot commented on pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
hudi-bot commented on PR #5272: URL: https://github.com/apache/hudi/pull/5272#issuecomment-1097455377 ## CI report: * 1cb81a90f3053bb03de07c384771723ebaa80cce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7957

[GitHub] [hudi] hudi-bot commented on pull request #5306: [WIP] Disable Data Skipping by default

2022-04-12 Thread GitBox
hudi-bot commented on PR #5306: URL: https://github.com/apache/hudi/pull/5306#issuecomment-1097453858 ## CI report: * c216326443ec14bca96e1f9e482b0642e2f888ba UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] alexeykudinkin opened a new pull request, #5306: [WIP] Disable Data Skipping by default

2022-04-12 Thread GitBox
alexeykudinkin opened a new pull request, #5306: URL: https://github.com/apache/hudi/pull/5306 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
XuQianJin-Stars commented on code in PR #5272: URL: https://github.com/apache/hudi/pull/5272#discussion_r848985798 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala: ## @@ -372,4 +376,32 @@ object HoodieSqlCommonUtils

[jira] [Closed] (HUDI-3855) Hudi's metadata field "_hoodie_file_name" not updated in MergeHandle

2022-04-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin closed HUDI-3855. - Resolution: Fixed > Hudi's metadata field "_hoodie_file_name" not updated in MergeHandle > ---

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
XuQianJin-Stars commented on code in PR #5272: URL: https://github.com/apache/hudi/pull/5272#discussion_r848975952 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala: ## @@ -372,4 +376,32 @@ object HoodieSqlCommonUtils

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
XuQianJin-Stars commented on code in PR #5272: URL: https://github.com/apache/hudi/pull/5272#discussion_r848975952 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala: ## @@ -372,4 +376,32 @@ object HoodieSqlCommonUtils

[hudi] branch master updated: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle` (#5296)

2022-04-12 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 7b78dff45f [HUDI-3855] Fixing `FILENAME_METADAT

[GitHub] [hudi] nsivabalan merged pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
nsivabalan merged PR #5296: URL: https://github.com/apache/hudi/pull/5296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5272: [HUDI-3826] Commands deleting partitions do so incorrectly

2022-04-12 Thread GitBox
XuQianJin-Stars commented on code in PR #5272: URL: https://github.com/apache/hudi/pull/5272#discussion_r848974096 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestTruncateTable.scala: ## @@ -69,7 +69,7 @@ class TestTruncateTable extends TestHoodi

[GitHub] [hudi] hudi-bot commented on pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
hudi-bot commented on PR #5296: URL: https://github.com/apache/hudi/pull/5296#issuecomment-1097337679 ## CI report: * 9458d847182b0628d228211d010310ade743d431 UNKNOWN * 0028d5a8fc860de4f222a35f80323003b69b957b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] yihua commented on a diff in pull request #4443: [HUDI-3101] Excluding compaction instants from pending rollback info

2022-04-12 Thread GitBox
yihua commented on code in PR #4443: URL: https://github.com/apache/hudi/pull/4443#discussion_r848939875 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java: ## @@ -902,10 +902,14 @@ private HoodieTimeline getInflightTimelineExc

[jira] [Updated] (HUDI-3686) Fix areAnyTableServicesInline() in HoodieWriteConfig

2022-04-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3686: Fix Version/s: 0.11.0 > Fix areAnyTableServicesInline() in HoodieWriteConfig > -

[GitHub] [hudi] yihua commented on pull request #5048: [HUDI-3634] Could read empty or partial HoodieCommitMetaData in downstream if using HDFS

2022-04-12 Thread GitBox
yihua commented on PR #5048: URL: https://github.com/apache/hudi/pull/5048#issuecomment-1097320642 @boneanxs is this more like a transient issue for readers? There could be cases where empty or partial complete instant file is left in the timeline. There are fixes to it: #4016 #5261 . Do

[GitHub] [hudi] alexeykudinkin commented on issue #5231: [SUPPORT] Inconsistent query result using GetLatestBaseFiles compared to Snapshot Query

2022-04-12 Thread GitBox
alexeykudinkin commented on issue #5231: URL: https://github.com/apache/hudi/issues/5231#issuecomment-1097314266 Validated that #5296 addresses the issue: ``` cala> val partitions = FSUtils.getAllPartitionPaths(engineContext, metadataConfig, basePath).iterator().asScala.toList;

[jira] [Closed] (HUDI-2319) Integrate hudi with dbt (data build tool)

2022-04-12 Thread Vinoth Govindarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Govindarajan closed HUDI-2319. - Resolution: Fixed > Integrate hudi with dbt (data build tool) > --

[jira] [Resolved] (HUDI-3838) Make Drop partition column config work with deltastreamer

2022-04-12 Thread Vinoth Govindarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Govindarajan resolved HUDI-3838. --- > Make Drop partition column config work with deltastreamer >

  1   2   3   >