[GitHub] [hudi] xushiyan commented on pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-14 Thread GitBox
xushiyan commented on PR #5201: URL: https://github.com/apache/hudi/pull/5201#issuecomment-1098785166 @danny0405 BQ integration is limited to COW partitioned table for this version we'll highlight that in the release notes. We're not aiming to tackle all cases but iteratively improve; the t

[GitHub] [hudi] xushiyan merged pull request #5272: [HUDI-3826] Make truncate partition use delete_partition operation

2022-04-14 Thread GitBox
xushiyan merged PR #5272: URL: https://github.com/apache/hudi/pull/5272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated (a081c2b9b5 -> 44b3630b5d)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from a081c2b9b5 [HUDI-3876] Fixing fetching partitions in GlueSyncClient (#5318) add 44b3630b5d [HUDI-3826] Make trun

[jira] [Updated] (HUDI-3845) Fix delete mor table's partition with urlencode's error

2022-04-14 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3845: - Fix Version/s: 0.11.0 (was: 0.12.0) > Fix delete mor table's partition with urlenco

[GitHub] [hudi] xushiyan merged pull request #5282: [HUDI-3845] Fix delete mor table's partition with urlencode's error

2022-04-14 Thread GitBox
xushiyan merged PR #5282: URL: https://github.com/apache/hudi/pull/5282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated: [HUDI-3845] Fix delete mor table's partition with urlencode's error (#5282)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 6621f3cdbb [HUDI-3845] Fix delete mor table's pa

[hudi] branch release-0.11.0 updated (209d541648 -> 8c186eba32)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch release-0.11.0 in repository https://gitbox.apache.org/repos/asf/hudi.git from 209d541648 Create release branch for version 0.11.0. new 0048b78d18 [HUDI-3800] Fixed preserve commit met

[hudi] branch HUDI-3406-revert created (now de4d5aa429)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch HUDI-3406-revert in repository https://gitbox.apache.org/repos/asf/hudi.git at de4d5aa429 Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instead of Com… (#4957)" This br

[hudi] 01/01: Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instead of Com… (#4957)"

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch HUDI-3406-revert in repository https://gitbox.apache.org/repos/asf/hudi.git commit de4d5aa429f0aaaf1bb86123238fd40c69e49d85 Author: Raymond Xu AuthorDate: Thu Apr 14 17:03:46 2022 +0800 Re

[GitHub] [hudi] xushiyan opened a new pull request, #5321: Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instea…

2022-04-14 Thread GitBox
xushiyan opened a new pull request, #5321: URL: https://github.com/apache/hudi/pull/5321 …d of Com… (#4957)" This reverts commit 98b4e9796e1e3e1f69954afa698ace5b28bde4a0. ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apach

[jira] [Updated] (HUDI-3406) Rollback incorrectly relying on FS listing instead of Commit Metadata

2022-04-14 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-3406: -- Fix Version/s: 0.12.0 (was: 0.11.0) > Rollback incorrectly relying on FS listing

[GitHub] [hudi] hudi-bot commented on pull request #5321: Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instea…

2022-04-14 Thread GitBox
hudi-bot commented on PR #5321: URL: https://github.com/apache/hudi/pull/5321#issuecomment-1098906476 ## CI report: * de4d5aa429f0aaaf1bb86123238fd40c69e49d85 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI

2022-04-14 Thread GitBox
hudi-bot commented on PR #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1098910301 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 87a5cf42e33fa9ba33475da02bbcd88db1386c20 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI

2022-04-14 Thread GitBox
hudi-bot commented on PR #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1098914426 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 87a5cf42e33fa9ba33475da02bbcd88db1386c20 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] prashantwason commented on pull request #2673: [HUDI-1688] Uncache Rdd once write operation is complete

2022-04-14 Thread GitBox
prashantwason commented on PR #2673: URL: https://github.com/apache/hudi/pull/2673#issuecomment-1098916259 @vinothchandar @xiarixiaoyao This fix assumes that there is a single SparkHoodieWriteClient on the JVM. We have a usecase where a single process handles ingestion for multiple HUDI da

[GitHub] [hudi] prashantwason commented on a diff in pull request #3207: [HUDI-2117] Unpersist the input rdd after the commit is completed to …

2022-04-14 Thread GitBox
prashantwason commented on code in PR #3207: URL: https://github.com/apache/hudi/pull/3207#discussion_r850259966 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java: ## @@ -482,4 +483,10 @@ protected void initWrapperFSMetrics() {

[GitHub] [hudi] xushiyan merged pull request #5321: Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instea…

2022-04-14 Thread GitBox
xushiyan merged PR #5321: URL: https://github.com/apache/hudi/pull/5321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch release-0.11.0 updated: Revert "[HUDI-3406] Rollback incorrectly relying on FS listing instead of Com… (#4957)" (#5321)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch release-0.11.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.11.0 by this push: new d6ba8d8bb7 Revert "[HUDI-3406] R

[GitHub] [hudi] xushiyan merged pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-14 Thread GitBox
xushiyan merged PR #5060: URL: https://github.com/apache/hudi/pull/5060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated (6621f3cdbb -> f0ab4a6e9e)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 6621f3cdbb [HUDI-3845] Fix delete mor table's partition with urlencode's error (#5282) add f0ab4a6e9e [HUDI-3652

[hudi] branch cherrypick-f0ab4a6e9ef433ac943d2409051418f7f80a6902 created (now 4a5d6a39b2)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch cherrypick-f0ab4a6e9ef433ac943d2409051418f7f80a6902 in repository https://gitbox.apache.org/repos/asf/hudi.git at 4a5d6a39b2 [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce

[hudi] 01/01: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint (#5060)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch cherrypick-f0ab4a6e9ef433ac943d2409051418f7f80a6902 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 4a5d6a39b2a28a7b72056a4280202a0d97e1445f Author: sekaiga AuthorDate: Thu A

[hudi] branch HUDI-3652-cherrypick-to-0.11.0 created (now 4a5d6a39b2)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch HUDI-3652-cherrypick-to-0.11.0 in repository https://gitbox.apache.org/repos/asf/hudi.git at 4a5d6a39b2 [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint (#506

[GitHub] [hudi] hudi-bot commented on pull request #5078: [HUDI-3667] Run unit tests of hudi-integ-tests in CI

2022-04-14 Thread GitBox
hudi-bot commented on PR #5078: URL: https://github.com/apache/hudi/pull/5078#issuecomment-1099062528 ## CI report: * 9fac106587c2652d77d4753875e8759952781a55 UNKNOWN * 5078d29eb429d7eca46c3d5c3aa72d94e088d43e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Created] (HUDI-3881) Implement index syntax for spark sql

2022-04-14 Thread Forward Xu (Jira)
Forward Xu created HUDI-3881: Summary: Implement index syntax for spark sql Key: HUDI-3881 URL: https://issues.apache.org/jira/browse/HUDI-3881 Project: Apache Hudi Issue Type: New Feature

[jira] [Updated] (HUDI-3881) Implement index syntax for spark sql

2022-04-14 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu updated HUDI-3881: - Epic Link: HUDI-1658 > Implement index syntax for spark sql > > >

[jira] [Updated] (HUDI-3881) Implement index syntax for spark sql

2022-04-14 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu updated HUDI-3881: - Description: {code:java} CREATE INDEX [IF NOT EXISTS] index_name ON TABLE [db_name.]table_name (column_nam

[GitHub] [hudi] sekaiga commented on a diff in pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-14 Thread GitBox
sekaiga commented on code in PR #5052: URL: https://github.com/apache/hudi/pull/5052#discussion_r850357531 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordReader.java: ## @@ -346,6 +349,19 @@ public synchronized void scan(Option> keys) {

[hudi] branch release-0.11.0 updated: Bumping release candidate number 2

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch release-0.11.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.11.0 by this push: new bafe564daf Bumping release candi

[hudi] annotated tag release-0.11.0-rc2 updated (bafe564daf -> fc980e40eb)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to annotated tag release-0.11.0-rc2 in repository https://gitbox.apache.org/repos/asf/hudi.git *** WARNING: tag release-0.11.0-rc2 was modified! *** from bafe564daf (commit) to fc980e40eb (tag

[GitHub] [hudi] vingov commented on pull request #5201: [HUDI-3748] write and select hudi table when enable hoodie.datasource.write.drop.partition.columns

2022-04-14 Thread GitBox
vingov commented on PR #5201: URL: https://github.com/apache/hudi/pull/5201#issuecomment-1099237144 > > > Shouldn’t the BigQuery inputformat adapter the dataset with partition columns? And why bring in the complexities to the writer/reader of spark based on little gains. > > > > Hey

[GitHub] [hudi] xushiyan commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-14 Thread GitBox
xushiyan commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1099265580 hey @sekaiga this patch seems to break in CI environment. https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=8070&view=results would you be able to look into i

[jira] [Updated] (HUDI-3873) 0.11 release blog

2022-04-14 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3873: - Status: In Progress (was: Open) > 0.11 release blog > - > > Key: HUDI-387

[jira] [Closed] (HUDI-3826) Commands deleting partitions do so incorrectly

2022-04-14 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3826. Resolution: Fixed > Commands deleting partitions do so incorrectly > ---

[jira] [Updated] (HUDI-3826) Make truncate partition use delete_partition operation

2022-04-14 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3826: - Summary: Make truncate partition use delete_partition operation (was: Commands deleting partitions do so

[jira] [Updated] (HUDI-3724) Too many open files w/ COW spark long running tests

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3724: -- Status: Patch Available (was: In Progress) > Too many open files w/ COW spark long runn

[jira] [Closed] (HUDI-3724) Too many open files w/ COW spark long running tests

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3724. - Resolution: Fixed not reproducible anymore. will revisit if we run into issues. > Too man

[GitHub] [hudi] alvarolemos commented on a diff in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…

2022-04-14 Thread GitBox
alvarolemos commented on code in PR #4724: URL: https://github.com/apache/hudi/pull/4724#discussion_r850644656 ## hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Found

[jira] [Updated] (HUDI-3806) Improve HoodieBloomIndex using bloom_filter and col_stats in MDT

2022-04-14 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3806: - Fix Version/s: 0.12.0 (was: 0.11.0) > Improve HoodieBloomIndex using bloom_filter a

svn commit: r53866 - in /dev/hudi/hudi-0.11.0-rc2: ./ hudi-0.11.0-rc2.src.tgz hudi-0.11.0-rc2.src.tgz.asc hudi-0.11.0-rc2.src.tgz.sha512

2022-04-14 Thread xushiyan
Author: xushiyan Date: Thu Apr 14 18:19:15 2022 New Revision: 53866 Log: Add hudi-0.11.0-rc2 Added: dev/hudi/hudi-0.11.0-rc2/ dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz (with props) dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz.asc dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-r

svn commit: r53868 - in /dev/hudi/hudi-0.11.0-rc2: hudi-0.11.0-rc2.src.tgz hudi-0.11.0-rc2.src.tgz.asc hudi-0.11.0-rc2.src.tgz.sha512

2022-04-14 Thread xushiyan
Author: xushiyan Date: Thu Apr 14 18:54:23 2022 New Revision: 53868 Log: Update hudi-0.11.0-rc2 Modified: dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz.asc dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz.sha512 Modified: dev/hudi/

[hudi] 01/01: [HOTFIX] add missing license

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch HOTFIX-license in repository https://gitbox.apache.org/repos/asf/hudi.git commit 65efa453c3514a2dd9b9770ae098d4f440aee8f0 Author: Raymond Xu AuthorDate: Fri Apr 15 03:03:13 2022 +0800 [HOT

[hudi] branch HOTFIX-license created (now 65efa453c3)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch HOTFIX-license in repository https://gitbox.apache.org/repos/asf/hudi.git at 65efa453c3 [HOTFIX] add missing license This branch includes the following new commits: new 65efa453c3

[GitHub] [hudi] xushiyan merged pull request #5322: [HOTFIX] add missing license

2022-04-14 Thread GitBox
xushiyan merged PR #5322: URL: https://github.com/apache/hudi/pull/5322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch release-0.11.0 updated: [HOTFIX] add missing license (#5322)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch release-0.11.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.11.0 by this push: new fbc6595b34 [HOTFIX] add missing

svn commit: r53870 - in /dev/hudi/hudi-0.11.0-rc2: hudi-0.11.0-rc2.src.tgz hudi-0.11.0-rc2.src.tgz.asc hudi-0.11.0-rc2.src.tgz.sha512

2022-04-14 Thread xushiyan
Author: xushiyan Date: Thu Apr 14 19:07:26 2022 New Revision: 53870 Log: Update hudi-0.11.0-rc2 Modified: dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz.asc dev/hudi/hudi-0.11.0-rc2/hudi-0.11.0-rc2.src.tgz.sha512 Modified: dev/hudi/

[hudi] annotated tag release-0.11.0-rc2 updated (fbc6595b34 -> e706608b0b)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to annotated tag release-0.11.0-rc2 in repository https://gitbox.apache.org/repos/asf/hudi.git *** WARNING: tag release-0.11.0-rc2 was modified! *** from fbc6595b34 (commit) to e706608b0b (tag

[hudi] branch revert-5060-feature/threadlocal_ObjectSizeCalculator2 created (now a45e51938a)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch revert-5060-feature/threadlocal_ObjectSizeCalculator2 in repository https://gitbox.apache.org/repos/asf/hudi.git at a45e51938a Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal

[hudi] 01/01: Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint (#5060)"

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch revert-5060-feature/threadlocal_ObjectSizeCalculator2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit a45e51938aa0eec643dbae4f27bfe33eaeaf4155 Author: Raymond Xu <2701446+xush

[GitHub] [hudi] xushiyan opened a new pull request, #5323: Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint"

2022-04-14 Thread GitBox
xushiyan opened a new pull request, #5323: URL: https://github.com/apache/hudi/pull/5323 Reverts apache/hudi#5060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[GitHub] [hudi] xushiyan merged pull request #5323: Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint"

2022-04-14 Thread GitBox
xushiyan merged PR #5323: URL: https://github.com/apache/hudi/pull/5323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated: Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint (#5060)" (#5323)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d6a64f765e Revert "[HUDI-3652] Make ObjectSizeCa

[GitHub] [hudi] xushiyan merged pull request #5324: [HOTFIX] add missing license (#5322)

2022-04-14 Thread GitBox
xushiyan merged PR #5324: URL: https://github.com/apache/hudi/pull/5324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[hudi] branch master updated (d6a64f765e -> 9e8664f4d2)

2022-04-14 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from d6a64f765e Revert "[HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint (#5060)" (#5323)

[jira] [Created] (HUDI-3882) Make sure Hudi Spark relations implementations provide similar file-scanning metrics

2022-04-14 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created HUDI-3882: - Summary: Make sure Hudi Spark relations implementations provide similar file-scanning metrics Key: HUDI-3882 URL: https://issues.apache.org/jira/browse/HUDI-3882 Pr

[jira] [Created] (HUDI-3883) File-sizing issues when writing COW table to S3

2022-04-14 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created HUDI-3883: - Summary: File-sizing issues when writing COW table to S3 Key: HUDI-3883 URL: https://issues.apache.org/jira/browse/HUDI-3883 Project: Apache Hudi Issue Typ

[GitHub] [hudi] hudi-bot commented on pull request #5325: [MINOR][NOMERGE] turn drop partition col on

2022-04-14 Thread GitBox
hudi-bot commented on PR #5325: URL: https://github.com/apache/hudi/pull/5325#issuecomment-1099613216 ## CI report: * efe99c7b646e5d5e116426ec8987c7af5b0a794a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5325: [MINOR][NOMERGE] turn drop partition col on

2022-04-14 Thread GitBox
hudi-bot commented on PR #5325: URL: https://github.com/apache/hudi/pull/5325#issuecomment-1099615532 ## CI report: * efe99c7b646e5d5e116426ec8987c7af5b0a794a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8076

[GitHub] [hudi] hudi-bot commented on pull request #5325: [MINOR][NOMERGE] turn drop partition col on

2022-04-14 Thread GitBox
hudi-bot commented on PR #5325: URL: https://github.com/apache/hudi/pull/5325#issuecomment-1099678498 ## CI report: * efe99c7b646e5d5e116426ec8987c7af5b0a794a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8076

[jira] [Updated] (HUDI-3883) File-sizing issues when writing COW table to S3

2022-04-14 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3883: -- Description: Even after HUDI-3709, i still see that when writing partitioned-table file-sizing

[GitHub] [hudi] zxding opened a new issue, #5326: [SUPPORT] prometheus metrics labels

2022-04-14 Thread GitBox
zxding opened a new issue, #5326: URL: https://github.com/apache/hudi/issues/5326 'hoodie.metrics.reporter.metricsname.prefix' = 'hudi_metrics', Then I got a metrics like `hudi_metrics_commit_totalBytesWritten{exported_job="my-hudi-metrics-2", instance="pushgateway", job="pushgateway"}`

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-04-14 Thread GitBox
XuQianJin-Stars commented on code in PR #5320: URL: https://github.com/apache/hudi/pull/5320#discussion_r850952202 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala: ## @@ -46,6 +46,17 @@ class AlterHo

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-04-14 Thread GitBox
XuQianJin-Stars commented on code in PR #5320: URL: https://github.com/apache/hudi/pull/5320#discussion_r850953428 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala: ## @@ -46,6 +46,17 @@ class AlterHo

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-04-14 Thread GitBox
XuQianJin-Stars commented on code in PR #5320: URL: https://github.com/apache/hudi/pull/5320#discussion_r850953428 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala: ## @@ -46,6 +46,17 @@ class AlterHo

[jira] [Created] (HUDI-3884) Inspect why archival stops at first savepoint. Add support if possible

2022-04-14 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-3884: - Summary: Inspect why archival stops at first savepoint. Add support if possible Key: HUDI-3884 URL: https://issues.apache.org/jira/browse/HUDI-3884 Project:

[jira] [Updated] (HUDI-3884) Inspect why archival stops at first savepoint. Add support if possible

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3884: -- Priority: Blocker (was: Major) > Inspect why archival stops at first savepoint. Add sup

[jira] [Assigned] (HUDI-3884) Inspect why archival stops at first savepoint. Add support if possible

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3884: - Assignee: sivabalan narayanan > Inspect why archival stops at first savepoint. Ad

[jira] [Updated] (HUDI-3884) Inspect why archival stops at first savepoint. Add support if possible

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3884: -- Fix Version/s: 0.12.0 > Inspect why archival stops at first savepoint. Add support if po

[GitHub] [hudi] KnightChess commented on a diff in pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-04-14 Thread GitBox
KnightChess commented on code in PR #5320: URL: https://github.com/apache/hudi/pull/5320#discussion_r850971092 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala: ## @@ -46,6 +46,17 @@ class AlterHoodie

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-04-14 Thread GitBox
XuQianJin-Stars commented on code in PR #5320: URL: https://github.com/apache/hudi/pull/5320#discussion_r850974829 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala: ## @@ -46,6 +46,17 @@ class AlterHo

[jira] [Resolved] (HUDI-3845) Fix delete mor table's partition with urlencode's error

2022-04-14 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu resolved HUDI-3845. -- > Fix delete mor table's partition with urlencode's error >

[jira] [Commented] (HUDI-2606) Ensure query engines not access MDT if disabled

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522611#comment-17522611 ] sivabalan narayanan commented on HUDI-2606: --- verified that not setting explicitl

[GitHub] [hudi] stayrascal commented on a diff in pull request #4724: [HUDI-2815] add partial overwrite payload to support partial overwrit…

2022-04-14 Thread GitBox
stayrascal commented on code in PR #4724: URL: https://github.com/apache/hudi/pull/4724#discussion_r851000714 ## hudi-common/src/main/java/org/apache/hudi/common/model/PartialOverwriteWithLatestAvroPayload.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Founda

[jira] [Created] (HUDI-3885) Fix issues when enabling drop.partition.columns

2022-04-14 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-3885: --- Summary: Fix issues when enabling drop.partition.columns Key: HUDI-3885 URL: https://issues.apache.org/jira/browse/HUDI-3885 Project: Apache Hudi Issue Type: Bug A

[jira] [Updated] (HUDI-3877) Support Java reader for hudi

2022-04-14 Thread Simon Su (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Su updated HUDI-3877: --- Status: In Progress (was: Open) > Support Java reader for hudi > > >

[GitHub] [hudi] CodeCooker17 opened a new issue, #5327: [SUPPORT]Mor table hive synchronization supports more flexible configuration

2022-04-14 Thread GitBox
CodeCooker17 opened a new issue, #5327: URL: https://github.com/apache/hudi/issues/5327 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes - Join the mailing list to engage in conversations and get faster support at dev

[jira] [Closed] (HUDI-2606) Ensure query engines not access MDT if disabled

2022-04-14 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-2606. --- Assignee: sivabalan narayanan (was: Tao Meng) Resolution: Fixed > Ensure query engines not access MDT i

[GitHub] [hudi] alexeykudinkin opened a new pull request, #5328: [WIP] Fix Bulk Insert to repartition the dataset based on Partition Path

2022-04-14 Thread GitBox
alexeykudinkin opened a new pull request, #5328: URL: https://github.com/apache/hudi/pull/5328 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[GitHub] [hudi] hudi-bot commented on pull request #5328: [WIP] Fix Bulk Insert to repartition the dataset based on Partition Path

2022-04-14 Thread GitBox
hudi-bot commented on PR #5328: URL: https://github.com/apache/hudi/pull/5328#issuecomment-1099810246 ## CI report: * 96b33942edf6a1d6d89361d2e056ed1c3a8d326b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5328: [WIP] Fix Bulk Insert to repartition the dataset based on Partition Path

2022-04-14 Thread GitBox
hudi-bot commented on PR #5328: URL: https://github.com/apache/hudi/pull/5328#issuecomment-1099811861 ## CI report: * 96b33942edf6a1d6d89361d2e056ed1c3a8d326b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8077

[GitHub] [hudi] hj2016 commented on pull request #4015: [HUDI-2780] Fix the issue of Mor log skipping complete blocks when reading data

2022-04-14 Thread GitBox
hj2016 commented on PR #4015: URL: https://github.com/apache/hudi/pull/4015#issuecomment-1099815631 @nsivabalan ![image](https://user-images.githubusercontent.com/18521084/163513206-d457fb5c-dedf-4180-90d9-1ed5da85a43d.png) The hudi log file consists of blocks. A log may contain s

[GitHub] [hudi] XuQianJin-Stars commented on issue #5327: [SUPPORT]Mor table hive synchronization supports more flexible configuration

2022-04-14 Thread GitBox
XuQianJin-Stars commented on issue #5327: URL: https://github.com/apache/hudi/issues/5327#issuecomment-1099818042 See if this PR can be satisfied? https://github.com/apache/hudi/commit/3449e86989f86121a9b9a93de602bc8497021a27 -- This is an automated message from the Apache Git Service. T

[jira] [Commented] (HUDI-3255) Add HoodieFlinkSink for flink datastream api

2022-04-14 Thread Simon Su (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522640#comment-17522640 ] Simon Su commented on HUDI-3255: IMO, we don't need to add a new class, this can be implem

[GitHub] [hudi] hudi-bot commented on pull request #5328: [WIP] Fix Bulk Insert to repartition the dataset based on Partition Path

2022-04-14 Thread GitBox
hudi-bot commented on PR #5328: URL: https://github.com/apache/hudi/pull/5328#issuecomment-1099842852 ## CI report: * 96b33942edf6a1d6d89361d2e056ed1c3a8d326b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8077

[jira] [Commented] (HUDI-3255) Add HoodieFlinkSink for flink datastream api

2022-04-14 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522661#comment-17522661 ] Forward Xu commented on HUDI-3255: -- Yes, in pipline implemented, I just wanted to make a

[jira] [Comment Edited] (HUDI-3255) Add HoodieFlinkSink for flink datastream api

2022-04-14 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522661#comment-17522661 ] Forward Xu edited comment on HUDI-3255 at 4/15/22 5:31 AM: --- Yes,

[jira] [Created] (HUDI-3886) Fix col stats filename to have default null value

2022-04-14 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-3886: - Summary: Fix col stats filename to have default null value Key: HUDI-3886 URL: https://issues.apache.org/jira/browse/HUDI-3886 Project: Apache Hudi

[GitHub] [hudi] nsivabalan opened a new pull request, #5329: [HUDI-3886] Adding default null for some of the fields in col stats in MDT schema

2022-04-14 Thread GitBox
nsivabalan opened a new pull request, #5329: URL: https://github.com/apache/hudi/pull/5329 ## What is the purpose of the pull request Adding default null for some of the fields in col stats in MDT schema ## Brief change log *(for example:)* - *Modify AnnotationLocati

[jira] [Updated] (HUDI-3886) Fix col stats filename to have default null value

2022-04-14 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3886: - Labels: pull-request-available (was: ) > Fix col stats filename to have default null value >

[jira] [Updated] (HUDI-3886) Fix col stats filename to have default null value

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3886: -- Fix Version/s: 0.12.0 > Fix col stats filename to have default null value >

[jira] [Assigned] (HUDI-3886) Fix col stats filename to have default null value

2022-04-14 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3886: - Assignee: sivabalan narayanan > Fix col stats filename to have default null value

[GitHub] [hudi] hudi-bot commented on pull request #5329: [HUDI-3886] Adding default null for some of the fields in col stats in MDT schema

2022-04-14 Thread GitBox
hudi-bot commented on PR #5329: URL: https://github.com/apache/hudi/pull/5329#issuecomment-1099875138 ## CI report: * f5bde1e0961619a6ba26d6d8221a68ec4e5d0395 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5329: [HUDI-3886] Adding default null for some of the fields in col stats in MDT schema

2022-04-14 Thread GitBox
hudi-bot commented on PR #5329: URL: https://github.com/apache/hudi/pull/5329#issuecomment-1099876227 ## CI report: * f5bde1e0961619a6ba26d6d8221a68ec4e5d0395 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8078