[jira] [Commented] (HUDI-3189) Fallback to full table scan with incremental query when files are cleaned up or achived for MOR table

2022-07-17 Thread Hui An (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17567832#comment-17567832 ] Hui An commented on HUDI-3189: -- Hi [~shivnarayan] Are you working on this? If not, I can impl

[GitHub] [hudi] fengjian428 commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
fengjian428 commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186816730 > > I also found this, but there is a LOCK_ACQUIRE_WAIT_TIMEOUT_MS when using zk or hive lock provider, so I feel this may not be a big problem. > > The config items: LOCK_ACQUIRE_

[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider

2022-07-17 Thread GitBox
hudi-bot commented on PR #6071: URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186815223 ## CI report: * 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #5659: [HUDI-4137] SnowflakeSyncTool MVP implementation to integrate with Snowflake

2022-07-17 Thread GitBox
hudi-bot commented on PR #5659: URL: https://github.com/apache/hudi/pull/5659#issuecomment-1186814815 ## CI report: * 5b8c6c515dfd37f7fbce8a98468b17e6998947b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] liujinhui1994 commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
liujinhui1994 commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186807647 > Cool~ I also found this, but there is a LOCK_ACQUIRE_WAIT_TIMEOUT_MS when using zk or hive lock provider, so I feel this may not be a big problem. I have another idea, can we let eve

[GitHub] [hudi] LinMingQiang commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
LinMingQiang commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186803549 👌🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [hudi] liujinhui1994 commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
liujinhui1994 commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186803283 I have another idea, can we let every lock provider return false if it cannot acquire a lock? WDYT @fengjian428 It's actually quite good to do so. In this way, the logic of the

[GitHub] [hudi] yanghua commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
yanghua commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186797199 > I also found this, but there is a LOCK_ACQUIRE_WAIT_TIMEOUT_MS when using zk or hive lock provider, so I feel this may not be a big problem. The config items: LOCK_ACQUIRE_WAIT_TIMEO

[GitHub] [hudi] satishmalladi-m commented on issue #6107: [SUPPORT] RO table did not get updated while RT table did

2022-07-17 Thread GitBox
satishmalladi-m commented on issue #6107: URL: https://github.com/apache/hudi/issues/6107#issuecomment-1186795668 please find below configuration which we are using currently hudi_options = { 'hoodie.datasource.write.table.type': self._write_table_type, 'hoo

[GitHub] [hudi] prasannarajaperumal commented on pull request #5885: [RFC-51][HUDI-3478] Hudi CDC

2022-07-17 Thread GitBox
prasannarajaperumal commented on PR #5885: URL: https://github.com/apache/hudi/pull/5885#issuecomment-1186787667 I understand CDC is a database concept. My point was incremental query is also just a form of CDC if you think about how it is used. Yes the schema is different based on the mode

[GitHub] [hudi] codope commented on a diff in pull request #4958: [HUDI-3558] Consistent bucket index: bucket resizing (split&merge) & concurrent write during resizing

2022-07-17 Thread GitBox
codope commented on code in PR #4958: URL: https://github.com/apache/hudi/pull/4958#discussion_r922970792 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java: ## @@ -30,6 +30,9 @@ import org.apache.hudi.index.HoodieIndex; import org.ap

[GitHub] [hudi] hudi-bot commented on pull request #5954: [HUDI-4303] Use Hive sentinel value as partition default to avoid casting err

2022-07-17 Thread GitBox
hudi-bot commented on PR #5954: URL: https://github.com/apache/hudi/pull/5954#issuecomment-1186777517 ## CI report: * db141230ef226ff1f8422e4802886812d66a401b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9952

[GitHub] [hudi] fengjian428 commented on issue #6127: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql and Sync Hive

2022-07-17 Thread GitBox
fengjian428 commented on issue #6127: URL: https://github.com/apache/hudi/issues/6127#issuecomment-1186775480 Yeah, KnightChess is right, I think this can answer this question > only _ro and _rt table will sync partition info to metastore, when I use presto to query raw table, cannot

[GitHub] [hudi] hudi-bot commented on pull request #5954: [HUDI-4303] Use Hive sentinel value as partition default to avoid casting err

2022-07-17 Thread GitBox
hudi-bot commented on PR #5954: URL: https://github.com/apache/hudi/pull/5954#issuecomment-1186775449 ## CI report: * db141230ef226ff1f8422e4802886812d66a401b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9952

[GitHub] [hudi] hudi-bot commented on pull request #6124: [HUDI-4404] Fix insert into dynamic partition write misalignment

2022-07-17 Thread GitBox
hudi-bot commented on PR #6124: URL: https://github.com/apache/hudi/pull/6124#issuecomment-1186773320 ## CI report: * 11afb71302431adf00ebd2a851e05fab269ad531 UNKNOWN * c37b27ddff41e13e7d4c590f11dfb05c3c762d15 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
hudi-bot commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186773262 ## CI report: * cf4238bd32291c0003bf4157bfa84db90fb2f82d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186773240 ## CI report: * aceed136ae834d5dc085b5cab900ccae0ab2be6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186749920 ## CI report: * aceed136ae834d5dc085b5cab900ccae0ab2be6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] KnightChess commented on issue #6107: [SUPPORT] RO table did not get updated while RT table did

2022-07-17 Thread GitBox
KnightChess commented on issue #6107: URL: https://github.com/apache/hudi/issues/6107#issuecomment-1186743466 for RO table delta commit, It can only be seen after compact action -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] hudi-bot commented on pull request #6129: [HUDI-4411] Bump Spark versions to 3.2.2

2022-07-17 Thread GitBox
hudi-bot commented on PR #6129: URL: https://github.com/apache/hudi/pull/6129#issuecomment-1186743253 ## CI report: * b4a9570abf7db4e829a9e487c18c1fec76342659 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1002

[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
hudi-bot commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186743165 ## CI report: * f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890

[GitHub] [hudi] hudi-bot commented on pull request #6129: [HUDI-4411] Bump Spark versions to 3.2.2

2022-07-17 Thread GitBox
hudi-bot commented on PR #6129: URL: https://github.com/apache/hudi/pull/6129#issuecomment-1186740647 ## CI report: * b4a9570abf7db4e829a9e487c18c1fec76342659 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
hudi-bot commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186740532 ## CI report: * f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890

[GitHub] [hudi] hudi-bot commented on pull request #5659: [HUDI-4137] SnowflakeSyncTool MVP implementation to integrate with Snowflake

2022-07-17 Thread GitBox
hudi-bot commented on PR #5659: URL: https://github.com/apache/hudi/pull/5659#issuecomment-1186740180 ## CI report: * 5b8c6c515dfd37f7fbce8a98468b17e6998947b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] KnightChess commented on issue #6127: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql and Sync Hive

2022-07-17 Thread GitBox
KnightChess commented on issue #6127: URL: https://github.com/apache/hudi/issues/6127#issuecomment-1186739851 only _ro and _rt table will sync partition info to metastore, when I use presto to query raw table, cannot query the data, but _ro or _rt success. Depending on the implementation of

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186738156 ## CI report: * aceed136ae834d5dc085b5cab900ccae0ab2be6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
hudi-bot commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186738081 ## CI report: * f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890

[GitHub] [hudi] hudi-bot commented on pull request #5659: [HUDI-4137] SnowflakeSyncTool MVP implementation to integrate with Snowflake

2022-07-17 Thread GitBox
hudi-bot commented on PR #5659: URL: https://github.com/apache/hudi/pull/5659#issuecomment-1186737709 ## CI report: * 5b8c6c515dfd37f7fbce8a98468b17e6998947b7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] KnightChess commented on issue #6127: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql and Sync Hive

2022-07-17 Thread GitBox
KnightChess commented on issue #6127: URL: https://github.com/apache/hudi/issues/6127#issuecomment-1186737362 @jiangbiao910 which query engine do you use to query data? make sure `hudi_spark_table0718_mor_0111` has no partition info in hive metastore -- This is an automated message from t

[jira] [Updated] (HUDI-4411) Bump Spark version to 3.2.2

2022-07-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4411: - Labels: pull-request-available (was: ) > Bump Spark version to 3.2.2 > --

[GitHub] [hudi] deadwind4 opened a new pull request, #6129: [HUDI-4411] Bump Spark versions to 3.2.2

2022-07-17 Thread GitBox
deadwind4 opened a new pull request, #6129: URL: https://github.com/apache/hudi/pull/6129 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpo

[GitHub] [hudi] txl2017 opened a new issue, #6128: [SUPPORT] spark-sql can't create hudi table

2022-07-17 Thread GitBox
txl2017 opened a new issue, #6128: URL: https://github.com/apache/hudi/issues/6128 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr..

[jira] [Commented] (HUDI-4089) Support HMS for flink HoodieCatalog

2022-07-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17567800#comment-17567800 ] Danny Chen commented on HUDI-4089: -- Fixed via master branch: 9282611baec5069156f627a2c96e

[jira] [Resolved] (HUDI-4089) Support HMS for flink HoodieCatalog

2022-07-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-4089. -- > Support HMS for flink HoodieCatalog > --- > > Key: HUDI-40

[hudi] branch master updated (3964c476e0 -> 9282611bae)

2022-07-17 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 3964c476e0 Fix file group count issue with metadata partitions (#5892) add 9282611bae [HUDI-4098] Support HMS fo

[GitHub] [hudi] danny0405 merged pull request #6082: [HUDI-4098]Support HMS for flink HudiCatalog

2022-07-17 Thread GitBox
danny0405 merged PR #6082: URL: https://github.com/apache/hudi/pull/6082 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] fengjian428 commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
fengjian428 commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186730868 Cool~ I also found this, but there is a LOCK_ACQUIRE_WAIT_TIMEOUT_MS when using zk or hive lock provider, so I feel this may not be a big problem. -- This is an automated message from

[jira] [Assigned] (HUDI-3877) Support Java reader for hudi

2022-07-17 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu reassigned HUDI-3877: Assignee: Forward Xu > Support Java reader for hudi > > >

[jira] [Updated] (HUDI-4156) AsyncIndexer fails for column stats partition

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-4156: -- Priority: Blocker (was: Major) > AsyncIndexer fails for column stats partition > -

[jira] [Updated] (HUDI-4156) AsyncIndexer fails for column stats partition

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-4156: -- Epic Link: HUDI-2488 > AsyncIndexer fails for column stats partition >

[jira] [Closed] (HUDI-4156) AsyncIndexer fails for column stats partition

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-4156. - Resolution: Fixed > AsyncIndexer fails for column stats partition > -

[jira] [Updated] (HUDI-2083) Hudi CLI does not work with S3

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2083: -- Status: Patch Available (was: In Progress) > Hudi CLI does not work with S3 > -

[jira] [Updated] (HUDI-2083) Hudi CLI does not work with S3

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2083: -- Fix Version/s: 0.11.1 > Hudi CLI does not work with S3 > -- > >

[jira] [Closed] (HUDI-2083) Hudi CLI does not work with S3

2022-07-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-2083. - Resolution: Fixed The docs have been updated with the correct usage https://hudi.apache.org/docs/next/cli

[GitHub] [hudi] codope closed pull request #3222: [HUDI-2083] Support Hudi Cli to work with S3

2022-07-17 Thread GitBox
codope closed pull request #3222: [HUDI-2083] Support Hudi Cli to work with S3 URL: https://github.com/apache/hudi/pull/3222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] codope commented on pull request #3222: [HUDI-2083] Support Hudi Cli to work with S3

2022-07-17 Thread GitBox
codope commented on PR #3222: URL: https://github.com/apache/hudi/pull/3222#issuecomment-1186726144 This patch is not strictly necessary. `S3AFileSystem` not found issue was due to missing correct version of hadoop-aws in the classpath. We have updated our docs on how to use hudi-cli for ta

[GitHub] [hudi] yanghua commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
yanghua commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186716687 cc @fengjian428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [hudi] hudi-bot commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
hudi-bot commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186715598 ## CI report: * d3c896da8d2e2d50ca806a9ca630ac5b3d0ef6d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[jira] [Created] (HUDI-4411) Bump Spark version to 3.2.2

2022-07-17 Thread LuNing Wang (Jira)
LuNing Wang created HUDI-4411: - Summary: Bump Spark version to 3.2.2 Key: HUDI-4411 URL: https://issues.apache.org/jira/browse/HUDI-4411 Project: Apache Hudi Issue Type: Improvement R

[GitHub] [hudi] hudi-bot commented on pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
hudi-bot commented on PR #6122: URL: https://github.com/apache/hudi/pull/6122#issuecomment-1186713392 ## CI report: * d3c896da8d2e2d50ca806a9ca630ac5b3d0ef6d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] yanghua commented on a diff in pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
yanghua commented on code in PR #6122: URL: https://github.com/apache/hudi/pull/6122#discussion_r922945243 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java: ## @@ -74,6 +74,11 @@ public void lock() { if (retryCou

[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider

2022-07-17 Thread GitBox
hudi-bot commented on PR #6071: URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186711279 ## CI report: * 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996

[GitHub] [hudi] hudi-bot commented on pull request #6124: [HUDI-4404] Fix insert into dynamic partition write misalignment

2022-07-17 Thread GitBox
hudi-bot commented on PR #6124: URL: https://github.com/apache/hudi/pull/6124#issuecomment-1186709751 ## CI report: * e48b64115273515ec00061a0ecc81506a8784f03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider

2022-07-17 Thread GitBox
hudi-bot commented on PR #6071: URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186709681 ## CI report: * 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996

[GitHub] [hudi] todd5167 commented on issue #6094: [SUPPORT] hudi rollback throw java.lang.IllegalArgumentException

2022-07-17 Thread GitBox
todd5167 commented on issue #6094: URL: https://github.com/apache/hudi/issues/6094#issuecomment-1186709177 I'll take a look after cherry pick -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [hudi] hudi-bot commented on pull request #6124: [HUDI-4404] Fix insert into dynamic partition write misalignment

2022-07-17 Thread GitBox
hudi-bot commented on PR #6124: URL: https://github.com/apache/hudi/pull/6124#issuecomment-1186708269 ## CI report: * e48b64115273515ec00061a0ecc81506a8784f03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] xushiyan closed issue #6075: Hudi's current driver cache management has some problems

2022-07-17 Thread GitBox
xushiyan closed issue #6075: Hudi's current driver cache management has some problems URL: https://github.com/apache/hudi/issues/6075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [hudi] xushiyan commented on issue #6075: Hudi's current driver cache management has some problems

2022-07-17 Thread GitBox
xushiyan commented on issue #6075: URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186707533 @Zhangshunyu you meant fixed in spark source code? it would be interesting to see the upstream patch for spark. -- This is an automated message from the Apache Git Service. To respon

[jira] [Updated] (HUDI-4410) Incremental read from archived commits for Spark

2022-07-17 Thread Hui An (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HUDI-4410: - Issue Type: New Feature (was: Improvement) > Incremental read from archived commits for Spark > -

[jira] [Created] (HUDI-4410) Incremental read from archived commits for Spark

2022-07-17 Thread Hui An (Jira)
Hui An created HUDI-4410: Summary: Incremental read from archived commits for Spark Key: HUDI-4410 URL: https://issues.apache.org/jira/browse/HUDI-4410 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] danny0405 commented on issue #6094: [SUPPORT] hudi rollback throw java.lang.IllegalArgumentException

2022-07-17 Thread GitBox
danny0405 commented on issue #6094: URL: https://github.com/apache/hudi/issues/6094#issuecomment-1186706075 It may be related to this PR: https://github.com/apache/hudi/pull/5357, which has tweaked the compaction rollback strategy. -- This is an automated message from the Apache Git Servi

[GitHub] [hudi] boneanxs commented on pull request #6028: [HUDI-4355] Bulk insert As Row: Should also repartiiton records if populateMetaFields is false

2022-07-17 Thread GitBox
boneanxs commented on PR #6028: URL: https://github.com/apache/hudi/pull/6028#issuecomment-1186705943 ping @yihua, Hi, Can we merge this now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [hudi] yanghua commented on a diff in pull request #6122: [HUDI-4409] LockManager improve wait time logic

2022-07-17 Thread GitBox
yanghua commented on code in PR #6122: URL: https://github.com/apache/hudi/pull/6122#discussion_r922941155 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java: ## @@ -71,6 +71,12 @@ public void lock() { LOG.info("Re

[GitHub] [hudi] danny0405 commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
danny0405 commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186705005 I think we can cc @LinMingQiang ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [hudi] xushiyan commented on issue #6067: [SUPPORT] what the Incremental query should be ?

2022-07-17 Thread GitBox
xushiyan commented on issue #6067: URL: https://github.com/apache/hudi/issues/6067#issuecomment-1186703529 @fengjian428 incremental query is to give you changes between BEGIN_INSTANTTIME_OPT_KEY and now. In your example, you set the BEGIN_INSTANTTIME_OPT_KEY to 0, it means give all records'

[GitHub] [hudi] xushiyan closed issue #6067: [SUPPORT] what the Incremental query should be ?

2022-07-17 Thread GitBox
xushiyan closed issue #6067: [SUPPORT] what the Incremental query should be ? URL: https://github.com/apache/hudi/issues/6067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] Zhangshunyu commented on issue #6075: Hudi's current driver cache management has some problems

2022-07-17 Thread GitBox
Zhangshunyu commented on issue #6075: URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186698754 @xushiyan Hi shiyan, thanks for you reply, i find this is a spark problem as spark didnt release the cache for 1 session when the session is closed. we have fixed this in spark. -

[GitHub] [hudi] xushiyan commented on issue #6127: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql and Sync Hive

2022-07-17 Thread GitBox
xushiyan commented on issue #6127: URL: https://github.com/apache/hudi/issues/6127#issuecomment-1186696783 cc @fengjian428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] xushiyan commented on issue #6075: Hudi's current driver cache management has some problems

2022-07-17 Thread GitBox
xushiyan commented on issue #6075: URL: https://github.com/apache/hudi/issues/6075#issuecomment-1186694753 @Zhangshunyu thanks for the analysis. i see you've already done some profiling and code analysis > When a session is released, the corresponding cache of the session is not rele

[GitHub] [hudi] hudi-bot commented on pull request #6124: [HUDI-4404] Fix insert into dynamic partition write misalignment

2022-07-17 Thread GitBox
hudi-bot commented on PR #6124: URL: https://github.com/apache/hudi/pull/6124#issuecomment-1186689556 ## CI report: * e48b64115273515ec00061a0ecc81506a8784f03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] hudi-bot commented on pull request #6082: [HUDI-4098]Support HMS for flink HudiCatalog

2022-07-17 Thread GitBox
hudi-bot commented on PR #6082: URL: https://github.com/apache/hudi/pull/6082#issuecomment-1186689451 ## CI report: * 5eab65631fe3ad51e0b7cf7af4b64c3b23666c00 UNKNOWN * e66aff62e11f58d0304a69097571a6a98dcb5001 UNKNOWN * 02696e18ec3a8529dca26a1d8dec254c614811fc UNKNOWN * 8a

[GitHub] [hudi] danny0405 commented on a diff in pull request #6082: [HUDI-4098]Support HMS for flink HudiCatalog

2022-07-17 Thread GitBox
danny0405 commented on code in PR #6082: URL: https://github.com/apache/hudi/pull/6082#discussion_r922932085 ## hudi-flink-datasource/hudi-flink/pom.xml: ## @@ -269,7 +269,42 @@ - + +javax.transaction +jta

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186685838 ## CI report: * ee3a4dcd4323de805a78ede5f03b0c35417bc42b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #6082: [HUDI-4098]Support HMS for flink HudiCatalog

2022-07-17 Thread GitBox
hudi-bot commented on PR #6082: URL: https://github.com/apache/hudi/pull/6082#issuecomment-1186685762 ## CI report: * 5eab65631fe3ad51e0b7cf7af4b64c3b23666c00 UNKNOWN * e66aff62e11f58d0304a69097571a6a98dcb5001 UNKNOWN * 02696e18ec3a8529dca26a1d8dec254c614811fc UNKNOWN * 8a

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186683560 ## CI report: * ee3a4dcd4323de805a78ede5f03b0c35417bc42b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #6082: [HUDI-4098]Support HMS for flink HudiCatalog

2022-07-17 Thread GitBox
hudi-bot commented on PR #6082: URL: https://github.com/apache/hudi/pull/6082#issuecomment-1186683452 ## CI report: * 5eab65631fe3ad51e0b7cf7af4b64c3b23666c00 UNKNOWN * e66aff62e11f58d0304a69097571a6a98dcb5001 UNKNOWN * 02696e18ec3a8529dca26a1d8dec254c614811fc UNKNOWN * 8a

[hudi] branch master updated (ded197800a -> 3964c476e0)

2022-07-17 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from ded197800a [HUDI-4170] Make user can use hoodie.datasource.read.paths to read necessary files (#5722) add 3964c476

[GitHub] [hudi] codope merged pull request #5892: [HUDI-4156] Fixing file group count issues with metadata partitions

2022-07-17 Thread GitBox
codope merged PR #5892: URL: https://github.com/apache/hudi/pull/5892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.or

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186681191 ## CI report: * ee3a4dcd4323de805a78ede5f03b0c35417bc42b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] rahil-c commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
rahil-c commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186679615 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] jiangbiao910 opened a new issue, #6127: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql and Sync Hive

2022-07-17 Thread GitBox
jiangbiao910 opened a new issue, #6127: URL: https://github.com/apache/hudi/issues/6127 Hudi has recently released the latest version of Hudi, 0.11.1, which is pulled from Github and modified as follows for our Hadoop environment CDH6.3.2。 Upgrading to 0.11.1 resulting use sparksql: `

[GitHub] [hudi] xushiyan closed issue #6086: Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
xushiyan closed issue #6086: Support to trigger the compaction in the flink batch mode write. URL: https://github.com/apache/hudi/issues/6086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [hudi] xushiyan commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.

2022-07-17 Thread GitBox
xushiyan commented on PR #6093: URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186653728 @LinMingQiang @danny0405 can we aim to land this for 0.12? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[jira] [Updated] (HUDI-4385) Support to trigger the compaction in the flink batch mode.

2022-07-17 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4385: - Fix Version/s: 0.12.0 > Support to trigger the compaction in the flink batch mode. >

[GitHub] [hudi] xushiyan commented on issue #6097: spark timestamp issue help!!!

2022-07-17 Thread GitBox
xushiyan commented on issue #6097: URL: https://github.com/apache/hudi/issues/6097#issuecomment-1186648399 @leoyy0316 can you provide script and config for reproducing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [hudi] xushiyan commented on issue #6104: [SUPPORT] Hope to maintain a stable version

2022-07-17 Thread GitBox
xushiyan commented on issue #6104: URL: https://github.com/apache/hudi/issues/6104#issuecomment-1186641237 it does make sense to have one. I also feel 1.0 can be good candidate for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] xushiyan commented on issue #6125: [SUPPORT] hudi-examples-dbt not running with spark thrift server

2022-07-17 Thread GitBox
xushiyan commented on issue #6125: URL: https://github.com/apache/hudi/issues/6125#issuecomment-1186639295 hey @vingov can you take a look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186631379 ## CI report: * ee3a4dcd4323de805a78ede5f03b0c35417bc42b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1001

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186631323 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * 4eff7886efc685d32c6eba883f81baa0c327961b Azure: [SUCCES

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186618741 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * af12dbd43e995bfde3d689ee8752549e2f2fb4c5 Azure: [SUCCES

[GitHub] [hudi] hudi-bot commented on pull request #5630: [HUDI-3994] - Added support for initializing DeltaStreamer without a …

2022-07-17 Thread GitBox
hudi-bot commented on PR #5630: URL: https://github.com/apache/hudi/pull/5630#issuecomment-1186611159 ## CI report: * c0f7b56b2b0bb8caafc487a1e4da4380e51fc856 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186610656 ## CI report: * 8987303a15c59816cd71604d42e01e0277ab121d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9994

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186610615 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * c92b5a37055a9dc9a5c505939c3ea60ffe35c823 Azure: [FAILUR

[GitHub] [hudi] hudi-bot commented on pull request #5630: [HUDI-3994] - Added support for initializing DeltaStreamer without a …

2022-07-17 Thread GitBox
hudi-bot commented on PR #5630: URL: https://github.com/apache/hudi/pull/5630#issuecomment-1186610540 ## CI report: * c0f7b56b2b0bb8caafc487a1e4da4380e51fc856 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] Neuw84 commented on pull request #5630: [HUDI-3994] - Added support for initializing DeltaStreamer without a …

2022-07-17 Thread GitBox
Neuw84 commented on PR #5630: URL: https://github.com/apache/hudi/pull/5630#issuecomment-1186610132 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [hudi] hudi-bot commented on pull request #6117: Use Spark 3.2 as default Spark version, (older rebase)

2022-07-17 Thread GitBox
hudi-bot commented on PR #6117: URL: https://github.com/apache/hudi/pull/6117#issuecomment-1186610067 ## CI report: * 8987303a15c59816cd71604d42e01e0277ab121d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9994

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186610012 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * c92b5a37055a9dc9a5c505939c3ea60ffe35c823 Azure: [FAILUR

[GitHub] [hudi] hudi-bot commented on pull request #5659: [HUDI-4137] SnowflakeSyncTool MVP implementation to integrate with Snowflake

2022-07-17 Thread GitBox
hudi-bot commented on PR #5659: URL: https://github.com/apache/hudi/pull/5659#issuecomment-1186609253 ## CI report: * 5b8c6c515dfd37f7fbce8a98468b17e6998947b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1000

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186601547 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * c92b5a37055a9dc9a5c505939c3ea60ffe35c823 Azure: [FAILUR

[GitHub] [hudi] hudi-bot commented on pull request #5943: [HUDI-4186] Support Hudi with Spark 3.3.0

2022-07-17 Thread GitBox
hudi-bot commented on PR #5943: URL: https://github.com/apache/hudi/pull/5943#issuecomment-1186600925 ## CI report: * fa048b175c2b3b5a80c6ef8d0b9709097b822cfb UNKNOWN * b94604147edcfc5040b6cf8a1a649e9a0cf1eb2a UNKNOWN * c92b5a37055a9dc9a5c505939c3ea60ffe35c823 Azure: [FAILUR

  1   2   >