[GitHub] [hudi] xushiyan commented on issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

2021-10-18 Thread GitBox
xushiyan commented on issue #3814: URL: https://github.com/apache/hudi/issues/3814#issuecomment-946390788 @rubenssoto since you're on EMR, please use EMR pre-installed hudi jars instead of open source ones ``` --packages

[GitHub] [hudi] xushiyan closed issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

2021-10-18 Thread GitBox
xushiyan closed issue #3814: URL: https://github.com/apache/hudi/issues/3814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] rohit-m-99 commented on issue #3821: [SUPPORT] Ingestion taking very long time getting small files from partitions/

2021-10-18 Thread GitBox
rohit-m-99 commented on issue #3821: URL: https://github.com/apache/hudi/issues/3821#issuecomment-946385171 Thank you for the advice, how do you set the number of partitions when using df.write()? Currently basing my code off of the intro guide found here:

[GitHub] [hudi] xushiyan commented on issue #3821: [SUPPORT] Ingestion taking very long time getting small files from partitions/

2021-10-18 Thread GitBox
xushiyan commented on issue #3821: URL: https://github.com/apache/hudi/issues/3821#issuecomment-946383159 @rohit-m-99 this is likely due to non-partitioned dataset

[jira] [Comment Edited] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430316#comment-17430316 ] liyuanzhao435 edited comment on HUDI-2576 at 10/19/21, 5:33 AM: flink

[jira] [Comment Edited] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430316#comment-17430316 ] liyuanzhao435 edited comment on HUDI-2576 at 10/19/21, 5:33 AM: flink

[jira] [Commented] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430316#comment-17430316 ] liyuanzhao435 commented on HUDI-2576: - flink jobmanager deleted the file :   *2021-10-19

[jira] [Commented] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430311#comment-17430311 ] liyuanzhao435 commented on HUDI-2576: - I checked the hdfs audit log, the parquet file created and then

[GitHub] [hudi] xushiyan closed issue #3728: [SUPPORT] Hudi Flink S3 Java Example

2021-10-18 Thread GitBox
xushiyan closed issue #3728: URL: https://github.com/apache/hudi/issues/3728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #2544: [SUPPORT]failed to read timestamp column in version 0.7.0 even when HIVE_SUPPORT_TIMESTAMP is enabled

2021-10-18 Thread GitBox
nsivabalan commented on issue #2544: URL: https://github.com/apache/hudi/issues/2544#issuecomment-946369974 Closing due to inactivity and the issue is not reproducible anymore. thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan closed issue #2544: [SUPPORT]failed to read timestamp column in version 0.7.0 even when HIVE_SUPPORT_TIMESTAMP is enabled

2021-10-18 Thread GitBox
nsivabalan closed issue #2544: URL: https://github.com/apache/hudi/issues/2544 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #3603: [SUPPORT] delta streamer Failed to archive commits

2021-10-18 Thread GitBox
nsivabalan commented on issue #3603: URL: https://github.com/apache/hudi/issues/3603#issuecomment-946369090 @fengjian428 : hey, can you give us any updates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] nsivabalan commented on issue #3559: [SUPPORT] Failed to archive commits

2021-10-18 Thread GitBox
nsivabalan commented on issue #3559: URL: https://github.com/apache/hudi/issues/3559#issuecomment-946367293 This was fixed in 090. closing it out. If you run into any issues, do reach out to us. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan closed issue #3559: [SUPPORT] Failed to archive commits

2021-10-18 Thread GitBox
nsivabalan closed issue #3559: URL: https://github.com/apache/hudi/issues/3559 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.

2021-10-18 Thread GitBox
nsivabalan commented on issue #2802: URL: https://github.com/apache/hudi/issues/2802#issuecomment-946364855 @aditiwari01 : when you get a chance can you respond. Will close out in a week if we don't hear from you. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-10-18 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-946363380 @codope : Can you create a ticket for adding ability via hudi-cli to clean up dangling data files. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Updated] (HUDI-2511) Aggressive archival configs compared to cleaner configs make cleaning moot

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2511: -- Priority: Blocker (was: Major) > Aggressive archival configs compared to cleaner

[GitHub] [hudi] nsivabalan closed issue #2564: Hoodie clean is not deleting old files

2021-10-18 Thread GitBox
nsivabalan closed issue #2564: URL: https://github.com/apache/hudi/issues/2564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] absognety edited a comment on issue #3758: [SUPPORT] Issues when writing dataframe to hudi format with hive syncing enabled for AWS Athena and Glue metadata persistence

2021-10-18 Thread GitBox
absognety edited a comment on issue #3758: URL: https://github.com/apache/hudi/issues/3758#issuecomment-946311994 @nsivabalan I can confidently say that this is intermittently occurring issue, especially when we have concurrency in our code - doing concurrent writes to multiple tables in

[GitHub] [hudi] absognety edited a comment on issue #3758: [SUPPORT] Issues when writing dataframe to hudi format with hive syncing enabled for AWS Athena and Glue metadata persistence

2021-10-18 Thread GitBox
absognety edited a comment on issue #3758: URL: https://github.com/apache/hudi/issues/3758#issuecomment-946311994 @nsivabalan I can confidently say that this is intermittently occurring issue, especially when we have concurrency in our code - doing concurrent writes to different hudi

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * d108ef91b835ec89276863ac062bcc5cad6a2081 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * c4ed928cfa949daca478608bee6046995b106c7d Azure:

[GitHub] [hudi] yanghua commented on pull request #3773: [HUDI-2507] Generate more dependency list file for other bundles

2021-10-18 Thread GitBox
yanghua commented on pull request #3773: URL: https://github.com/apache/hudi/pull/3773#issuecomment-946329906 @vinothchandar Do you have any thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] yanghua commented on pull request #3773: [HUDI-2507] Generate more dependency list file for other bundles

2021-10-18 Thread GitBox
yanghua commented on pull request #3773: URL: https://github.com/apache/hudi/pull/3773#issuecomment-946329594 > LGTM. Optional: maybe having a test PR to show what diffs people will get if changed/added a dependency can help understand the impact easily. sounds good, will try to

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * c4ed928cfa949daca478608bee6046995b106c7d Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * 0e29ebfbbc37cd342017bdd8290e34bf5336210d Azure:

[jira] [Resolved] (HUDI-2572) Strength flink compaction rollback strategy

2021-10-18 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2572. -- Resolution: Fixed Fixed via master branch: 3a78be9203a9c3cea33fa6120c89f7702275fc31 > Strength flink

[hudi] branch master updated: [HUDI-2572] Strength flink compaction rollback strategy (#3819)

2021-10-18 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3a78be9 [HUDI-2572] Strength flink compaction

[GitHub] [hudi] danny0405 merged pull request #3819: [HUDI-2572] Strength flink compaction rollback strategy

2021-10-18 Thread GitBox
danny0405 merged pull request #3819: URL: https://github.com/apache/hudi/pull/3819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430282#comment-17430282 ] liyuanzhao435 commented on HUDI-2576: - the missing parquet file , either not created or deleted. 

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
nsivabalan commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946312014 @davehagman let's proceed with the approach you suggested. If others have any thoughts, I can take it up in a follow up PR. but lets proceed with this for now. One more

[GitHub] [hudi] absognety commented on issue #3758: [SUPPORT] Issues when writing dataframe to hudi format with hive syncing enabled for AWS Athena and Glue metadata persistence

2021-10-18 Thread GitBox
absognety commented on issue #3758: URL: https://github.com/apache/hudi/issues/3758#issuecomment-946311994 @nsivabalan I can confidently say that this is intermittently occurring issue, especially when we have concurrency in our code - doing concurrent writes to different hudi partitions

[GitHub] [hudi] rohit-m-99 opened a new issue #3821: [SUPPORT] Ingestion taking very long time getting small files from partitions/

2021-10-18 Thread GitBox
rohit-m-99 opened a new issue #3821: URL: https://github.com/apache/hudi/issues/3821 **Describe the problem you faced** Currently running Hudi 0.9.0 in production without a specific partition field. We are running using 6 workers each with 7 cores and 28GB of RAM. The files are stored

[jira] [Updated] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyuanzhao435 updated HUDI-2576: Attachment: error.txt > flink do checkpoint error because parquet file is missing >

[jira] [Created] (HUDI-2576) flink do checkpoint error because parquet file is missing

2021-10-18 Thread liyuanzhao435 (Jira)
liyuanzhao435 created HUDI-2576: --- Summary: flink do checkpoint error because parquet file is missing Key: HUDI-2576 URL: https://issues.apache.org/jira/browse/HUDI-2576 Project: Apache Hudi

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
nsivabalan commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946148718 yeah, the naming looks fine by me. btw, Can you please attach jira ticket to PR. prefix w/ ticket id. Especially for bugs, we need a tracking ticket. -- This is an

[hudi] branch master updated (588a34a -> 335e80e)

2021-10-18 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 588a34a [HUDI-2571] Remove include-flink-sql-connector-hive profile from flink bundle (#3818) add 335e80e

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
davehagman commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946035803 I like that idea a lot. It reduces the chance of error as well. Here are some thoughts: > a new config called `hoodie.copy.over.deltastreamer.checkpoints` Since

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
nsivabalan edited a comment on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720 thanks a lot for fixing this Dave. I would like to propose something here. I am wondering why do we need to retrofit copying over delta streamer checkpoint into logic

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
nsivabalan edited a comment on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720 thanks a lot for fixing this Dave. I would like to propose something here. I am wondering why do we need to retrofit copying over delta streamer checkpoint into

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
nsivabalan commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720 thanks a lot for fixing this Dave. I would like to propose something here. I am wondering why do we need to retrofit copying over delta streamer checkpoint into

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915 ## CI report: * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure:

[GitHub] [hudi] vinothchandar merged pull request #3811: [HUDI-2561] BitCaskDiskMap - avoiding hostname resolution when logging messages

2021-10-18 Thread GitBox
vinothchandar merged pull request #3811: URL: https://github.com/apache/hudi/pull/3811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] xushiyan commented on pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-18 Thread GitBox
xushiyan commented on pull request #3781: URL: https://github.com/apache/hudi/pull/3781#issuecomment-945974656 @RocMarshal for this PR's failure, it's most likely based on an impacted master build. you may want to rebase next time to stay on top of master. -- This is an automated

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Priority: Blocker (was: Major) > Deadlock w/ multi writer due to double locking >

[jira] [Updated] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2559: -- Priority: Blocker (was: Major) > Ensure unique timestamps are generated for commit

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Parent: HUDI-1292 Issue Type: Sub-task (was: Bug) > Deadlock w/ multi writer

[jira] [Updated] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2559: -- Parent: HUDI-1292 Issue Type: Sub-task (was: Improvement) > Ensure unique

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915 ## CI report: * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure:

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
davehagman commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-945961702 I also noticed that there isn't any documentation around `hoodie.write.meta.key.prefixes` config in the multi-writer docs. We should add something about it since it is very

[GitHub] [hudi] hudi-bot commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
hudi-bot commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915 ## CI report: * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Labels: release-blocker sev:critical (was: ) > Deadlock w/ multi writer due to double

[jira] [Updated] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2559: -- Labels: release-blocker sev:critical (was: ) > Ensure unique timestamps are generated

[jira] [Updated] (HUDI-1912) Presto defaults to GenericHiveRecordCursor for all Hudi tables

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1912: -- Status: Patch Available (was: In Progress) > Presto defaults to GenericHiveRecordCursor for all Hudi

[jira] [Updated] (HUDI-1856) Upstream changes made in PrestoDB to eliminate file listing to Trino

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1856: -- Status: Patch Available (was: In Progress) > Upstream changes made in PrestoDB to eliminate file

[jira] [Updated] (HUDI-1500) Support incrementally reading clustering commit via Spark Datasource/DeltaStreamer

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1500: -- Status: Patch Available (was: In Progress) > Support incrementally reading clustering commit via

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

2021-10-18 Thread GitBox
davehagman commented on pull request #3820: URL: https://github.com/apache/hudi/pull/3820#issuecomment-945952200 Also worth noting that the config used to determine which keys are merged from the past commit into the current one is generic (`hoodie.write.meta.key.prefixes`). At the moment

[jira] [Updated] (HUDI-2287) Partition pruning not working on Hudi dataset

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2287: -- Remaining Estimate: 24h Original Estimate: 24h > Partition pruning not working on

[GitHub] [hudi] davehagman opened a new pull request #3820: [BUGFIX] Merge commit state from previous instant instead of current

2021-10-18 Thread GitBox
davehagman opened a new pull request #3820: URL: https://github.com/apache/hudi/pull/3820 ## What is the purpose of the pull request In order to support multi-writer concurrency where one writer is the Deltastreamer, other writers must copy any checkpoint state from previous

[jira] [Commented] (HUDI-2575) [UMBRELLA] Revamp CI bot

2021-10-18 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430073#comment-17430073 ] Raymond Xu commented on HUDI-2575: -- from [~codope] {quote}c) Apart from what you have already listed, may

[jira] [Updated] (HUDI-2575) [UMBRELLA] Revamp CI bot

2021-10-18 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2575: - Component/s: Testing > [UMBRELLA] Revamp CI bot > - > > Key:

[jira] [Created] (HUDI-2575) [UMBRELLA] Revamp CI bot

2021-10-18 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-2575: Summary: [UMBRELLA] Revamp CI bot Key: HUDI-2575 URL: https://issues.apache.org/jira/browse/HUDI-2575 Project: Apache Hudi Issue Type: New Feature

[jira] [Updated] (HUDI-1183) PrestoDB dependency on Apache Hudi

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1183: -- Parent: HUDI-2574 Issue Type: Sub-task (was: Improvement) > PrestoDB dependency on Apache Hudi

[jira] [Updated] (HUDI-1912) Presto defaults to GenericHiveRecordCursor for all Hudi tables

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1912: -- Parent: HUDI-2574 Issue Type: Sub-task (was: Bug) > Presto defaults to GenericHiveRecordCursor

[jira] [Updated] (HUDI-2409) Using HBase shaded jars in Hudi presto bundle

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2409: -- Parent: HUDI-2574 Issue Type: Sub-task (was: Task) > Using HBase shaded jars in Hudi presto

[jira] [Updated] (HUDI-2409) Using HBase shaded jars in Hudi presto bundle

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2409: -- Status: In Progress (was: Open) > Using HBase shaded jars in Hudi presto bundle >

[jira] [Updated] (HUDI-1978) [UMBRELLA] Support for Hudi tables in trino-hive connector

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1978: -- Summary: [UMBRELLA] Support for Hudi tables in trino-hive connector (was: [UMBRELLA] Support for Trino

[jira] [Created] (HUDI-2574) [UMBRELLA] Support for Hudi tables in presto-hive connector

2021-10-18 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-2574: - Summary: [UMBRELLA] Support for Hudi tables in presto-hive connector Key: HUDI-2574 URL: https://issues.apache.org/jira/browse/HUDI-2574 Project: Apache Hudi

[jira] [Updated] (HUDI-1978) [UMBRELLA] Support for Trino in Hive connector

2021-10-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1978: -- Summary: [UMBRELLA] Support for Trino in Hive connector (was: [UMBRELLA] Support for Trino) >

[jira] [Commented] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430050#comment-17430050 ] Dave Hagman commented on HUDI-2559: --- Testing approach 1 should be very easy given the way my branch is

[jira] [Comment Edited] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430048#comment-17430048 ] Dave Hagman edited comment on HUDI-2559 at 10/18/21, 2:40 PM: -- I have been

[jira] [Commented] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430048#comment-17430048 ] Dave Hagman commented on HUDI-2559: --- I have been extensively testing approach #2 and so far it has

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Summary: Deadlock w/ multi writer due to double locking (was: Deadlock w/ multi writer

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Description: With synchronous metadata patch, we added locking for cleaning and

[jira] [Created] (HUDI-2573) Deadlock w/ multi writer

2021-10-18 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-2573: - Summary: Deadlock w/ multi writer Key: HUDI-2573 URL: https://issues.apache.org/jira/browse/HUDI-2573 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Strength flink compaction rollback strategy

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 79056a50227330ba6965f7db5ca137fbdeff13ff UNKNOWN * 60ed0b24f27c773801db3228e5f53532ea3e0ae6 Azure:

[jira] [Commented] (HUDI-2563) Refactor XScheduleCompactionActionExecutor and CompactionTriggerStrategy.

2021-10-18 Thread Yuepeng Pan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430028#comment-17430028 ] Yuepeng Pan commented on HUDI-2563: --- Hi, [~danny0405] Could you help me to review this pr ? Thank you.

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Strength flink compaction rollback strategy

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 4c8a7378651c3911d30102d1c396fa59ef795c88 Azure:

[jira] [Comment Edited] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430012#comment-17430012 ] sivabalan narayanan edited comment on HUDI-2559 at 10/18/21, 1:24 PM: --

[jira] [Commented] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

2021-10-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430012#comment-17430012 ] sivabalan narayanan commented on HUDI-2559: --- Here are the possible solutions: # add millisec

[jira] [Updated] (HUDI-2572) Strength flink compaction rollback strategy

2021-10-18 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-2572: - Summary: Strength flink compaction rollback strategy (was: Make flink compaction commit fail-safe) >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Strength flink compaction rollback strategy

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 4c8a7378651c3911d30102d1c396fa59ef795c88 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Strength flink compaction rollback strategy

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 4c8a7378651c3911d30102d1c396fa59ef795c88 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 8c27c74a4b89d2242f846e85c725fdfc09f8786a Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 8c27c74a4b89d2242f846e85c725fdfc09f8786a Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 8c27c74a4b89d2242f846e85c725fdfc09f8786a Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-939200284 ## CI report: * 9e64e88d819b6b6bf5ccc5811ea5f4714138fc9e UNKNOWN * e7e7f170612fcecc8b07839d296f2c06972f2f44 UNKNOWN *

[GitHub] [hudi] novakov-alexey commented on pull request #3817: fix: HoodieDatasetBulkInsertHelper concurrently rowkey not found

2021-10-18 Thread GitBox
novakov-alexey commented on pull request #3817: URL: https://github.com/apache/hudi/pull/3817#issuecomment-945682795 @Carl-Zhou-CN feel free to take it to your PR. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] hudi-bot edited a comment on pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 8c27c74a4b89d2242f846e85c725fdfc09f8786a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
hudi-bot commented on pull request #3819: URL: https://github.com/apache/hudi/pull/3819#issuecomment-945644747 ## CI report: * 8c27c74a4b89d2242f846e85c725fdfc09f8786a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] hudi-bot edited a comment on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-939200284 ## CI report: * 9e64e88d819b6b6bf5ccc5811ea5f4714138fc9e UNKNOWN * c2bc8115f70b89dfc31f27645f98cfbff8d79c0f Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * 0e29ebfbbc37cd342017bdd8290e34bf5336210d Azure:

[GitHub] [hudi] danny0405 opened a new pull request #3819: [HUDI-2572] Make flink compaction commit fail-safe

2021-10-18 Thread GitBox
danny0405 opened a new pull request #3819: URL: https://github.com/apache/hudi/pull/3819 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[jira] [Updated] (HUDI-2572) Make flink compaction commit fail-safe

2021-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2572: - Labels: pull-request-available (was: ) > Make flink compaction commit fail-safe >

[GitHub] [hudi] RocMarshal commented on pull request #3813: [HUDI-2563][hudi-client] Refactor XScheduleCompactionActionExecutor and CompactionTriggerStrategy.

2021-10-18 Thread GitBox
RocMarshal commented on pull request #3813: URL: https://github.com/apache/hudi/pull/3813#issuecomment-945626073 Could someone help me to review it ? Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[jira] [Created] (HUDI-2572) Make flink compaction commit fail-safe

2021-10-18 Thread Danny Chen (Jira)
Danny Chen created HUDI-2572: Summary: Make flink compaction commit fail-safe Key: HUDI-2572 URL: https://issues.apache.org/jira/browse/HUDI-2572 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running

2021-10-18 Thread GitBox
fengjian428 commented on issue #3755: URL: https://github.com/apache/hudi/issues/3755#issuecomment-945612669 ![image](https://user-images.githubusercontent.com/4403474/137711199-f69d86e0-af37-4b87-8792-b7ea1bde89ec.png) I found two sparkHoodieBloomIndex were running, is that means

[GitHub] [hudi] hudi-bot edited a comment on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-939200284 ## CI report: * 9e64e88d819b6b6bf5ccc5811ea5f4714138fc9e UNKNOWN * c2bc8115f70b89dfc31f27645f98cfbff8d79c0f Azure:

[GitHub] [hudi] Carl-Zhou-CN commented on issue #3759: [SUPPORT] HoodieKeyException: recordKey value: "null"

2021-10-18 Thread GitBox
Carl-Zhou-CN commented on issue #3759: URL: https://github.com/apache/hudi/issues/3759#issuecomment-945608998 yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] hudi-bot edited a comment on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2021-10-18 Thread GitBox
hudi-bot edited a comment on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-939200284 ## CI report: * 9e64e88d819b6b6bf5ccc5811ea5f4714138fc9e UNKNOWN * c2bc8115f70b89dfc31f27645f98cfbff8d79c0f Azure:

  1   2   >