[GitHub] [hudi] Tandoy commented on issue #3598: [SUPPORT] An exception occurs when using OCC and the second concurrent writer

2021-09-30 Thread GitBox
Tandoy commented on issue #3598: URL: https://github.com/apache/hudi/issues/3598#issuecomment-931816793 > @Tandoy : any updates please. I'm very sorry that I didn't test it in time due to the holiday and other arrangements, but I will try it as soon as possible. -- This is an

[GitHub] [hudi] leesf merged pull request #3736: Add jfs support doc for hudi

2021-09-30 Thread GitBox
leesf merged pull request #3736: URL: https://github.com/apache/hudi/pull/3736 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch asf-site updated: [DOCS] Add jfs support doc for hudi (#3736)

2021-09-30 Thread leesf
This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 691e407 [DOCS] Add jfs support doc for hudi

[GitHub] [hudi] nsivabalan commented on a change in pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
nsivabalan commented on a change in pull request #3740: URL: https://github.com/apache/hudi/pull/3740#discussion_r719809110 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieConcatHandle.java ## @@ -44,38 +51,44 @@ * Simplified

[GitHub] [hudi] nsivabalan commented on a change in pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
nsivabalan commented on a change in pull request #3740: URL: https://github.com/apache/hudi/pull/3740#discussion_r719807050 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieConcatHandle.java ## @@ -94,4 +107,34 @@ public void

[hudi] branch master updated: [HUDI-2497] Refactor clean and restore actions in hudi-client module (#3734)

2021-09-30 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 46808dc [HUDI-2497] Refactor clean and

[GitHub] [hudi] nsivabalan merged pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
nsivabalan merged pull request #3734: URL: https://github.com/apache/hudi/pull/3734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931743495 sure. can we try hoodie.keep.max.commits: 20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] mauropelucchi commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
mauropelucchi commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931730823 @nsivabalan We already use your suggestions, so our actual conf is 'hoodie.table.name': table_name, 'hoodie.datasource.write.recordkey.field':

[GitHub] [hudi] hudi-bot edited a comment on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 0425dc32bc4b1acca25feef0d3481570ff7f2dbc Azure:

[GitHub] [hudi] davehagman edited a comment on issue #3598: [SUPPORT] An exception occurs when using OCC and the second concurrent writer

2021-09-30 Thread GitBox
davehagman edited a comment on issue #3598: URL: https://github.com/apache/hudi/issues/3598#issuecomment-931718635 @nsivabalan I created an issue ([here](https://issues.apache.org/jira/browse/HUDI-2275)) for this a while ago that has since been updated to `blocker` but it remains

[GitHub] [hudi] davehagman commented on issue #3598: [SUPPORT] An exception occurs when using OCC and the second concurrent writer

2021-09-30 Thread GitBox
davehagman commented on issue #3598: URL: https://github.com/apache/hudi/issues/3598#issuecomment-931718635 @nsivabalan I created an issue ([here](https://issues.apache.org/jira/browse/HUDI-2275)) for this a while ago that has since been updated to `blocker` but it remains unresolved. I

[GitHub] [hudi] yihua commented on a change in pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3741: URL: https://github.com/apache/hudi/pull/3741#discussion_r719765696 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/HoodieSparkMergeOnReadTableCompactor.java ## @@ -19,82 +19,36

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719758782 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java ## @@ -68,6 +77,21 @@ public

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719752579 ## File path: hudi-common/src/main/java/org/apache/hudi/common/function/SerializablePairFlatMapFunction.java ## @@ -0,0 +1,33 @@ +/* + * Licensed to the

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719756887 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java ## @@ -321,15 +324,14 @@ protected void

[GitHub] [hudi] hudi-bot edited a comment on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 0425dc32bc4b1acca25feef0d3481570ff7f2dbc Azure:

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719752579 ## File path: hudi-common/src/main/java/org/apache/hudi/common/function/SerializablePairFlatMapFunction.java ## @@ -0,0 +1,33 @@ +/* + * Licensed to the

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719752331 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java ## @@ -86,6 +110,17 @@ public

[jira] [Created] (HUDI-2510) QuickStart html page is showing 404

2021-09-30 Thread Rajesh Mahindra (Jira)
Rajesh Mahindra created HUDI-2510: - Summary: QuickStart html page is showing 404 Key: HUDI-2510 URL: https://issues.apache.org/jira/browse/HUDI-2510 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719751200 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -369,7 +372,14 @@ public void

[GitHub] [hudi] hudi-bot commented on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot commented on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 0425dc32bc4b1acca25feef0d3481570ff7f2dbc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] yihua commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719749416 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java ## @@ -0,0 +1,178 @@ +/* + * Licensed to

[GitHub] [hudi] yihua commented on pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on pull request #3727: URL: https://github.com/apache/hudi/pull/3727#issuecomment-931656856 @nsivabalan I addressed your comments in this PR on refactoring compaction actions: https://github.com/apache/hudi/pull/3741 -- This is an automated message from the Apache Git

[jira] [Updated] (HUDI-2501) Refactor compaction actions in hudi-client module

2021-09-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2501: - Labels: pull-request-available (was: ) > Refactor compaction actions in hudi-client module >

[GitHub] [hudi] yihua opened a new pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-09-30 Thread GitBox
yihua opened a new pull request #3741: URL: https://github.com/apache/hudi/pull/3741 ## What is the purpose of the pull request This PR introduces the `HoodieData` abstraction for the hudi-common and hudi-client modules, to hide the actual engine-specific data structures such as

[GitHub] [hudi] hudi-bot edited a comment on pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3734: URL: https://github.com/apache/hudi/pull/3734#issuecomment-930618920 ## CI report: * 68a20446360cb258d8f896d0d6bc0d49696f2788 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3734: URL: https://github.com/apache/hudi/pull/3734#issuecomment-930618920 ## CI report: * 3cec644131a4fda77510a97d548d3633b4731e78 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3734: URL: https://github.com/apache/hudi/pull/3734#issuecomment-930618920 ## CI report: * 3cec644131a4fda77510a97d548d3633b4731e78 Azure:

[GitHub] [hudi] yihua commented on a change in pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
yihua commented on a change in pull request #3734: URL: https://github.com/apache/hudi/pull/3734#discussion_r719653450 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java ## @@ -68,13 +78,85 @@ protected

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931544474 likely its the archival thats coming into play. let me know how it goes if you set higher value for hoodie.keep.max.commits. -- This is an automated message from the Apache Git

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931540799 also, can you try setting this to higher no hoodie.keep.max.commits: may be 10. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931537607 may be a dumb thought. can you try setting the config values as string. 'hoodie.compact.inline': 'true' 'hoodie.cleaner.commits.retained':'4' -- This is an automated

[GitHub] [hudi] nsivabalan edited a comment on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931533803 sorry, I see that you do have 5 compactions. Let me dig in more. Can you enable info logs and share it here. I don't even see cleaning getting triggered. else we would

[GitHub] [hudi] FeiZou commented on issue #3418: [SUPPORT] Hudi Upsert Very Slow/ Failed With No Space Left on Device

2021-09-30 Thread GitBox
FeiZou commented on issue #3418: URL: https://github.com/apache/hudi/issues/3418#issuecomment-931535592 Hey @nsivabalan @vinothchandar sorry to trouble you again, but anything you can suggested here? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] nsivabalan edited a comment on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931533803 sorry, I see that you do have 5 compactions. Let me dig in more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931533803 sorry, I see that you do have 5 compactions. Can you try enabling inline cleaning? hoodie.compact.inline=true -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] nsivabalan edited a comment on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931525996 thanks. Here is what is possibly happening. If you can tigger more updates, eventually you will see cleaning kicking in. In short, this has something to do w/ MOR table.

[GitHub] [hudi] nsivabalan edited a comment on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931525996 thanks. Here is what is possibly happening. If you can tigger more updates, eventually you will see cleaning kicking in. In short, this has something to do w/ MOR table.

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931525996 thanks. Here is what is possibly happening. If you can tigger more updates, eventually you will see cleaning kicking in. In short, this has something to do w/ MOR table. cleaner

[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3671: URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882 ## CI report: * b744966057d6850bbfa85b0d79750fb3ca3b0325 Azure:

[GitHub] [hudi] nsivabalan edited a comment on issue #3297: HoodieMetadataException throwed when execute merge and hoodie.metadata.enable='true'

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3297: URL: https://github.com/apache/hudi/issues/3297#issuecomment-931514853 I could not reproduce the issue where enabling metadata is failing. I tried locally. Can you try w/ latest master and let us know how it goes. ``` create table

[GitHub] [hudi] nsivabalan commented on issue #3297: HoodieMetadataException throwed when execute merge and hoodie.metadata.enable='true'

2021-09-30 Thread GitBox
nsivabalan commented on issue #3297: URL: https://github.com/apache/hudi/issues/3297#issuecomment-931514853 I could not reproduce the issue where enabling metadata is failing. I tried w/ latest master. ``` create table hudi_gh_ext2 using hudi location

[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3671: URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882 ## CI report: * 35e7f483618be007802784caa5c64b953cc0ffda Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3671: URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882 ## CI report: * 35e7f483618be007802784caa5c64b953cc0ffda Azure:

[GitHub] [hudi] nsivabalan edited a comment on issue #3582: [SUPPORT] Upsert to hudi table fails that got bootstrapped (w/ metadata only)

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3582: URL: https://github.com/apache/hudi/issues/3582#issuecomment-931414918 I got it working. not sure what was the issue earlier. here is the script. Non-partitioned. ``` val df =

[hudi] branch master updated (47ed917 -> 73e8ba7)

2021-09-30 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 47ed917 [HUDI-2440] Add dependency change diff script for dependency governace (#3674) add 73e8ba7

[GitHub] [hudi] nsivabalan merged pull request #3732: [HUDI-2499] Make jdbc-url, user and pass as non-required for other sync-modes

2021-09-30 Thread GitBox
nsivabalan merged pull request #3732: URL: https://github.com/apache/hudi/pull/3732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #3598: [SUPPORT] An exception occurs when using OCC and the second concurrent writer

2021-09-30 Thread GitBox
nsivabalan commented on issue #3598: URL: https://github.com/apache/hudi/issues/3598#issuecomment-931435751 @Tandoy : any updates please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426

2021-09-30 Thread GitBox
nsivabalan commented on a change in pull request #3590: URL: https://github.com/apache/hudi/pull/3590#discussion_r719513873 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -241,13 +242,16 @@ void

[GitHub] [hudi] hudi-bot edited a comment on pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3740: URL: https://github.com/apache/hudi/pull/3740#issuecomment-931381693 ## CI report: * f354e5708d11ffa375c3758e21eeb41d2e0566ed Azure:

[GitHub] [hudi] nsivabalan closed issue #3582: [SUPPORT] Upsert to hudi table fails that got bootstrapped (w/ metadata only)

2021-09-30 Thread GitBox
nsivabalan closed issue #3582: URL: https://github.com/apache/hudi/issues/3582 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #3582: [SUPPORT] Upsert to hudi table fails that got bootstrapped (w/ metadata only)

2021-09-30 Thread GitBox
nsivabalan commented on issue #3582: URL: https://github.com/apache/hudi/issues/3582#issuecomment-931414918 I got it working. not sure what was the issue earlier. here is the script. Non-partitioned. ``` val df =

[GitHub] [hudi] codope commented on issue #3724: [SUPPORT] Spark start reading stream from hudi dataset starting from given commit time

2021-09-30 Thread GitBox
codope commented on issue #3724: URL: https://github.com/apache/hudi/issues/3724#issuecomment-931404620 Generally, for incremental queries we need to set following configs: ``` "hoodie.datasource.query.type" : "incremental", "hoodie.datasource.read.begin.instanttime" :

[GitHub] [hudi] vinothchandar commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426

2021-09-30 Thread GitBox
vinothchandar commented on a change in pull request #3590: URL: https://github.com/apache/hudi/pull/3590#discussion_r719356462 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -241,13 +242,16 @@ void

[GitHub] [hudi] hudi-bot edited a comment on pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3740: URL: https://github.com/apache/hudi/pull/3740#issuecomment-931381693 ## CI report: * f354e5708d11ffa375c3758e21eeb41d2e0566ed Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
hudi-bot commented on pull request #3740: URL: https://github.com/apache/hudi/pull/3740#issuecomment-931381693 ## CI report: * f354e5708d11ffa375c3758e21eeb41d2e0566ed UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Commented] (HUDI-2496) Inserts are precombined even with dedup disabled

2021-09-30 Thread Helias Antoniou (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422816#comment-17422816 ] Helias Antoniou commented on HUDI-2496: --- Hey [~xushiyan], I opened the draft

[jira] [Updated] (HUDI-2488) Support bootstrapping a single or more partitions in metadata table while regular writers and table services are in progress

2021-09-30 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2488: -- Description: For now, we have only FILES partition in metadata table. and our

[jira] [Updated] (HUDI-2496) Inserts are precombined even with dedup disabled

2021-09-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2496: - Labels: pull-request-available sev:critical (was: sev:critical) > Inserts are precombined even

[GitHub] [hudi] helanto opened a new pull request #3740: [HUDI-2496] Insert duplicate keys when precombined is deactivated

2021-09-30 Thread GitBox
helanto opened a new pull request #3740: URL: https://github.com/apache/hudi/pull/3740 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[GitHub] [hudi] mauropelucchi commented on issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
mauropelucchi commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931369093 Attached the data for the 1 - first execution of the spark job

[GitHub] [hudi] mauropelucchi commented on issue #2564: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
mauropelucchi commented on issue #2564: URL: https://github.com/apache/hudi/issues/2564#issuecomment-931364899 @nsivabalan Here the new issue https://github.com/apache/hudi/issues/3739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hudi] mauropelucchi opened a new issue #3739: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
mauropelucchi opened a new issue #3739: URL: https://github.com/apache/hudi/issues/3739 I am trying to see if hudi clean is triggering and cleaning my files, but however I do not see any action being performed on cleaning the old log files. **To Reproduce** I am writing some

[jira] [Assigned] (HUDI-2488) Support bootstrapping a single or more partitions in metadata table while regular writers and table services are in progress

2021-09-30 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2488: - Assignee: sivabalan narayanan > Support bootstrapping a single or more

[GitHub] [hudi] mauropelucchi commented on issue #2564: Hoodie clean is not deleting old files

2021-09-30 Thread GitBox
mauropelucchi commented on issue #2564: URL: https://github.com/apache/hudi/issues/2564#issuecomment-931358701 > @mauropelucchi : curious as to your table type choice. I see you are setting max delta commits to compact to 1. So, you might as well choose COW to easier operability.

[jira] [Updated] (HUDI-2472) Tests failure follow up when metadata is enabled by default

2021-09-30 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2472: -- Description: We plan to enable metadata by default. but there are some tests that fail

[GitHub] [hudi] nsivabalan edited a comment on issue #3603: [SUPPORT] delta streamer Failed to archive commits

2021-09-30 Thread GitBox
nsivabalan edited a comment on issue #3603: URL: https://github.com/apache/hudi/issues/3603#issuecomment-931267143 I do see from

[GitHub] [hudi] nsivabalan commented on issue #3603: [SUPPORT] delta streamer Failed to archive commits

2021-09-30 Thread GitBox
nsivabalan commented on issue #3603: URL: https://github.com/apache/hudi/issues/3603#issuecomment-931267143 I do see from code that we do add entry to extra metadata as below ``` HashMap checkpointCommitMetadata = new HashMap<>(); checkpointCommitMetadata.put(CHECKPOINT_KEY,

[GitHub] [hudi] nsivabalan commented on issue #3603: [SUPPORT] delta streamer Failed to archive commits

2021-09-30 Thread GitBox
nsivabalan commented on issue #3603: URL: https://github.com/apache/hudi/issues/3603#issuecomment-931261175 @fengjian428 : looks like its related to null in extra metadata. would you mind trying out the patch given by Sagar

[GitHub] [hudi] nsivabalan commented on a change in pull request #3727: [HUDI-2497] Refactor clean, restore, and compaction actions in hudi-client module

2021-09-30 Thread GitBox
nsivabalan commented on a change in pull request #3727: URL: https://github.com/apache/hudi/pull/3727#discussion_r719331449 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java ## @@ -0,0 +1,178 @@ +/* + *

[GitHub] [hudi] nsivabalan commented on a change in pull request #3734: [HUDI-2497] Refactor clean and restore actions in hudi-client module

2021-09-30 Thread GitBox
nsivabalan commented on a change in pull request #3734: URL: https://github.com/apache/hudi/pull/3734#discussion_r719326281 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java ## @@ -86,6 +90,17 @@ public

[GitHub] [hudi] absognety commented on issue #1867: [SUPPORT] hudi is incurring emrfs eTag inconsistency issue with s3 and emrfs consistent view

2021-09-30 Thread GitBox
absognety commented on issue #1867: URL: https://github.com/apache/hudi/issues/1867#issuecomment-931199668 @tsolanki95 what resolved this issue, I am facing the same issue when reading data written in hudi format from S3 -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive

2021-09-30 Thread GitBox
xiarixiaoyao commented on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-931167837 @danny0405 @leesf already update the code。 could you pls review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hudi] codope commented on issue #3735: [SUPPORT] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column

2021-09-30 Thread GitBox
codope commented on issue #3735: URL: https://github.com/apache/hudi/issues/3735#issuecomment-931133022 I think this is a bug. What's happening here is that the [overwrite

[jira] [Created] (HUDI-2509) OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column

2021-09-30 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-2509: - Summary: OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column Key: HUDI-2509 URL: https://issues.apache.org/jira/browse/HUDI-2509

[GitHub] [hudi] tangyoupeng commented on a change in pull request #3736: Add jfs support doc for hudi

2021-09-30 Thread GitBox
tangyoupeng commented on a change in pull request #3736: URL: https://github.com/apache/hudi/pull/3736#discussion_r719227177 ## File path: website/docs/jfs_hoodie.md ## @@ -0,0 +1,90 @@ +--- +title: JuiceFS keywords: [ hudi, hive, jfs, spark, flink] +summary: On this page, we

[jira] [Created] (HUDI-2508) Build GA for the dependeny diff check workflow

2021-09-30 Thread vinoyang (Jira)
vinoyang created HUDI-2508: -- Summary: Build GA for the dependeny diff check workflow Key: HUDI-2508 URL: https://issues.apache.org/jira/browse/HUDI-2508 Project: Apache Hudi Issue Type: Sub-task

[jira] [Assigned] (HUDI-2508) Build GA for the dependeny diff check workflow

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2508: -- Assignee: vinoyang > Build GA for the dependeny diff check workflow >

[jira] [Assigned] (HUDI-2507) Generate more dependency list file for other bundles

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2507: -- Assignee: vinoyang > Generate more dependency list file for other bundles >

[jira] [Created] (HUDI-2507) Generate more dependency list file for other bundles

2021-09-30 Thread vinoyang (Jira)
vinoyang created HUDI-2507: -- Summary: Generate more dependency list file for other bundles Key: HUDI-2507 URL: https://issues.apache.org/jira/browse/HUDI-2507 Project: Apache Hudi Issue Type:

[jira] [Assigned] (HUDI-2506) Hudi dependency governance

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2506: -- Assignee: vinoyang > Hudi dependency governance > -- > > Key:

[jira] [Closed] (HUDI-2440) Add dependency change diff script for dependency governace

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2440. -- Resolution: Done > Add dependency change diff script for dependency governace >

[jira] [Created] (HUDI-2506) Hudi dependency governance

2021-09-30 Thread vinoyang (Jira)
vinoyang created HUDI-2506: -- Summary: Hudi dependency governance Key: HUDI-2506 URL: https://issues.apache.org/jira/browse/HUDI-2506 Project: Apache Hudi Issue Type: Task Components:

[jira] [Reopened] (HUDI-2440) Add dependency change diff script for dependency governace

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reopened HUDI-2440: > Add dependency change diff script for dependency governace >

[jira] [Updated] (HUDI-2440) Add dependency change diff script for dependency governace

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2440: --- Parent: HUDI-2506 Issue Type: Sub-task (was: Improvement) > Add dependency change diff script for

[GitHub] [hudi] hudi-bot edited a comment on pull request #3738: [MINOR] Fix typo,'properites' corrected to 'properties'

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3738: URL: https://github.com/apache/hudi/pull/3738#issuecomment-931010144 ## CI report: * 4a58d8dd619b830e735c33d1f9c40b1359b50812 Azure:

[jira] [Closed] (HUDI-2440) Add dependency change diff script for dependency governace

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2440. -- Resolution: Implemented 47ed91799943271f219419cf209793a98b3f09b5 > Add dependency change diff script for

[jira] [Updated] (HUDI-2440) Add dependency change diff script for dependency governace

2021-09-30 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2440: --- Fix Version/s: 0.10.0 > Add dependency change diff script for dependency governace >

[hudi] branch master updated: [HUDI-2440] Add dependency change diff script for dependency governace (#3674)

2021-09-30 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 47ed917 [HUDI-2440] Add dependency change diff

[GitHub] [hudi] yanghua merged pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace

2021-09-30 Thread GitBox
yanghua merged pull request #3674: URL: https://github.com/apache/hudi/pull/3674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace

2021-09-30 Thread GitBox
yanghua commented on a change in pull request #3674: URL: https://github.com/apache/hudi/pull/3674#discussion_r719198647 ## File path: scripts/dependency.sh ## @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive

2021-09-30 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * c14bded2e903e9c55d86fddc77d40effccca5e01 Azure:

[jira] [Updated] (HUDI-2505) [UMBRELLA] Spark DataSource APIs and Spark SQL discrepancies

2021-09-30 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2505: - Labels: sev:critical (was: ) > [UMBRELLA] Spark DataSource APIs and Spark SQL discrepancies >

[jira] [Updated] (HUDI-2495) Difference in behavior between GenericRecord based key gen and Row based key gen

2021-09-30 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2495: - Parent: HUDI-2505 Issue Type: Sub-task (was: Bug) > Difference in behavior between GenericRecord

[jira] [Updated] (HUDI-2390) KeyGenerator discrepancy between DataFrame writer and SQL

2021-09-30 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2390: - Parent: HUDI-2505 Issue Type: Sub-task (was: Improvement) > KeyGenerator discrepancy between

[jira] [Updated] (HUDI-2500) Spark datasource delete not working on Spark SQL created table

2021-09-30 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2500: - Parent: HUDI-2505 Issue Type: Sub-task (was: Bug) > Spark datasource delete not working on Spark

[jira] [Created] (HUDI-2505) [UMBRELLA] Spark DataSource APIs and Spark SQL discrepancies

2021-09-30 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-2505: Summary: [UMBRELLA] Spark DataSource APIs and Spark SQL discrepancies Key: HUDI-2505 URL: https://issues.apache.org/jira/browse/HUDI-2505 Project: Apache Hudi

[GitHub] [hudi] xushiyan commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace

2021-09-30 Thread GitBox
xushiyan commented on a change in pull request #3674: URL: https://github.com/apache/hudi/pull/3674#discussion_r719181196 ## File path: scripts/dependency.sh ## @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[jira] [Updated] (HUDI-2500) Spark datasource delete not working on Spark SQL created table

2021-09-30 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2500: - Description: Original issue [https://github.com/apache/hudi/issues/3670]   Script to re-produce

  1   2   >