[GitHub] [hudi] n3nash commented on issue #2623: org.apache.hudi.exception.HoodieDependentSystemUnavailableException:System HBASE unavailable.

2021-04-07 Thread GitBox
n3nash commented on issue #2623: URL: https://github.com/apache/hudi/issues/2623#issuecomment-815470053 @root18039532923 Let me know if your issue was resolved after backporting that PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hudi] danny0405 closed pull request #2785: [HUDI-1775] Add option for compaction parallelism

2021-04-07 Thread GitBox
danny0405 closed pull request #2785: URL: https://github.com/apache/hudi/pull/2785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] n3nash commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
n3nash commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-815465310 @ztcheck What changes did you make to the `run_sync_too.sh` ? Can you list out the jars you added to the classpath ? It seems like some of the classes should be packaged in the

[GitHub] [hudi] n3nash commented on issue #2692: [SUPPORT] Corrupt Blocks in Google Cloud Storage

2021-04-07 Thread GitBox
n3nash commented on issue #2692: URL: https://github.com/apache/hudi/issues/2692#issuecomment-815463131 @stackfun Can you respond to @vburenin question ? We can try to go from there.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hudi] n3nash edited a comment on issue #2692: [SUPPORT] Corrupt Blocks in Google Cloud Storage

2021-04-07 Thread GitBox
n3nash edited a comment on issue #2692: URL: https://github.com/apache/hudi/issues/2692#issuecomment-815462588 @vburenin Can you please open a JIRA ticket with the details on "huge data losses with hudi 0.5.0 and EMR" ? This seems super critical and I would like to know the issues ASAP,

[GitHub] [hudi] n3nash commented on issue #2692: [SUPPORT] Corrupt Blocks in Google Cloud Storage

2021-04-07 Thread GitBox
n3nash commented on issue #2692: URL: https://github.com/apache/hudi/issues/2692#issuecomment-815462588 @vburenin Can you please open a JIRA ticket with the details on "huge data losses with hudi 0.5.0 and EMR" ? I don't want to pollute this thread. -- This is an automated message from

[GitHub] [hudi] aditiwari01 commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-07 Thread GitBox
aditiwari01 commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-815462565 @n3nash Thanks for the clarificatio. Can we create a jira for the same. I can't pick this right away but would try to conntribute as and when I get time. Meanwhile I will try to

[jira] [Updated] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-04-07 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1711: -- Labels: sev:critical user-support-issues (was: sev:triage user-support-issues) > Avro Schema

[GitHub] [hudi] n3nash closed issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-04-07 Thread GitBox
n3nash closed issue #2705: URL: https://github.com/apache/hudi/issues/2705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact

[GitHub] [hudi] n3nash commented on issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-04-07 Thread GitBox
n3nash commented on issue #2705: URL: https://github.com/apache/hudi/issues/2705#issuecomment-815461755 Closing this issue since this requires a bug fix, please follow the JIRA above for updates/details. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Assigned] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-04-07 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-1711: - Assignee: sivabalan narayanan > Avro Schema Exception with Spark 3.0 in 0.7 >

[hudi] branch master updated: [MINOR] Some unit test code optimize (#2782)

2021-04-07 Thread wangxianghu
This is an automated email from the ASF dual-hosted git repository. wangxianghu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 18459d4 [MINOR] Some unit test code

[GitHub] [hudi] wangxianghu merged pull request #2782: [MINOR] Some unit test code optimize

2021-04-07 Thread GitBox
wangxianghu merged pull request #2782: URL: https://github.com/apache/hudi/pull/2782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] n3nash closed issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-07 Thread GitBox
n3nash closed issue #2707: URL: https://github.com/apache/hudi/issues/2707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact

[GitHub] [hudi] n3nash commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-07 Thread GitBox
n3nash commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-815460958 @ssdong Thanks for opening the PR! Closing this issue now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] n3nash commented on pull request #2783: [DOCS]Add docs for 0.8.0 release

2021-04-07 Thread GitBox
n3nash commented on pull request #2783: URL: https://github.com/apache/hudi/pull/2783#issuecomment-815457616 @garyli1019 The CI is failing, can you take a look ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] n3nash commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-07 Thread GitBox
n3nash commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-815456351 @aditiwari01 I think you mentioned 2 issues here 1. Record level TTL -> We don't have such a feature in Hudi. Like others have pointed out, using the

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (151b9d4) into

[GitHub] [hudi] yanghua commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-07 Thread GitBox
yanghua commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-815419885 > @wangxianghu: It's OK now. Thanks for your patience, I will do a final check soon. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] yanghua commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer

2021-04-07 Thread GitBox
yanghua commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-815417193 > @yanghua - I don't see the unit tests for the existing transformers except for two functions, I don't have time now to write unit tests, can I handle it in a separate pull

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (46516da) into

[jira] [Created] (HUDI-1776) Support AlterCommand For Hoodie

2021-04-07 Thread pengzhiwei (Jira)
pengzhiwei created HUDI-1776: Summary: Support AlterCommand For Hoodie Key: HUDI-1776 URL: https://issues.apache.org/jira/browse/HUDI-1776 Project: Apache Hudi Issue Type: Sub-task

[GitHub] [hudi] ssdong commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
ssdong commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815413234 @satishkotha Thank you for your review! I’ll take a look when I get back. Currently on a day trip.  Basically, I wanna stop the abuse of `REQUESTED` here, at least for the

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609231425 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -72,9 +76,14 @@ public static

[GitHub] [hudi] susudong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
susudong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609229811 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -72,9 +76,14 @@ public static

[GitHub] [hudi] lw309637554 commented on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
lw309637554 commented on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-815405833 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [hudi] lw309637554 commented on pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on pull request #2773: URL: https://github.com/apache/hudi/pull/2773#issuecomment-815405634 @jintaoguan add some minor comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609227257 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609227046 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609224532 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] codecov-io commented on pull request #2785: [HUDI-1775] Add option for compaction parallelism

2021-04-07 Thread GitBox
codecov-io commented on pull request #2785: URL: https://github.com/apache/hudi/pull/2785#issuecomment-815400787 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2785?src=pr=h1) Report > Merging [#2785](https://codecov.io/gh/apache/hudi/pull/2785?src=pr=desc) (4fca1f0) into

[jira] [Updated] (HUDI-1775) Add option for compaction parallelism

2021-04-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1775: - Labels: pull-request-available (was: ) > Add option for compaction parallelism >

[GitHub] [hudi] danny0405 opened a new pull request #2785: [HUDI-1775] Add option for compaction parallelism

2021-04-07 Thread GitBox
danny0405 opened a new pull request #2785: URL: https://github.com/apache/hudi/pull/2785 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609215373 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[jira] [Updated] (HUDI-1775) Add option for compaction parallelism

2021-04-07 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1775: - Issue Type: Task (was: New Feature) > Add option for compaction parallelism >

[jira] [Created] (HUDI-1775) Add option for compaction parallelism

2021-04-07 Thread Danny Chen (Jira)
Danny Chen created HUDI-1775: Summary: Add option for compaction parallelism Key: HUDI-1775 URL: https://issues.apache.org/jira/browse/HUDI-1775 Project: Apache Hudi Issue Type: New Feature

[jira] [Commented] (HUDI-1674) add partition level delete DOC or example

2021-04-07 Thread liwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316820#comment-17316820 ] liwei commented on HUDI-1674: - [~shivnarayan] spark datasource do not have the delete partition API. It need

[GitHub] [hudi] zherenyu831 commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
zherenyu831 commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609200500 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java ## @@ -245,7 +245,7 @@ public final void

[GitHub] [hudi] zherenyu831 commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
zherenyu831 commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609192912 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java ## @@ -245,7 +245,7 @@ public final void

[GitHub] [hudi] zherenyu831 commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
zherenyu831 commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609186934 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -105,14 +114,15 @@ public

[GitHub] [hudi] codecov-io edited a comment on pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2773: URL: https://github.com/apache/hudi/pull/2773#issuecomment-813928206 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=h1) Report > Merging [#2773](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=desc) (582e348) into

[GitHub] [hudi] hddong commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-07 Thread GitBox
hddong commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-815376490 @wangxianghu: had upgrade to `2.13.3` and fix the warning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Resolved] (HUDI-1750) Fail to load user's class if user move hudi-spark-bundle_2.11-0.7.0.jar into spark classpath

2021-04-07 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1750. --- Resolution: Fixed > Fail to load user's class if user move hudi-spark-bundle_2.11-0.7.0.jar into > spark classpath >

[jira] [Resolved] (HUDI-1751) DeltaStream print many unnecessary warn log because of passing hoodie config to kafka consumer

2021-04-07 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1751. --- Resolution: Fixed > DeltaStream print many unnecessary warn log because of passing hoodie config > to kafka consumer

[jira] [Resolved] (HUDI-1749) Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-07 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1749. --- Resolution: Fixed > Clean/Compaction/Rollback command maybe never exit when operation fail >

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609161570 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[GitHub] [hudi] nsivabalan commented on issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-07 Thread GitBox
nsivabalan commented on issue #2770: URL: https://github.com/apache/hudi/issues/2770#issuecomment-815315164 Sorry, what feature you are looking for. can you please clarify. hudi automatically deletes those records which has "_hoodie_is_deleted" set to true. in other words, if you have a

[GitHub] [hudi] rubenssoto commented on issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-07 Thread GitBox
rubenssoto commented on issue #2770: URL: https://github.com/apache/hudi/issues/2770#issuecomment-815296739 @nsivabalan I think the error in on my side. I didn't filter the deleted records on the first batch, it could be a great feature to Hudi in the future. -- This is an

[GitHub] [hudi] stackfun commented on issue #2771: [SUPPORT] Log files are not compacted

2021-04-07 Thread GitBox
stackfun commented on issue #2771: URL: https://github.com/apache/hudi/issues/2771#issuecomment-815292886 Setting the "hoodie.compaction.target.io" config worked like a charm. Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] stackfun closed issue #2771: [SUPPORT] Log files are not compacted

2021-04-07 Thread GitBox
stackfun closed issue #2771: URL: https://github.com/apache/hudi/issues/2771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] satishkotha commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
satishkotha commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815291682 @ssdong thanks for bringing this up and contributing. I added some comments, please take a look. Also, looks like there are some CI failures. Please fix those as well. --

[GitHub] [hudi] satishkotha commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
satishkotha commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609094597 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -105,14 +114,15 @@ public

[GitHub] [hudi] kvallala commented on issue #2528: [SUPPORT] Spark read hudi data from hive (metastore)

2021-04-07 Thread GitBox
kvallala commented on issue #2528: URL: https://github.com/apache/hudi/issues/2528#issuecomment-815182803 We are having the same issue. It works with `spark.sql.hive.convertMetastoreParquet=false` when querying Hudi table from spark session, but see duplicates when querying through

[GitHub] [hudi] ze-engineering-code-challenge commented on pull request #2665: [HUDI-1160] Support update partial fields for CoW table

2021-04-07 Thread GitBox
ze-engineering-code-challenge commented on pull request #2665: URL: https://github.com/apache/hudi/pull/2665#issuecomment-815168500 Hello @liujinhui1994 Should I enable any option to work? Im trying to do an upsert in a Hudi table with 0.8.0 version and didn't work :(

[GitHub] [hudi] vingov commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer

2021-04-07 Thread GitBox
vingov commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-815167427 @yanghua - I don't see the unit tests for the existing transformers except for two functions, I don't have time now to write unit tests, can I handle it in a separate pull request

[GitHub] [hudi] codecov-io commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
codecov-io commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815166346 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=h1) Report > Merging [#2784](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=desc) (5572b9f) into

[GitHub] [hudi] ssdong commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-07 Thread GitBox
ssdong commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-815156685 Hi @satishkotha @jsbali ! I've created the pull request for this issue. Had observed more when going down the road and I've tried my best to clarify them and hopefully had written a

[GitHub] [hudi] ssdong opened a new pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
ssdong opened a new pull request #2784: URL: https://github.com/apache/hudi/pull/2784 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] nsivabalan commented on a change in pull request #2783: [DOCS]Add docs for 0.8.0 release

2021-04-07 Thread GitBox
nsivabalan commented on a change in pull request #2783: URL: https://github.com/apache/hudi/pull/2783#discussion_r608886318 ## File path: docs/_docs/0.8.0/1_1_spark_quick_start_guide.md ## @@ -0,0 +1,530 @@ +--- +version: 0.8.0 +title: "Quick-Start Guide" +permalink:

[jira] [Updated] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds

2021-04-07 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Susu Dong updated HUDI-1740: Description: insert_overwrite_table and insert_overwrite first replacecommit has empty

[jira] [Assigned] (HUDI-1739) insert_overwrite_table and insert_overwrite create empty replacecommit.requested file which breaks archival

2021-04-07 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Susu Dong reassigned HUDI-1739: --- Assignee: Susu Dong > insert_overwrite_table and insert_overwrite create empty >

[jira] [Assigned] (HUDI-1774) Add support or delete_partition with spark ds

2021-04-07 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1774: - Assignee: liwei > Add support or delete_partition with spark ds >

[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-07 Thread GitBox
nsivabalan commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-815015923 @lw309637554 @satishkotha : fyi we are yet to add spark ds support for this "delete_partition" operation. -- This is an automated message from the Apache Git Service. To respond

[jira] [Commented] (HUDI-1674) add partition level delete DOC or example

2021-04-07 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316432#comment-17316432 ] sivabalan narayanan commented on HUDI-1674: --- [~309637554]: we are yet to add this operation to

[GitHub] [hudi] nsivabalan commented on issue #2399: [SUPPORT] Hudi deletes not being properly commited

2021-04-07 Thread GitBox
nsivabalan commented on issue #2399: URL: https://github.com/apache/hudi/issues/2399#issuecomment-815011207 btw, we have filed a feature request to support reusing existing hudi configs https://issues.apache.org/jira/browse/HUDI-1640 -- This is an automated message from the Apache Git

[jira] [Assigned] (HUDI-1760) Incorrect Documentation for HoodieWriteConfigs

2021-04-07 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1760: - Assignee: Gary Li > Incorrect Documentation for HoodieWriteConfigs >

[GitHub] [hudi] BenjMaq commented on issue #2399: [SUPPORT] Hudi deletes not being properly commited

2021-04-07 Thread GitBox
BenjMaq commented on issue #2399: URL: https://github.com/apache/hudi/issues/2399#issuecomment-814990246 Just want to add that I faced the same issue. For me, the problem was related to the option `.option(DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY,

[jira] [Updated] (HUDI-73) Support vanilla Avro Kafka Source in HoodieDeltaStreamer

2021-04-07 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-73?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-73: Fix Version/s: (was: 0.8.0) 0.9.0 > Support vanilla Avro Kafka Source in HoodieDeltaStreamer >

[jira] [Created] (HUDI-1774) Add support or delete_partition with spark ds

2021-04-07 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-1774: - Summary: Add support or delete_partition with spark ds Key: HUDI-1774 URL: https://issues.apache.org/jira/browse/HUDI-1774 Project: Apache Hudi

[GitHub] [hudi] garyli1019 opened a new pull request #2783: [DOCS]Add docs for 0.8.0 release

2021-04-07 Thread GitBox
garyli1019 opened a new pull request #2783: URL: https://github.com/apache/hudi/pull/2783 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] li36909 commented on pull request #2754: [HUDI-1751] Remove irrelevant properties from passing to kafkaConsumer which in turn prints lot of warn logs

2021-04-07 Thread GitBox
li36909 commented on pull request #2754: URL: https://github.com/apache/hudi/pull/2754#issuecomment-814932095 @n3nash @pratyakshsharma thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] li36909 commented on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-07 Thread GitBox
li36909 commented on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-814931307 @n3nash thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] li36909 commented on pull request #2753: [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath

2021-04-07 Thread GitBox
li36909 commented on pull request #2753: URL: https://github.com/apache/hudi/pull/2753#issuecomment-814930600 @nsivabalan thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (647e322) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-813008111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [hudi] nsivabalan commented on a change in pull request #2769: [HUDI-1762] Added HiveStylePartitionExtractor to support Hive style partitions

2021-04-07 Thread GitBox
nsivabalan commented on a change in pull request #2769: URL: https://github.com/apache/hudi/pull/2769#discussion_r608577463 ## File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveStylePartitionValueExtractor.java ## @@ -0,0 +1,42 @@ +/* + * Licensed to

[GitHub] [hudi] nsivabalan commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-04-07 Thread GitBox
nsivabalan commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-814844457 @xushiyan : I see you have disabled test in TestHoodieCombineHiveInputFormat. can you help explain the reason. -- This is an automated message from the Apache Git Service.

[GitHub] [hudi] codecov-io commented on pull request #2782: [MINOR] ut code optimize

2021-04-07 Thread GitBox
codecov-io commented on pull request #2782: URL: https://github.com/apache/hudi/pull/2782#issuecomment-814831752 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2782?src=pr=h1) Report > Merging [#2782](https://codecov.io/gh/apache/hudi/pull/2782?src=pr=desc) (ca38e68) into

[jira] [Closed] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1773. -- Resolution: Done 3a926aacf6552fc06005db4a7880a233db904330 > HoodieFileGroup code optimize >

[jira] [Updated] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1773: --- Fix Version/s: 0.9.0 > HoodieFileGroup code optimize > - > > Key:

[hudi] branch master updated: [HUDI-1773] HoodieFileGroup code optimize (#2781)

2021-04-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3a926aa [HUDI-1773] HoodieFileGroup code

[GitHub] [hudi] yanghua merged pull request #2781: [HUDI-1773] HoodieFileGroup code optimize

2021-04-07 Thread GitBox
yanghua merged pull request #2781: URL: https://github.com/apache/hudi/pull/2781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[jira] [Closed] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1772. -- Resolution: Fixed f4f9dd9d83a6a852c0e733802c6c49747cde5531 > HoodieFileGroupId compareTo logical error(fileId

[hudi] branch master updated (dadd081 -> f4f9dd9)

2021-04-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from dadd081 [HUDI-1751] DeltaStreamer print many unnecessary warn log (#2754) add f4f9dd9 [HUDI-1772]

[GitHub] [hudi] yanghua merged pull request #2780: [HUDI-1772] HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread GitBox
yanghua merged pull request #2780: URL: https://github.com/apache/hudi/pull/2780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script, such like '

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Such like '

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-800020976 My environment is k8s. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Just like '

[GitHub] [hudi] ztcheck commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Just like `HUDI_HIVE_UBER_JAR=`ls -c

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608501941 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[GitHub] [hudi] ztcheck removed a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck removed a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814771640 > @ztcheck Are you using the `hudi-hive-sync-bundle` to execute the script `run_sync_tool .sh` ? > > Can you please provide what `HUDI_HIVE_UBER_JAR` is in your env ?

[GitHub] [hudi] ztcheck commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814771640 > @ztcheck Are you using the `hudi-hive-sync-bundle` to execute the script `run_sync_tool .sh` ? > > Can you please provide what `HUDI_HIVE_UBER_JAR` is in your env ?

[GitHub] [hudi] simon824 opened a new pull request #2782: [MINOR] Optimized code

2021-04-07 Thread GitBox
simon824 opened a new pull request #2782: URL: https://github.com/apache/hudi/pull/2782 ## What is the purpose of the pull request Optimized some code ## Brief change log ## Verify this pull request ## Committer checklist - [ ] Has a corresponding

[GitHub] [hudi] codecov-io edited a comment on pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2773: URL: https://github.com/apache/hudi/pull/2773#issuecomment-813928206 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=h1) Report > Merging [#2773](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=desc) (fc2f340) into

[GitHub] [hudi] hddong commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-07 Thread GitBox
hddong commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-814734415 @wangxianghu: It's OK now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608436934 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -49,7 +51,48 @@ object AvroConversionUtils {

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608434413 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala ## @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608433997 ## File path: hudi-utilities/src/test/resources/delta-streamer-config/source-jdbc.avsc ## @@ -26,34 +26,42 @@ }, { "name": "TIMESTAMP", -

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608427661 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -156,15 +155,15 @@ private int

  1   2   >