[GitHub] [hudi] codecov-io commented on pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-02 Thread GitBox
codecov-io commented on pull request #2761: URL: https://github.com/apache/hudi/pull/2761#issuecomment-812815750 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2761?src=pr=h1) Report > Merging [#2761](https://codecov.io/gh/apache/hudi/pull/2761?src=pr=desc) (f404051) into

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-02 Thread GitBox
xiarixiaoyao commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r606622137 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -302,6 +302,10 @@ case class

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-02 Thread GitBox
xiarixiaoyao commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r606622137 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -302,6 +302,10 @@ case class

[GitHub] [hudi] xiarixiaoyao commented on pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-02 Thread GitBox
xiarixiaoyao commented on pull request #2761: URL: https://github.com/apache/hudi/pull/2761#issuecomment-812815163 @vinothchandar , could you help me to review this pr, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[jira] [Updated] (HUDI-1676) Support SQL with spark3

2021-04-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1676: - Labels: pull-request-available (was: ) > Support SQL with spark3 > --- > >

[GitHub] [hudi] xiarixiaoyao opened a new pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-02 Thread GitBox
xiarixiaoyao opened a new pull request #2761: URL: https://github.com/apache/hudi/pull/2761 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] codecov-io edited a comment on pull request #2757: [HUDI-1757] Assigns the buckets by record key for Flink writer

2021-04-02 Thread GitBox
codecov-io edited a comment on pull request #2757: URL: https://github.com/apache/hudi/pull/2757#issuecomment-812247500 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2757?src=pr=h1) Report > Merging [#2757](https://codecov.io/gh/apache/hudi/pull/2757?src=pr=desc) (9372602) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2757: [HUDI-1757] Assigns the buckets by record key for Flink writer

2021-04-02 Thread GitBox
codecov-io edited a comment on pull request #2757: URL: https://github.com/apache/hudi/pull/2757#issuecomment-812247500 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2757?src=pr=h1) Report > Merging [#2757](https://codecov.io/gh/apache/hudi/pull/2757?src=pr=desc) (8dd3a6f) into

[GitHub] [hudi] nsivabalan commented on issue #2338: [SUPPORT] MOR table found duplicate and process so slowly

2021-04-02 Thread GitBox
nsivabalan commented on issue #2338: URL: https://github.com/apache/hudi/issues/2338#issuecomment-812713376 Closing due to inactivity. but feel free to reopen to create a new ticket. would be happy to assist you. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] nsivabalan closed issue #2338: [SUPPORT] MOR table found duplicate and process so slowly

2021-04-02 Thread GitBox
nsivabalan closed issue #2338: URL: https://github.com/apache/hudi/issues/2338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] nsivabalan commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?

2021-04-02 Thread GitBox
nsivabalan commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-812713158 CC @n3nash -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] nsivabalan commented on issue #2586: [SUPPORT] - How to guarantee snapshot isolation when reading Hudi tables in S3?

2021-04-02 Thread GitBox
nsivabalan commented on issue #2586: URL: https://github.com/apache/hudi/issues/2586#issuecomment-812712095 Closing this for now. please feel free to reopen or open a new ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan closed issue #2586: [SUPPORT] - How to guarantee snapshot isolation when reading Hudi tables in S3?

2021-04-02 Thread GitBox
nsivabalan closed issue #2586: URL: https://github.com/apache/hudi/issues/2586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] nsivabalan closed issue #2675: [SUPPORT] Unable to query MOR table after schema evolution

2021-04-02 Thread GitBox
nsivabalan closed issue #2675: URL: https://github.com/apache/hudi/issues/2675 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] nsivabalan commented on issue #2675: [SUPPORT] Unable to query MOR table after schema evolution

2021-04-02 Thread GitBox
nsivabalan commented on issue #2675: URL: https://github.com/apache/hudi/issues/2675#issuecomment-812710497 Closing this as we have a tracking jira. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Updated] (HUDI-79) how to query hoodie tables with 'Hive on Spark' engine?

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-79: Labels: sev:normal user-support-issues (was: sev:critical user-support-issues) > how to

[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

2021-04-02 Thread GitBox
nsivabalan commented on issue #2756: URL: https://github.com/apache/hudi/issues/2756#issuecomment-812703790 CC @n3nash -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[jira] [Commented] (HUDI-1036) HoodieCombineHiveInputFormat not picking up HoodieRealtimeFileSplit

2021-04-02 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314046#comment-17314046 ] Nishith Agarwal commented on HUDI-1036: --- [~shivnarayan] Thanks for the reminder, I will take a look

[jira] [Updated] (HUDI-1036) HoodieCombineHiveInputFormat not picking up HoodieRealtimeFileSplit

2021-04-02 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1036: -- Labels: sev:normal user-support-issues (was: sev:critical user-support-issues) >

[GitHub] [hudi] jkdll opened a new issue #2760: [SUPPORT] Possibly Incorrect Documentation

2021-04-02 Thread GitBox
jkdll opened a new issue #2760: URL: https://github.com/apache/hudi/issues/2760 Hi, I am using the HudiWriteClient library and have been following the documentation at [this link](https://hudi.apache.org/docs/configurations.html#writeclient-configs) to instantiate the

[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-02 Thread GitBox
codecov-io edited a comment on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr=h1) Report > Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr=desc) (8052abf) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-02 Thread GitBox
codecov-io edited a comment on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [hudi] ssdong edited a comment on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-02 Thread GitBox
ssdong edited a comment on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-811619487 @jsbali To give out extra insights and details, as @zherenyu831 has posted in the beginning: ``` [20210323080718__replacecommit__COMPLETED]: size : 0

[jira] [Commented] (HUDI-1453) Throw Exception when input data schema is not equal to the hoodie table schema

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313858#comment-17313858 ] sivabalan narayanan commented on HUDI-1453: --- double to int is not backwards compatible schema

[jira] [Resolved] (HUDI-1453) Throw Exception when input data schema is not equal to the hoodie table schema

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1453. --- Resolution: Invalid > Throw Exception when input data schema is not equal to the

[jira] [Updated] (HUDI-1453) Throw Exception when input data schema is not equal to the hoodie table schema

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1453: -- Status: Open (was: New) > Throw Exception when input data schema is not equal to the

[GitHub] [hudi] aditiwari01 edited a comment on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

2021-04-02 Thread GitBox
aditiwari01 edited a comment on issue #2756: URL: https://github.com/apache/hudi/issues/2756#issuecomment-812516535 I think I couldn't explain myself. I am using DefaultHoodieRecordPayload only. I have attached sample command regardinng same. The issue is not with

[GitHub] [hudi] aditiwari01 commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

2021-04-02 Thread GitBox
aditiwari01 commented on issue #2756: URL: https://github.com/apache/hudi/issues/2756#issuecomment-812516535 I think I couldn't explain myself. I am using DefaultHoodieRecordPayload only. I am attached sample command regardinng same. The issue is not with "combineAndGetUpdateValue",

[GitHub] [hudi] nsivabalan commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-04-02 Thread GitBox
nsivabalan commented on pull request #2334: URL: https://github.com/apache/hudi/pull/2334#issuecomment-812516130 Yeah, I did verify by enabling schema compatability check. it will fail if we try to evolve a field from double to int. ``` scala> dfFromData5.write.format("hudi").

[jira] [Commented] (HUDI-1716) rt view w/ MOR tables fails after schema evolution

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313842#comment-17313842 ] sivabalan narayanan commented on HUDI-1716: --- related issue: HUDI-774 > rt view w/ MOR tables

[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313841#comment-17313841 ] sivabalan narayanan commented on HUDI-774: -- related issue : HUDI-1716   > Spark to Avro

[jira] [Comment Edited] (HUDI-874) Schema evolution does not work with AWS Glue catalog

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313840#comment-17313840 ] sivabalan narayanan edited comment on HUDI-874 at 4/2/21, 12:10 PM:

[jira] [Commented] (HUDI-874) Schema evolution does not work with AWS Glue catalog

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313840#comment-17313840 ] sivabalan narayanan commented on HUDI-874: -- [~uditme]: is someone from AWS looking into this.  >

[jira] [Commented] (HUDI-1036) HoodieCombineHiveInputFormat not picking up HoodieRealtimeFileSplit

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313837#comment-17313837 ] sivabalan narayanan commented on HUDI-1036: --- [~nishith29]: this has been lying around for some

[GitHub] [hudi] li36909 commented on a change in pull request #2754: [HUDI-1751] DeltaStreamer print many unnecessary warn log

2021-04-02 Thread GitBox
li36909 commented on a change in pull request #2754: URL: https://github.com/apache/hudi/pull/2754#discussion_r606207765 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java ## @@ -173,9 +173,11 @@ public

[jira] [Commented] (HUDI-1063) Save in Google Cloud Storage not working

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313836#comment-17313836 ] sivabalan narayanan commented on HUDI-1063: --- [~WaterKnight]: Were you able to resolve your

[jira] [Resolved] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1288. --- Fix Version/s: 0.6.0 Resolution: Fixed > DeltaSync:writeToSink fails with

[GitHub] [hudi] li36909 commented on a change in pull request #2754: [HUDI-1751] DeltaStreamer print many unnecessary warn log

2021-04-02 Thread GitBox
li36909 commented on a change in pull request #2754: URL: https://github.com/apache/hudi/pull/2754#discussion_r606207354 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java ## @@ -173,9 +173,11 @@ public

[jira] [Commented] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313835#comment-17313835 ] sivabalan narayanan commented on HUDI-1288: --- Closing out this Jira as we don't have any plans to

[jira] [Updated] (HUDI-1288) DeltaSync:writeToSink fails with Unknown datum type org.apache.avro.JsonProperties$Null

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1288: -- Status: Open (was: New) > DeltaSync:writeToSink fails with Unknown datum type >

[jira] [Commented] (HUDI-1528) hudi-sync-tools error

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313834#comment-17313834 ] sivabalan narayanan commented on HUDI-1528: --- [~Trevorzhang]: can you update the Jira on how you

[jira] [Updated] (HUDI-1528) hudi-sync-tools error

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1528: -- Labels: pull-request-available user-support-issues (was: pull-request-available

[jira] [Updated] (HUDI-1528) hudi-sync-tools error

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1528: -- Labels: user-support-issues (was: pull-request-available user-support-issues) >

[jira] [Updated] (HUDI-1751) DeltaStream print many unnecessary warn log because of passing hoodie config to kafka consumer

2021-04-02 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz updated HUDI-1751: -- Summary: DeltaStream print many unnecessary warn log because of passing hoodie config to kafka consumer (was:

[GitHub] [hudi] nsivabalan edited a comment on pull request #2449: [HUDI-1528] hudi-sync-tools supports synchronization to remote hive

2021-04-02 Thread GitBox
nsivabalan edited a comment on pull request #2449: URL: https://github.com/apache/hudi/pull/2449#issuecomment-779341027 @Trevor-zhang : sorry, I didn't suggest to close this out. I am also getting conversant w/ hive sync in general. So, was trying to clarify few things. if I am not

[jira] [Commented] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313832#comment-17313832 ] sivabalan narayanan commented on HUDI-1652: --- [~hainanzhongjian]: can we close the Jira then

[jira] [Commented] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313828#comment-17313828 ] sivabalan narayanan commented on HUDI-1723: --- [~xushiyan]: I don't have much exp on the query

[GitHub] [hudi] li36909 edited a comment on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-02 Thread GitBox
li36909 edited a comment on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-812496806 just run any rollback/compaction command, and make it fail by injection fault, then the command will hang. For example, currently hudi only support rollback to the latest

[GitHub] [hudi] li36909 edited a comment on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-02 Thread GitBox
li36909 edited a comment on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-812496806 just run any rollback/compaction command, and make it fail by injection fault, then the command will hang. For example, currently hudi only support the latest commit, we

[jira] [Resolved] (HUDI-1734) Hive sync script (run_sync_tool.sh) fails w/ ClassNotFoundError : org/apache/log4j/LogManager

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1734. --- Fix Version/s: 0.8.0 Resolution: Invalid > Hive sync script (run_sync_tool.sh)

[GitHub] [hudi] nsivabalan commented on issue #2728: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-02 Thread GitBox
nsivabalan commented on issue #2728: URL: https://github.com/apache/hudi/issues/2728#issuecomment-812497980 thanks for letting us know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Resolved] (HUDI-1745) Hudi compilation fails w/ spark version < 2.4.4 due to usage of unavailable spark api

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1745. --- Resolution: Fixed > Hudi compilation fails w/ spark version < 2.4.4 due to usage of

[jira] [Commented] (HUDI-1745) Hudi compilation fails w/ spark version < 2.4.4 due to usage of unavailable spark api

2021-04-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313827#comment-17313827 ] sivabalan narayanan commented on HUDI-1745: --- We have always been testing w/ spark 2.4.4 and at

[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

2021-04-02 Thread GitBox
nsivabalan commented on issue #2756: URL: https://github.com/apache/hudi/issues/2756#issuecomment-812496813 yes, this is expected. if you are using OverwriteWithLatestAvroPayload as your payload class, combineAndGetUpdateValue does not honor ordering value. And so we added another payload

[GitHub] [hudi] li36909 commented on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-02 Thread GitBox
li36909 commented on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-812496806 just run any rollback/compaction command, and make it fail by injection fault, then the command will hang -- This is an automated message from the Apache Git Service. To respond

[GitHub] [hudi] pengzhiwei2018 commented on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-04-02 Thread GitBox
pengzhiwei2018 commented on pull request #2334: URL: https://github.com/apache/hudi/pull/2334#issuecomment-812496550 > https://gist.github.com/nsivabalan/91f12109e0fe1ca9749ff5290c946778 Hi @nsivabalan , I have take a review for your test code. First you write a "int" to the table,

[GitHub] [hudi] pengzhiwei2018 closed pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-04-02 Thread GitBox
pengzhiwei2018 closed pull request #2334: URL: https://github.com/apache/hudi/pull/2334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[jira] [Updated] (HUDI-1591) Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed table path and partition pruning

2021-04-02 Thread pengzhiwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei updated HUDI-1591: - Summary: Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed

[GitHub] [hudi] hddong commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-02 Thread GitBox
hddong commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-812460441 @yanghua @wangxianghu: had address them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] hddong commented on a change in pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-02 Thread GitBox
hddong commented on a change in pull request #2325: URL: https://github.com/apache/hudi/pull/2325#discussion_r606166490 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CompactionCommand.java ## @@ -175,25 +174,26 @@ public String compactionShowArchived(

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606123928 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/CommitUtils.java ## @@ -59,14 +61,24 @@ public static HoodieCommitMetadata

[GitHub] [hudi] ssdong edited a comment on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-02 Thread GitBox
ssdong edited a comment on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-811619487 @jsbali To give out extra insights and details, as @zherenyu831 has posted in the beginning: ``` [20210323080718__replacecommit__COMPLETED]: size : 0

[hudi] branch release-0.8.0 updated: [MINOR] Update release version to reflect published version 0.8.0

2021-04-02 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a commit to branch release-0.8.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.8.0 by this push: new da65d3c [MINOR] Update release

[GitHub] [hudi] li36909 commented on pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-04-02 Thread GitBox
li36909 commented on pull request #2759: URL: https://github.com/apache/hudi/pull/2759#issuecomment-812403045 cc @nsivabalan could you help to take a look, thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] li36909 commented on pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-04-02 Thread GitBox
li36909 commented on pull request #2759: URL: https://github.com/apache/hudi/pull/2759#issuecomment-812402879 The retry issue is cause by: when close metaStoreClient, or sessionState, or hiveDriver, they will all call 'Hive.closeCurrent()', so both sessionState and hiveDriver should be a

[jira] [Updated] (HUDI-1759) Save one connection retry when hiveSyncTool run with useJdbc=false

2021-04-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1759: - Labels: pull-request-available (was: ) > Save one connection retry when hiveSyncTool run with

[GitHub] [hudi] li36909 opened a new pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-04-02 Thread GitBox
li36909 opened a new pull request #2759: URL: https://github.com/apache/hudi/pull/2759 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Updated] (HUDI-1759) Save one connection retry when hiveSyncTool run with useJdbc=false

2021-04-02 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz updated HUDI-1759: -- Description: when sync metadata to hive with useJdbc=false, there will have two problem: first: if hive server enable

[jira] [Updated] (HUDI-1759) Save one connection retry when hiveSyncTool run with useJdbc=false

2021-04-02 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz updated HUDI-1759: -- Attachment: image-2021-04-02-15-48-42-895.png > Save one connection retry when hiveSyncTool run with useJdbc=false >

[jira] [Created] (HUDI-1759) Save one connection retry when hiveSyncTool run with useJdbc=false

2021-04-02 Thread lrz (Jira)
lrz created HUDI-1759: - Summary: Save one connection retry when hiveSyncTool run with useJdbc=false Key: HUDI-1759 URL: https://issues.apache.org/jira/browse/HUDI-1759 Project: Apache Hudi Issue Type:

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606117534 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialUpdatePayload.java ## @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache

[jira] [Commented] (HUDI-1657) build failed on AArch64, Fedora 33

2021-04-02 Thread shenjinxin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313672#comment-17313672 ] shenjinxin commented on HUDI-1657: -- I also encounter the same problem. My Java is JDK 1.8_281 > build

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606116196 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialUpdatePayload.java ## @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606108825 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/PartialUpdatePayload.java ## @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606108625 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java ## @@ -106,7 +110,7 @@ public

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-02 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Description: [^Screen Shot 2021-04-02 at 12.10.08 AM.zip]   Followed the steps mentioned in

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-02 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Description: !Screen Shot 2021-04-02 at 12.10.08 AM.png!   Followed the steps mentioned in

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-02 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Description: !image (1).png!   Followed the steps mentioned in

[jira] [Created] (HUDI-1758) Flink insert command does not update the record

2021-04-02 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1758: - Summary: Flink insert command does not update the record Key: HUDI-1758 URL: https://issues.apache.org/jira/browse/HUDI-1758 Project: Apache Hudi Issue

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606101834 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java ## @@ -123,6 +127,22 @@ public

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606100489 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java ## @@ -123,6 +127,22 @@ public

[GitHub] [hudi] n3nash commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

2021-04-02 Thread GitBox
n3nash commented on pull request #2751: URL: https://github.com/apache/hudi/pull/2751#issuecomment-812357498 Okay, thanks for the confirmation, I will try to reproduce this issue on my end and get back. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2666: [HUDI-1160] Support update partial fields for CoW table

2021-04-02 Thread GitBox
liujinhui1994 commented on a change in pull request #2666: URL: https://github.com/apache/hudi/pull/2666#discussion_r606098330 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -79,6 +80,7 @@ public static final

[GitHub] [hudi] n3nash commented on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-02 Thread GitBox
n3nash commented on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-812341544 @li36909 Have you run into a scenario where these commands hang ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] li36909 commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

2021-04-02 Thread GitBox
li36909 commented on pull request #2751: URL: https://github.com/apache/hudi/pull/2751#issuecomment-812341340 @n3nash yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] n3nash commented on a change in pull request #2754: [HUDI-1751] DeltaStreamer print many unnecessary warn log

2021-04-02 Thread GitBox
n3nash commented on a change in pull request #2754: URL: https://github.com/apache/hudi/pull/2754#discussion_r606087966 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java ## @@ -173,9 +173,11 @@ public

[hudi] branch master updated (94a5e72 -> e970e1f)

2021-04-02 Thread nagarwal
This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 94a5e72 [HUDI-1737][hudi-client] Code Cleanup: Extract common method in HoodieCreateHandle & FlinkCreateHandle

[GitHub] [hudi] n3nash merged pull request #2758: [HUDI-1696] add apache commons-codec dependency to flink-bundle explicitly

2021-04-02 Thread GitBox
n3nash merged pull request #2758: URL: https://github.com/apache/hudi/pull/2758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] n3nash commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-02 Thread GitBox
n3nash commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-812339166 @kimberlyamandalu Yes, you should be able to switch off your metadata table without any side-effect. Although, if you want to later turn on the metadata table, you will need to delete