[GitHub] [hudi] aditiwari01 commented on issue #2675: [SUPPORT] Unable to query MOR table after schema evolution

2021-03-23 Thread GitBox
aditiwari01 commented on issue #2675: URL: https://github.com/apache/hudi/issues/2675#issuecomment-804668465 > You can add null as default value for your new field if that would work for you. We can not provide defaults from spark. Need to change where hudi registers the avro schema

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue 3.

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599318844 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -79,39 +82,58 @@ class DefaultSource extends

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599319824 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the Apac

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599296787 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the Apac

[GitHub] [hudi] danny0405 closed pull request #2702: [HUDI-1710] Read optimized query type for Flink batch reader

2021-03-23 Thread GitBox
danny0405 closed pull request #2702: URL: https://github.com/apache/hudi/pull/2702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] cdmikechen edited a comment on issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-03-23 Thread GitBox
cdmikechen edited a comment on issue #2705: URL: https://github.com/apache/hudi/issues/2705#issuecomment-804636641 I've found the problem: There is a new configuration named `hoodie.deltastreamer.schemaprovider.spark_avro_post_processor.enable` and it is `true` by default. If I use my

[GitHub] [hudi] Sugamber commented on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber commented on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804748373 Can this use case be achieved using Hudi? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] hddong commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-03-23 Thread GitBox
hddong commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-804749723 @yanghua Can you have a review for this again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [hudi] zherenyu831 opened a new issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 opened a new issue #2707: URL: https://github.com/apache/hudi/issues/2707 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster s

[GitHub] [hudi] hddong commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-03-23 Thread GitBox
hddong commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-804760962 @yanghua @vinothchandar: pls have a review when free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804762627 simple workaround is delete first commit file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue 3.

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804748373 Can this use case be achieved using Hudi as target schema and incremental schema are not same? -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue 3.

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599474180 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the Apac

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804831651 But got new error ``` User class threw exception: org.apache.hudi.exception.HoodieIOException: Could not read commit details from s3://xxx/.hoodie/20210317155538.replacecom

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804849028 Also saw some comment and find below link https://issues.apache.org/jira/browse/HUDI-1518 Seems you are going to use cleaner to delete the file group, but currently, if I use

[GitHub] [hudi] pengzhiwei2018 commented on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-804856226 Hi @umehrot2 , I have updated the code. Main changes: 1、Introduce `FileStatusCache` to `HoodieFileIndex` for caching files across queries. 2、Save partition columns f

[GitHub] [hudi] yanghua merged pull request #2703: [MINOR][DOCUMENT] update README doc for integ test

2021-03-23 Thread GitBox
yanghua merged pull request #2703: URL: https://github.com/apache/hudi/pull/2703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please co

[hudi] branch master updated: [MINOR][DOCUMENT] Update README doc for integ test (#2703)

2021-03-23 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 0e6909d [MINOR][DOCUMENT] Update README doc for

[GitHub] [hudi] yanghua commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-03-23 Thread GitBox
yanghua commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-804864588 > @yanghua Can you have a review for this again? OK, sorry for the late reply. Will review soon. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] bvaradar commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
bvaradar commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-80482 @satishkotha : Can you take a look at this ? (cc @n3nash ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] vinothchandar commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-03-23 Thread GitBox
vinothchandar commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-804911990 @kimberlyamandalu do you have the full stack trace? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [hudi] vinothchandar commented on pull request #2697: [HUDI-1211] clean up spark session for each test of FunctionalTestHar…

2021-03-23 Thread GitBox
vinothchandar commented on pull request #2697: URL: https://github.com/apache/hudi/pull/2697#issuecomment-804913961 @kwondw this is done intentionally to speed up tests. cc @xushiyan can you please comment on this ? -- This is an automated message from the Apache Git Service. To

[jira] [Assigned] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1650: - Assignee: Volodymyr Burenin > Improve AvroKafkaSource that it can take into account the schema evolution

[jira] [Updated] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1650: -- Fix Version/s: 0.8.0 > Improve AvroKafkaSource that it can take into account the schema evolution >

[jira] [Resolved] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1650. --- Resolution: Fixed > Improve AvroKafkaSource that it can take into account the schema evolution > -

[jira] [Assigned] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1650: - Assignee: sivabalan narayanan (was: Volodymyr Burenin) > Improve AvroKafkaSource that it can take into a

[jira] [Resolved] (HUDI-1709) Improve lock config naming convention and add hive metastore uri config

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1709. --- Resolution: Fixed > Improve lock config naming convention and add hive metastore uri config >

[jira] [Updated] (HUDI-1709) Improve lock config naming convention and add hive metastore uri config

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1709: -- Fix Version/s: 0.8.0 > Improve lock config naming convention and add hive metastore uri config > ---

[jira] [Resolved] (HUDI-1653) Add support for composite keys in NonpartitionedKeyGenerator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1653. --- Resolution: Fixed > Add support for composite keys in NonpartitionedKeyGenerator > ---

[jira] [Closed] (HUDI-1653) Add support for composite keys in NonpartitionedKeyGenerator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1653. - > Add support for composite keys in NonpartitionedKeyGenerator > -

[jira] [Commented] (HUDI-1653) Add support for composite keys in NonpartitionedKeyGenerator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307084#comment-17307084 ] Gary Li commented on HUDI-1653: --- [~Guan] thanks for your contribution. I can't assign this t

[jira] [Resolved] (HUDI-1688) hudi write should uncache rdd, when the write operation is finnished

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1688. --- Resolution: Fixed > hudi write should uncache rdd, when the write operation is finnished > ---

[jira] [Closed] (HUDI-1688) hudi write should uncache rdd, when the write operation is finnished

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1688. - > hudi write should uncache rdd, when the write operation is finnished > -

[jira] [Commented] (HUDI-1688) hudi write should uncache rdd, when the write operation is finnished

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307089#comment-17307089 ] Gary Li commented on HUDI-1688: --- [~xiaotaotao] thanks for your contribution. I can't assign

[jira] [Updated] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-845: - Issue Type: New Feature (was: Improvement) > Allow parallel writing to Hudi tables using Locking service > ---

[jira] [Created] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-03-23 Thread Balaji Varadarajan (Jira)
Balaji Varadarajan created HUDI-1711: Summary: Avro Schema Exception with Spark 3.0 in 0.7 Key: HUDI-1711 URL: https://issues.apache.org/jira/browse/HUDI-1711 Project: Apache Hudi Issue T

[jira] [Resolved] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-845. -- Resolution: Fixed > Allow parallel writing to Hudi tables using Locking service > ---

[jira] [Closed] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-845. > Allow parallel writing to Hudi tables using Locking service > -

[jira] [Commented] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-03-23 Thread Balaji Varadarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307095#comment-17307095 ] Balaji Varadarajan commented on HUDI-1711: -- [~shivnarayan]: Can you triage this i

[GitHub] [hudi] bvaradar commented on issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-03-23 Thread GitBox
bvaradar commented on issue #2705: URL: https://github.com/apache/hudi/issues/2705#issuecomment-804927902 @nsivabalan : This looks like a bug and likely needs fixing. Opened : https://issues.apache.org/jira/browse/HUDI-1711 -- This is an automated message from the Apache Git Service. To

[jira] [Closed] (HUDI-1615) GH Issue 2515/ Failure to archive commits on row writer/delete paths

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1615. - > GH Issue 2515/ Failure to archive commits on row writer/delete paths > -

[jira] [Resolved] (HUDI-1615) GH Issue 2515/ Failure to archive commits on row writer/delete paths

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1615. --- Resolution: Fixed > GH Issue 2515/ Failure to archive commits on row writer/delete paths > ---

[jira] [Closed] (HUDI-1552) Improve performance of key lookups from base file (HFile) in Metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1552. - > Improve performance of key lookups from base file (HFile) in Metadata table > --

[jira] [Resolved] (HUDI-1552) Improve performance of key lookups from base file (HFile) in Metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1552. --- Resolution: Fixed > Improve performance of key lookups from base file (HFile) in Metadata table >

[GitHub] [hudi] kimberlyamandalu commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-03-23 Thread GitBox
kimberlyamandalu commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-804930443 > @kimberlyamandalu do you have the full stack trace? @vinothchandar there was actually no stack trace in the logs. Those messages I posted above would just repeat for al

[jira] [Closed] (HUDI-1496) Seek Error when querying MOR tables in GCP

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1496. - > Seek Error when querying MOR tables in GCP > -- > > Key: HUD

[jira] [Resolved] (HUDI-1496) Seek Error when querying MOR tables in GCP

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1496. --- Resolution: Fixed > Seek Error when querying MOR tables in GCP > -- >

[jira] [Updated] (HUDI-1496) Seek Error when querying MOR tables in GCP

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1496: -- Fix Version/s: 0.8.0 > Seek Error when querying MOR tables in GCP > -- >

[jira] [Updated] (HUDI-1552) Improve performance of key lookups from base file (HFile) in Metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1552: -- Fix Version/s: 0.8.0 > Improve performance of key lookups from base file (HFile) in Metadata table > ---

[jira] [Commented] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307102#comment-17307102 ] Gary Li commented on HUDI-1685: --- [~akanungo] thanks for your contribution. I can't assign th

[jira] [Resolved] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1685. --- Resolution: Fixed > Current date should not be initialized in the constructor > --

[jira] [Closed] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1685. - > Current date should not be initialized in the constructor >

[jira] [Updated] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1685: -- Fix Version/s: 0.8.0 > Current date should not be initialized in the constructor > -

[jira] [Resolved] (HUDI-1662) Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1662. --- Resolution: Fixed > Failed to query real-time view use hive/spark-sql when hudi mor table > contains dateTyp

[jira] [Updated] (HUDI-1662) Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1662: -- Fix Version/s: 0.8.0 > Failed to query real-time view use hive/spark-sql when hudi mor table > contains dateTy

[jira] [Resolved] (HUDI-1618) NPE with MultiTableDeltaStreamer with ParquetSource

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1618. --- Fix Version/s: 0.8.0 Resolution: Fixed > NPE with MultiTableDeltaStreamer with ParquetSource >

[jira] [Assigned] (HUDI-1618) NPE with MultiTableDeltaStreamer with ParquetSource

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1618: - Assignee: sivabalan narayanan > NPE with MultiTableDeltaStreamer with ParquetSource > ---

[jira] [Closed] (HUDI-1662) Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1662. - > Failed to query real-time view use hive/spark-sql when hudi mor table > contains dateType > --

[jira] [Closed] (HUDI-1618) NPE with MultiTableDeltaStreamer with ParquetSource

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1618. - > NPE with MultiTableDeltaStreamer with ParquetSource > --- > >

[jira] [Closed] (HUDI-1661) Change utility methods that help get extra metadata to ignore internal rewrite commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1661. - > Change utility methods that help get extra metadata to ignore internal > rewrite commits >

[jira] [Resolved] (HUDI-1661) Change utility methods that help get extra metadata to ignore internal rewrite commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1661. --- Resolution: Resolved > Change utility methods that help get extra metadata to ignore internal > rewrite commi

[jira] [Updated] (HUDI-1661) Change utility methods that help get extra metadata to ignore internal rewrite commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1661: -- Fix Version/s: 0.8.0 > Change utility methods that help get extra metadata to ignore internal > rewrite commits

[jira] [Resolved] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1660. --- Resolution: Fixed > Exclude pending compaction & clustering from rollback > --

[jira] [Closed] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1660. - > Exclude pending compaction & clustering from rollback > - > >

[jira] [Updated] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1660: -- Fix Version/s: 0.8.0 > Exclude pending compaction & clustering from rollback > -

[jira] [Resolved] (HUDI-1587) Add latency and freshness metrics for commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1587. --- Resolution: Implemented > Add latency and freshness metrics for commits >

[jira] [Closed] (HUDI-1587) Add latency and freshness metrics for commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1587. - > Add latency and freshness metrics for commits > - > > Ke

[jira] [Updated] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1553: -- Fix Version/s: 0.8.0 > Add configs for TimelineServer to configure Jetty > -

[jira] [Resolved] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1553. --- Resolution: Resolved > Add configs for TimelineServer to configure Jetty > ---

[jira] [Closed] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1553. - > Add configs for TimelineServer to configure Jetty > - > >

[jira] [Resolved] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1634. --- Resolution: Fixed > Handle the case metadata table cannot be synced due to instants being archived > -

[jira] [Closed] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1634. - > Handle the case metadata table cannot be synced due to instants being archived > ---

[jira] [Updated] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1634: -- Fix Version/s: 0.8.0 > Handle the case metadata table cannot be synced due to instants being archived >

[jira] [Resolved] (HUDI-1644) Do not delete rollback instants in RollbackActionExecutor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1644. --- Resolution: Fixed > Do not delete rollback instants in RollbackActionExecutor > --

[jira] [Closed] (HUDI-1644) Do not delete rollback instants in RollbackActionExecutor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1644. - > Do not delete rollback instants in RollbackActionExecutor >

[jira] [Assigned] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1583: - Assignee: Lietong Liu > Hudi will skip remaining log files if there is logFile with zero size in > logF

[jira] [Updated] (HUDI-1540) Fix commons-codec shading in hudi-spark-bundle

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1540: -- Fix Version/s: (was: 0.7.0) 0.8.0 > Fix commons-codec shading in hudi-spark-bundle >

[jira] [Resolved] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1611. --- Resolution: Resolved > Allow directories to be filtered during the bootstrap of the metadata table > -

[jira] [Closed] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1583. - > Hudi will skip remaining log files if there is logFile with zero size in > logFileList when merge on read. > -

[jira] [Updated] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1611: -- Fix Version/s: 0.8.0 > Allow directories to be filtered during the bootstrap of the metadata table > ---

[jira] [Closed] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1611. - > Allow directories to be filtered during the bootstrap of the metadata table > --

[jira] [Updated] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1583: -- Fix Version/s: (was: 0.6.0) 0.8.0 > Hudi will skip remaining log files if there is logFil

[jira] [Assigned] (HUDI-1620) TestPushGateWayReporter failed when run separately

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1620: - Assignee: Raymond Xu > TestPushGateWayReporter failed when run separately > -

[jira] [Closed] (HUDI-1620) TestPushGateWayReporter failed when run separately

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1620. - > TestPushGateWayReporter failed when run separately > -- > >

[jira] [Resolved] (HUDI-1620) TestPushGateWayReporter failed when run separately

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1620. --- Resolution: Resolved > TestPushGateWayReporter failed when run separately > --

[jira] [Updated] (HUDI-1477) Support CopyOnWriteTable in java client

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1477: -- Fix Version/s: 0.8.0 > Support CopyOnWriteTable in java client > --- > >

[jira] [Closed] (HUDI-1477) Support CopyOnWriteTable in java client

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1477. - > Support CopyOnWriteTable in java client > --- > > Key: HUDI-1477

[GitHub] [hudi] bvaradar commented on issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-03-23 Thread GitBox
bvaradar commented on issue #2676: URL: https://github.com/apache/hudi/issues/2676#issuecomment-804945810 Did you try increasing executor instances to see if the job completes in less time ? Also, it looks like GlobalBloomIndex is enabled. This is needed only when you want record-key u

[jira] [Closed] (HUDI-1586) Reduce the coupling of hadoop.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1586. - > Reduce the coupling of hadoop. > -- > > Key: HUDI-1586 >

[jira] [Updated] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1486: -- Status: Closed (was: Patch Available) > Remove pending rollback and move to cleaner > -

[jira] [Reopened] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reopened HUDI-1486: --- > Remove pending rollback and move to cleaner > --- > > Ke

[jira] [Assigned] (HUDI-1586) Reduce the coupling of hadoop.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1586: - Assignee: 张超明 > Reduce the coupling of hadoop. > -- > > Key:

[jira] [Resolved] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1486. --- Resolution: Resolved > Remove pending rollback and move to cleaner > -

[jira] [Updated] (HUDI-1539) Bug in HoodieCombineRealtimeRecordReader returns wrong results

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1539: -- Fix Version/s: 0.8.0 > Bug in HoodieCombineRealtimeRecordReader returns wrong results >

[jira] [Closed] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1486. - > Remove pending rollback and move to cleaner > --- > > Key: H

[jira] [Closed] (HUDI-1540) Fix commons-codec shading in hudi-spark-bundle

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1540. - > Fix commons-codec shading in hudi-spark-bundle > -- > >

[jira] [Updated] (HUDI-1582) HiveSyncTool - silently fails (RuntimeException is swallowed)

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1582: -- Fix Version/s: 0.8.0 > HiveSyncTool - silently fails (RuntimeException is swallowed) >

  1   2   3   4   5   6   >