[jira] [Closed] (HUDI-1878) Add max memory option for flink writer task

2021-05-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1878. -- Resolution: Done bfbf993cbe3f1e3fab93095f4342ed17423efab5 > Add max memory option for flink writer task > -

[hudi] branch master updated (2c5a661 -> bfbf993)

2021-05-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 2c5a661 [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false (#2759)

[GitHub] [hudi] yanghua merged pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
yanghua merged pull request #2920: URL: https://github.com/apache/hudi/pull/2920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please co

[GitHub] [hudi] pengzhiwei2018 commented on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-05-07 Thread GitBox
pengzhiwei2018 commented on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-835134388 Hi @vinothchandar @umehrot2 , The PR has updated, mainly with the follow changes: - Support atomic for CTAS - Support use timestamp type as partition field - Fix ex

[jira] [Created] (HUDI-1885) Support Delete/Update Non-Pk Table

2021-05-07 Thread pengzhiwei (Jira)
pengzhiwei created HUDI-1885: Summary: Support Delete/Update Non-Pk Table Key: HUDI-1885 URL: https://issues.apache.org/jira/browse/HUDI-1885 Project: Apache Hudi Issue Type: Sub-task C

[jira] [Created] (HUDI-1884) Support Partial Update For MergeInto

2021-05-07 Thread pengzhiwei (Jira)
pengzhiwei created HUDI-1884: Summary: Support Partial Update For MergeInto Key: HUDI-1884 URL: https://issues.apache.org/jira/browse/HUDI-1884 Project: Apache Hudi Issue Type: Sub-task

[GitHub] [hudi] tandonraghav commented on issue #2919: [SUPPORT] Schema Evolution Failing Spark+Hudi- Adding New fields

2021-05-07 Thread GitBox
tandonraghav commented on issue #2919: URL: https://github.com/apache/hudi/issues/2919#issuecomment-835063730 @vinothchandar @nsivabalan Can you guys please help here. I am stuck and cant things forward if Schema Evolution is not happening. -- This is an automated message from the A

[GitHub] [hudi] danny0405 commented on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
danny0405 commented on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-835060053 > @danny0405 I saw this failure post merging. > > ``` > [INFO] > [INFO] Results: > [INFO] > [ERROR] Failures: > [ERROR] TestRowDataToHoodieFunction.testR

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] wsxGit commented on issue #2830: [SUPPORT]same _hoodie_record_key has duplicates data

2021-05-07 Thread GitBox
wsxGit commented on issue #2830: URL: https://github.com/apache/hudi/issues/2830#issuecomment-835003864 I have found reason ,hoodie.datasource.write.operation use UPSERT sloved the issue > oh, I see you are using GLOBAL_BLOOM as your index. Can you tell us which version of hudi are

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[jira] [Assigned] (HUDI-1870) Move spark avro serialization class into hudi repo

2021-05-07 Thread huangfeifei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangfeifei reassigned HUDI-1870: - Assignee: (was: huangfeifei) > Move spark avro serialization class into hudi repo > -

[GitHub] [hudi] root18039532923 commented on issue #2924: [SUPPORT]presto query:could not initialize class org.apache.hudi.common.util.HoodieAvroUtils

2021-05-07 Thread GitBox
root18039532923 commented on issue #2924: URL: https://github.com/apache/hudi/issues/2924#issuecomment-834969846 @bvaradar Can you help me solve this problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [hudi] root18039532923 commented on issue #2924: [SUPPORT]presto query:could not initialize class org.apache.hudi.common.util.HoodieAvroUtils

2021-05-07 Thread GitBox
root18039532923 commented on issue #2924: URL: https://github.com/apache/hudi/issues/2924#issuecomment-834968018 I think it may be due to log files.Because I query mor_ro table,it is ok. @msridhar -- This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [hudi] root18039532923 opened a new issue #2924: [SUPPORT]presto query:could not initialize class org.apache.hudi.common.util.HoodieAvroUtils

2021-05-07 Thread GitBox
root18039532923 opened a new issue #2924: URL: https://github.com/apache/hudi/issues/2924 **Environment Description** * Hudi version :0.6.0 * Spark version :2.4.0 * Hive version :2.1.1 *presto version:0.240 * Hadoop version :3.0.0 * Storage (HDFS/S3/

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[jira] [Resolved] (HUDI-1759) Save one connection retry when hiveSyncTool run with useJdbc=false

2021-05-07 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1759. --- Resolution: Fixed > Save one connection retry when hiveSyncTool run with useJdbc=false > -

[GitHub] [hudi] li36909 commented on pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-05-07 Thread GitBox
li36909 commented on pull request #2759: URL: https://github.com/apache/hudi/pull/2759#issuecomment-834961606 @vinothchandar thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [hudi] danny0405 commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-07 Thread GitBox
danny0405 commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-834955549 > > each writer task takes a AbstractWriteClient, and there may be multiple write tasks in one process. > > @danny0405 please help me understand this better. By one process

[GitHub] [hudi] jsbali commented on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
jsbali commented on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-834954263 Hey @satishkotha It passed after the empty commit but failed on merging the master with it :( .Also the failure seems unrelated to this PR. -- This is an automated message from th

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] danny0405 closed pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
danny0405 closed pull request #2920: URL: https://github.com/apache/hudi/pull/2920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2714: [HUDI-1707] Reduces log level for too verbose messages from info to debug level.

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2714: URL: https://github.com/apache/hudi/pull/2714#issuecomment-830719410 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2714?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2714: [HUDI-1707] Reduces log level for too verbose messages from info to debug level.

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2714: URL: https://github.com/apache/hudi/pull/2714#issuecomment-830719410 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2714?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar merged pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-05-07 Thread GitBox
vinothchandar merged pull request #2759: URL: https://github.com/apache/hudi/pull/2759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, ple

[hudi] branch master updated (528f4ca -> 2c5a661)

2021-05-07 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 528f4ca [HUDI-1880] Support streaming read with compaction and cleaning (#2921) add 2c5a661 [HUDI-1759] Save on

[GitHub] [hudi] vinothchandar commented on a change in pull request #2759: [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false

2021-05-07 Thread GitBox
vinothchandar commented on a change in pull request #2759: URL: https://github.com/apache/hudi/pull/2759#discussion_r628578256 ## File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java ## @@ -386,39 +408,18 @@ public CommandProcessorRespon

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar commented on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
vinothchandar commented on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834783294 @danny0405 I saw this failure post merging. ``` [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] TestRowDataToHoodieFunction.testRateLimit:72

[GitHub] [hudi] danny0405 closed pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
danny0405 closed pull request #2920: URL: https://github.com/apache/hudi/pull/2920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-828792354 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2833?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-828792354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-828792354 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2833?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-828792354 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2833?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] zhedoubushishi commented on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-07 Thread GitBox
zhedoubushishi commented on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-834715657 @xushiyan are you intrested in reviewing this PR? This PR refactored the current Hudi configuration framework and could potentially simply Hudi configuration. -- This is

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2918: [HUDI-1877] Add support in clustering to not change record location

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2918: URL: https://github.com/apache/hudi/pull/2918#issuecomment-833971983 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2918?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] satishkotha commented on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
satishkotha commented on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-834688893 @jsbali could you check why CI is failing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2918: [HUDI-1877] Add support in clustering to not change record location

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2918: URL: https://github.com/apache/hudi/pull/2918#issuecomment-833971983 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2918?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-826325732 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2809?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[jira] [Commented] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit

2021-05-07 Thread Udit Mehrotra (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340927#comment-17340927 ] Udit Mehrotra commented on HUDI-1723: - We should consider something similar as the lon

[GitHub] [hudi] vinothchandar commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-07 Thread GitBox
vinothchandar commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-834595116 >each writer task takes a AbstractWriteClient, and there may be multiple write tasks in one process. @danny0405 please help me understand this better. By one process, y

[jira] [Updated] (HUDI-1879) Spark DataSource tables/HoodieFileIndex issues for Merge On Read

2021-05-07 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1879: - Fix Version/s: 0.9.0 > Spark DataSource tables/HoodieFileIndex issues for Merge On Read >

[jira] [Updated] (HUDI-1864) Support for java.time.LocalDate in TimestampBasedAvroKeyGenerator

2021-05-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1864: - Labels: pull-request-available sev:high (was: sev:high) > Support for java.time.LocalDate in Time

[GitHub] [hudi] vaibhav-sinha opened a new pull request #2923: [HUDI-1864] Added support for Date, Timestamp, LocalDate and LocalDateTime in TimestampBasedAvroKeyGenerator

2021-05-07 Thread GitBox
vaibhav-sinha opened a new pull request #2923: URL: https://github.com/apache/hudi/pull/2923 This PR adds the support for Date, Timestamp, LocalDate and LocalDateTime types in TimestampBasedAvroKeyGenerator. Hence, columns with these types can be used for partitioning. The support for Date

[GitHub] [hudi] jsbali commented on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-07 Thread GitBox
jsbali commented on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-834554256 @vinothchandar simplified the code and fixed a typo in javadoc as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2922: URL: https://github.com/apache/hudi/pull/2922#issuecomment-834349577 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2922: URL: https://github.com/apache/hudi/pull/2922#issuecomment-834349577 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] tandonraghavs commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
tandonraghavs commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834497022 @sathyaprakashg @vinothchandar Can this PR handle [this](https://github.com/apache/hudi/issues/2919) issue as well? -- This is an automated message from the Apache Git

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] nsivabalan commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
nsivabalan commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834436516 @sathyaprakashg and others: trying to understand the use-case here. I understand its related to deltastreamer receiving events in old schema after Hudi's dataset schema got evol

[jira] [Created] (HUDI-1883) Support Truncate Table For Hoodie

2021-05-07 Thread pengzhiwei (Jira)
pengzhiwei created HUDI-1883: Summary: Support Truncate Table For Hoodie Key: HUDI-1883 URL: https://issues.apache.org/jira/browse/HUDI-1883 Project: Apache Hudi Issue Type: Sub-task Co

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2922: URL: https://github.com/apache/hudi/pull/2922#issuecomment-834349577 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter commented on pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
codecov-commenter commented on pull request #2922: URL: https://github.com/apache/hudi/pull/2922#issuecomment-834349577 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache

[GitHub] [hudi] yanghua commented on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
yanghua commented on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834320533 @danny0405 Conflicting files... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Closed] (HUDI-1880) Support streaming read with compaction and cleaning

2021-05-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1880. -- Resolution: Implemented 528f4ca988209cbc9e519bd42b87b896aea992b6 > Support streaming read with compaction and c

[hudi] branch master updated: [HUDI-1880] Support streaming read with compaction and cleaning (#2921)

2021-05-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 528f4ca [HUDI-1880] Support streaming read with

[GitHub] [hudi] yanghua merged pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
yanghua merged pull request #2921: URL: https://github.com/apache/hudi/pull/2921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please co

[GitHub] [hudi] nsivabalan commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
nsivabalan commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834312959 > I spent sometime to understand this PR. thanks for putting it up @sathyaprakashg. I have few clarifications. > > 1. Can you fix the description wrt latest status. I don'

[GitHub] [hudi] pan3793 commented on a change in pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
pan3793 commented on a change in pull request #2922: URL: https://github.com/apache/hudi/pull/2922#discussion_r628086006 ## File path: hudi-utilities/hudi-utilities-flink/pom.xml ## @@ -0,0 +1,72 @@ + + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.o

[GitHub] [hudi] pan3793 commented on a change in pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-07 Thread GitBox
pan3793 commented on a change in pull request #2922: URL: https://github.com/apache/hudi/pull/2922#discussion_r628085696 ## File path: hudi-utilities/hudi-utilities-flink/pom.xml ## @@ -0,0 +1,72 @@ + + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.o

[jira] [Updated] (HUDI-1882) Hive query Hoodie realtime table not found hoodie table

2021-05-07 Thread shenjinxin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shenjinxin updated HUDI-1882: - Description: There is my slack linkĀ  [https://apache-hudi.slack.com/archives/C4D716NPQ/p1620376992150500]

[jira] [Created] (HUDI-1882) Hive query Hoodie realtime table not found hoodie table

2021-05-07 Thread shenjinxin (Jira)
shenjinxin created HUDI-1882: Summary: Hive query Hoodie realtime table not found hoodie table Key: HUDI-1882 URL: https://issues.apache.org/jira/browse/HUDI-1882 Project: Apache Hudi Issue Type:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834124145 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] danny0405 closed pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
danny0405 closed pull request #2920: URL: https://github.com/apache/hudi/pull/2920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834124145 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834124145 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar commented on a change in pull request #2915: [HUDI-251] Adds JDBC source support for DeltaStreamer

2021-05-07 Thread GitBox
vinothchandar commented on a change in pull request #2915: URL: https://github.com/apache/hudi/pull/2915#discussion_r627987063 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/SqlQueryBuilder.java ## @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software

[jira] [Created] (HUDI-1881) HoodieMultiTableDeltaStreamer does not ingest from all topics when using continuous mode

2021-05-07 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1881: - Summary: HoodieMultiTableDeltaStreamer does not ingest from all topics when using continuous mode Key: HUDI-1881 URL: https://issues.apache.org/jira/browse/HUDI-1881

[jira] [Updated] (HUDI-1881) HoodieMultiTableDeltaStreamer does not ingest from all topics when using continuous mode

2021-05-07 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1881: -- Labels: sev:critical (was: ) > HoodieMultiTableDeltaStreamer does not ingest from all topics wh

[GitHub] [hudi] giaosudau commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
giaosudau commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834140226 Yes I am following this PR for the last 6 months. When I presented to my teammate I thought it works out of the box but it didn't. Kind of sad. Please help put more focus o

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834124145 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] MyLanPangzi edited a comment on pull request #2719: [HUDI-1721] run_sync_tool support hive3

2021-05-07 Thread GitBox
MyLanPangzi edited a comment on pull request #2719: URL: https://github.com/apache/hudi/pull/2719#issuecomment-834130415 @vinothchandar i think the hive dependencies too many to control... i can use anthor way to mapping hive table to hudi table. e.g CREATE EXTERNAL TABLE t1 (

[GitHub] [hudi] MyLanPangzi commented on pull request #2719: [HUDI-1721] run_sync_tool support hive3

2021-05-07 Thread GitBox
MyLanPangzi commented on pull request #2719: URL: https://github.com/apache/hudi/pull/2719#issuecomment-834130415 @vinothchandar i think the hive dependencies too many to control... i can use anthor way to mapping hive table to hudi table. e.g CREATE EXTERNAL TABLE dwd_sale_sale_de

[GitHub] [hudi] codecov-commenter commented on pull request #2921: [HUDI-1880] Support streaming read with compaction and cleaning

2021-05-07 Thread GitBox
codecov-commenter commented on pull request #2921: URL: https://github.com/apache/hudi/pull/2921#issuecomment-834124145 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache

[GitHub] [hudi] vinothchandar edited a comment on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
vinothchandar edited a comment on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834118960 I actually see that @n3nash has approved this several months ago, pending the perf issue that @sathyaprakashg himself has pointed out. Agree with you,. Let me gather

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2920: [HUDI-1878] Add max memory option for flink writer task

2021-05-07 Thread GitBox
codecov-commenter edited a comment on pull request #2920: URL: https://github.com/apache/hudi/pull/2920#issuecomment-834095568 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-05-07 Thread GitBox
vinothchandar commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-834118960 I actually see that @n3nash has approved this several months ago, pending the perf issue that @sathyaprakashg himself has pointed out. Agree with you,. Let me gather some co