[GitHub] [hudi] wangxianghu commented on a change in pull request #3006: [HUDI-1943] Lose properties when hoodieWriteConfig initializtion

2021-05-28 Thread GitBox
wangxianghu commented on a change in pull request #3006: URL: https://github.com/apache/hudi/pull/3006#discussion_r641884136 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -242,7 +242,9 @@ public static TypedProperties

[GitHub] [hudi] wangxianghu commented on a change in pull request #3006: [HUDI-1943] Lose properties when hoodieWriteConfig initializtion

2021-05-28 Thread GitBox
wangxianghu commented on a change in pull request #3006: URL: https://github.com/apache/hudi/pull/3006#discussion_r641884136 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -242,7 +242,9 @@ public static TypedProperties

[jira] [Updated] (HUDI-1943) Lose properties when hoodieWriteConfig initializtion

2021-05-28 Thread Xianghu Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang updated HUDI-1943: --- Summary: Lose properties when hoodieWriteConfig initializtion (was: [hudi-flink]Lose properties when

[jira] [Created] (HUDI-1947) Hudi Commit Callback and commit in a single transaction

2021-05-28 Thread Pranoti Shanbhag (Jira)
Pranoti Shanbhag created HUDI-1947: -- Summary: Hudi Commit Callback and commit in a single transaction Key: HUDI-1947 URL: https://issues.apache.org/jira/browse/HUDI-1947 Project: Apache Hudi

[GitHub] [hudi] pranotishanbhag opened a new issue #3008: [SUPPORT] Hive Sync issues on deletes and non partitioned table

2021-05-28 Thread GitBox
pranotishanbhag opened a new issue #3008: URL: https://github.com/apache/hudi/issues/3008 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? YES - Join the mailing list to engage in conversations and

[GitHub] [hudi] jtmzheng commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?

2021-05-28 Thread GitBox
jtmzheng commented on issue #2983: URL: https://github.com/apache/hudi/issues/2983#issuecomment-850619877 @fanaticjo what kind of issues did you see? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] zhedoubushishi commented on a change in pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-28 Thread GitBox
zhedoubushishi commented on a change in pull request #2833: URL: https://github.com/apache/hudi/pull/2833#discussion_r641742577 ## File path: hudi-common/src/test/java/org/apache/hudi/common/config/TestConfigOption.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] fanaticjo commented on issue #2983: [SUPPORT] Is hoodie.consistency.check.enabled still relevant?

2021-05-28 Thread GitBox
fanaticjo commented on issue #2983: URL: https://github.com/apache/hudi/issues/2983#issuecomment-850545801 I think this is still required as i have seen issues still persisting in emr version below 5.33 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] fanaticjo commented on issue #2975: [SUPPORT] Read record using index

2021-05-28 Thread GitBox
fanaticjo commented on issue #2975: URL: https://github.com/apache/hudi/issues/2975#issuecomment-850544149 Hello @calleo i am trying to make this as a generic release in version 0.9.0 , till the time being you can clone this git hub repo -

[GitHub] [hudi] nsivabalan commented on pull request #2967: Added blog for Hudi cleaner service

2021-05-28 Thread GitBox
nsivabalan commented on pull request #2967: URL: https://github.com/apache/hudi/pull/2967#issuecomment-850502987 Can you build the site locally and take screenshot and attach it here. would be nice to review that as well. for eg: https://github.com/apache/hudi/pull/2969 -- This is

[GitHub] [hudi] wangxianghu commented on a change in pull request #2993: [HUDI-1929] Make HoodieDeltaStreamer support configure KeyGenerator by type

2021-05-28 Thread GitBox
wangxianghu commented on a change in pull request #2993: URL: https://github.com/apache/hudi/pull/2993#discussion_r641622004 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -72,8 +71,11 @@ public static final

[GitHub] [hudi] wangxianghu commented on a change in pull request #2993: [HUDI-1929] Make HoodieDeltaStreamer support configure KeyGenerator by type

2021-05-28 Thread GitBox
wangxianghu commented on a change in pull request #2993: URL: https://github.com/apache/hudi/pull/2993#discussion_r641622004 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -72,8 +71,11 @@ public static final

[jira] [Closed] (HUDI-1946) Enhance SqlQueryBasedTransformer to allow user use wildcard to represent all the columns

2021-05-28 Thread Xianghu Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang closed HUDI-1946. -- Resolution: Not A Problem > Enhance SqlQueryBasedTransformer to allow user use wildcard to represent

[GitHub] [hudi] wangxianghu closed pull request #3007: [HUDI-1946] Enhance SqlQueryBasedTransformer to allow user use wildca…

2021-05-28 Thread GitBox
wangxianghu closed pull request #3007: URL: https://github.com/apache/hudi/pull/3007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] codecov-commenter commented on pull request #3007: [HUDI-1946] Enhance SqlQueryBasedTransformer to allow user use wildca…

2021-05-28 Thread GitBox
codecov-commenter commented on pull request #3007: URL: https://github.com/apache/hudi/pull/3007#issuecomment-850460794 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3007?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)

[GitHub] [hudi] leesf merged pull request #3004: [HUDI-1940] Add SqlQueryBasedTransformer unit test

2021-05-28 Thread GitBox
leesf merged pull request #3004: URL: https://github.com/apache/hudi/pull/3004 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [HUDI-1940] Add SqlQueryBasedTransformer unit test (#3004)

2021-05-28 Thread leesf
This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 974b476 [HUDI-1940] Add SqlQueryBasedTransformer

[GitHub] [hudi] wangxianghu commented on pull request #3007: [HUDI-1946] Enhance SqlQueryBasedTransformer to allow user use wildca…

2021-05-28 Thread GitBox
wangxianghu commented on pull request #3007: URL: https://github.com/apache/hudi/pull/3007#issuecomment-850456430 will add the unit test after https://github.com/apache/hudi/pull/3004 merged -- This is an automated message from the Apache Git Service. To respond to the message, please

[jira] [Updated] (HUDI-1946) Enhance SqlQueryBasedTransformer to allow user use wildcard to represent all the columns

2021-05-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1946: - Labels: pull-request-available (was: ) > Enhance SqlQueryBasedTransformer to allow user use

[GitHub] [hudi] wangxianghu opened a new pull request #3007: [HUDI-1946] Enhance SqlQueryBasedTransformer to allow user use wildca…

2021-05-28 Thread GitBox
wangxianghu opened a new pull request #3007: URL: https://github.com/apache/hudi/pull/3007 …rd to represent all the columns ## What is the purpose of the pull request When the user wants to derive one or more columns from the existing columns and the existing columns are

[jira] [Created] (HUDI-1946) Enhance SqlQueryBasedTransformer to allow user use wildcard to represent all the columns

2021-05-28 Thread Xianghu Wang (Jira)
Xianghu Wang created HUDI-1946: -- Summary: Enhance SqlQueryBasedTransformer to allow user use wildcard to represent all the columns Key: HUDI-1946 URL: https://issues.apache.org/jira/browse/HUDI-1946

[GitHub] [hudi] wangxianghu commented on a change in pull request #3004: [HUDI-1940] Add SqlQueryBasedTransformer unit test

2021-05-28 Thread GitBox
wangxianghu commented on a change in pull request #3004: URL: https://github.com/apache/hudi/pull/3004#discussion_r641543572 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/transform/TestSqlQueryBasedTransformer.java ## @@ -0,0 +1,94 @@ +/* + * Licensed

[jira] [Commented] (HUDI-1910) Supporting Kafka based checkpointing for HoodieDeltaStreamer

2021-05-28 Thread Vinay (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17353335#comment-17353335 ] Vinay commented on HUDI-1910: - [~nishith29] can you pls delete one of the sub-tasks, got added twice. I added

[jira] [Created] (HUDI-1945) Support Hudi to read from Kafka Consumer Group Offset

2021-05-28 Thread Vinay (Jira)
Vinay created HUDI-1945: --- Summary: Support Hudi to read from Kafka Consumer Group Offset Key: HUDI-1945 URL: https://issues.apache.org/jira/browse/HUDI-1945 Project: Apache Hudi Issue Type: Sub-task

[jira] [Updated] (HUDI-1945) Support Hudi to read from Kafka Consumer Group Offset

2021-05-28 Thread Vinay (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HUDI-1945: Status: New (was: Open) > Support Hudi to read from Kafka Consumer Group Offset >

[jira] [Created] (HUDI-1944) Support Hudi to read from Kafka Consumer Group Offset

2021-05-28 Thread Vinay (Jira)
Vinay created HUDI-1944: --- Summary: Support Hudi to read from Kafka Consumer Group Offset Key: HUDI-1944 URL: https://issues.apache.org/jira/browse/HUDI-1944 Project: Apache Hudi Issue Type: Sub-task

[GitHub] [hudi] veenaypatil commented on pull request #2998: [HUDI-1426] Rename classname in camel case format

2021-05-28 Thread GitBox
veenaypatil commented on pull request #2998: URL: https://github.com/apache/hudi/pull/2998#issuecomment-850393465 @leesf I agree, this will be breaking change once users upgrade to new version, I think if we don't want to do this change, we should at least update the document to correct

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3006: [HUDI-1943]fix lose properties problem

2021-05-28 Thread GitBox
codecov-commenter edited a comment on pull request #3006: URL: https://github.com/apache/hudi/pull/3006#issuecomment-850365971 #

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3006: [HUDI-1943]fix lose properties problem

2021-05-28 Thread GitBox
codecov-commenter edited a comment on pull request #3006: URL: https://github.com/apache/hudi/pull/3006#issuecomment-850365971 #

[GitHub] [hudi] codecov-commenter commented on pull request #3006: [HUDI-1943]fix lose properties problem

2021-05-28 Thread GitBox
codecov-commenter commented on pull request #3006: URL: https://github.com/apache/hudi/pull/3006#issuecomment-850365971 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3006?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2923: [HUDI-1864] Added support for Date, Timestamp, LocalDate and LocalDateTime in TimestampBasedAvroKeyGenerator

2021-05-28 Thread GitBox
codecov-commenter edited a comment on pull request #2923: URL: https://github.com/apache/hudi/pull/2923#issuecomment-846613183 #

[GitHub] [hudi] leesf commented on a change in pull request #3004: [HUDI-1940] Add SqlQueryBasedTransformer unit test

2021-05-28 Thread GitBox
leesf commented on a change in pull request #3004: URL: https://github.com/apache/hudi/pull/3004#discussion_r641474949 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/transform/TestSqlQueryBasedTransformer.java ## @@ -0,0 +1,94 @@ +/* + * Licensed to the

[GitHub] [hudi] leesf commented on pull request #2998: [HUDI-1426] Rename classname in camel case format

2021-05-28 Thread GitBox
leesf commented on pull request #2998: URL: https://github.com/apache/hudi/pull/2998#issuecomment-850349161 @veenaypatil Thanks for you contribution, after changing the classname it would cause compatibility issues since there are many users used `NonpartitionedKeyGenerator` in their

[jira] [Updated] (HUDI-1943) [hudi-flink]Lose properties when hoodieWriteConfig initializtion

2021-05-28 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1943: - Labels: pull-request-available (was: ) > [hudi-flink]Lose properties when hoodieWriteConfig

[GitHub] [hudi] hk-lrzy opened a new pull request #3006: [HUDI-1943]fix lose properties problem

2021-05-28 Thread GitBox
hk-lrzy opened a new pull request #3006: URL: https://github.com/apache/hudi/pull/3006 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Created] (HUDI-1943) [hudi-flink]Lose properties when hoodieWriteConfig initializtion

2021-05-28 Thread hk__lrzy (Jira)
hk__lrzy created HUDI-1943: -- Summary: [hudi-flink]Lose properties when hoodieWriteConfig initializtion Key: HUDI-1943 URL: https://issues.apache.org/jira/browse/HUDI-1943 Project: Apache Hudi

[GitHub] [hudi] t0il3ts0ap commented on issue #2934: [SUPPORT] Parquet file does not exist when trying to read hudi table incrementally

2021-05-28 Thread GitBox
t0il3ts0ap commented on issue #2934: URL: https://github.com/apache/hudi/issues/2934#issuecomment-850312591 @n3nash Sorry for late reply, was away on vacation 1. We run one instance of deltastreamer job every 2 hrs on source table. Each run at the max sources 6 million records using

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp

2021-05-28 Thread GitBox
codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 #

[GitHub] [hudi] codecov-commenter commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp

2021-05-28 Thread GitBox
codecov-commenter commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2438?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)

[GitHub] [hudi] veenaypatil commented on pull request #2998: [HUDI-1426] Rename classname in camel case format

2021-05-28 Thread GitBox
veenaypatil commented on pull request #2998: URL: https://github.com/apache/hudi/pull/2998#issuecomment-850271396 @yanghua can you please look into this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] pratyakshsharma commented on pull request #2967: Added blog for Hudi cleaner service

2021-05-28 Thread GitBox
pratyakshsharma commented on pull request #2967: URL: https://github.com/apache/hudi/pull/2967#issuecomment-850210814 @n3nash @nsivabalan Please take a look. All the comments are addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2967: Added blog for Hudi cleaner service

2021-05-28 Thread GitBox
pratyakshsharma commented on a change in pull request #2967: URL: https://github.com/apache/hudi/pull/2967#discussion_r641327528 ## File path: docs/_posts/2021-05-19-employing-right-configurations-for-hudi-cleaner.md ## @@ -0,0 +1,77 @@ +--- +title: "Employing correct

[GitHub] [hudi] hushenmin opened a new issue #3005: [SUPPORT]

2021-05-28 Thread GitBox
hushenmin opened a new issue #3005: URL: https://github.com/apache/hudi/issues/3005 How to query history snapshot by given one history partition? At present, through the following method, I can query the historical snapshot of a Hudi partition, but for me, this method is very

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2967: Added blog for Hudi cleaner service

2021-05-28 Thread GitBox
pratyakshsharma commented on a change in pull request #2967: URL: https://github.com/apache/hudi/pull/2967#discussion_r641319609 ## File path: docs/_posts/2021-05-19-employing-right-configurations-for-hudi-cleaner.md ## @@ -0,0 +1,77 @@ +--- +title: "Employing correct

[jira] [Closed] (HUDI-1923) Add state in StreamWriteFunction to restore

2021-05-28 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1923. -- Fix Version/s: 0.9.0 Resolution: Done bc18c39835d6775b063ae072aea4ba43177d66b1 > Add state in

[GitHub] [hudi] yanghua merged pull request #3002: [HUDI-1923] Support exactly-once

2021-05-28 Thread GitBox
yanghua merged pull request #3002: URL: https://github.com/apache/hudi/pull/3002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [FLINK-1923] Exactly-once write for flink writer (#3002)

2021-05-28 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new bc18c39 [FLINK-1923] Exactly-once write for

[GitHub] [hudi] yanghua commented on a change in pull request #2993: [HUDI-1929] Make HoodieDeltaStreamer support configure KeyGenerator by type

2021-05-28 Thread GitBox
yanghua commented on a change in pull request #2993: URL: https://github.com/apache/hudi/pull/2993#discussion_r639466058 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -72,8 +71,11 @@ public static final

[GitHub] [hudi] zhedoubushishi commented on a change in pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework

2021-05-28 Thread GitBox
zhedoubushishi commented on a change in pull request #2833: URL: https://github.com/apache/hudi/pull/2833#discussion_r641303741 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieHBaseIndexConfig.java ## @@ -18,107 +18,133 @@ package