[GitHub] [hudi] codecov-io edited a comment on pull request #2560: [HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2560: URL: https://github.com/apache/hudi/pull/2560#issuecomment-775997974 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=h1) Report > Merging [#2560](https://codecov.io/gh/apache/hudi/pull/2560?src=pr&el=desc) (835aed0) in

[GitHub] [hudi] garyli1019 commented on a change in pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query

2021-02-17 Thread GitBox
garyli1019 commented on a change in pull request #2378: URL: https://github.com/apache/hudi/pull/2378#discussion_r578186547 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala ## @@ -56,7 +58,7 @@ class MergeOnReadSna

[GitHub] [hudi] garyli1019 commented on a change in pull request #2540: [HUDI-1586] [Common Core] [Flink Integration] Reduce the coupling of hadoop.

2021-02-17 Thread GitBox
garyli1019 commented on a change in pull request #2540: URL: https://github.com/apache/hudi/pull/2540#discussion_r578182457 ## File path: hudi-common/pom.xml ## @@ -154,6 +155,7 @@ org.apache.hadoop hadoop-hdfs + provided Review comment: +1, I t

[GitHub] [hudi] codecov-io commented on pull request #2582: [HUDI-393] Add azure pipelines configs

2021-02-17 Thread GitBox
codecov-io commented on pull request #2582: URL: https://github.com/apache/hudi/pull/2582#issuecomment-781108823 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2582?src=pr&el=h1) Report > Merging [#2582](https://codecov.io/gh/apache/hudi/pull/2582?src=pr&el=desc) (8d8ca80) into [ma

[GitHub] [hudi] codecov-io commented on pull request #2581: [HUDI-1624] The state based index should bootstrap from existing base…

2021-02-17 Thread GitBox
codecov-io commented on pull request #2581: URL: https://github.com/apache/hudi/pull/2581#issuecomment-781103554 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2581?src=pr&el=h1) Report > Merging [#2581](https://codecov.io/gh/apache/hudi/pull/2581?src=pr&el=desc) (0ebd0c7) into [ma

[jira] [Updated] (HUDI-1626) Support Hash partition key generator

2021-02-17 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1626: -- Status: In Progress (was: Open) > Support Hash partition key generator > >

[jira] [Created] (HUDI-1626) Support Hash partition key generator

2021-02-17 Thread Gary Li (Jira)
Gary Li created HUDI-1626: - Summary: Support Hash partition key generator Key: HUDI-1626 URL: https://issues.apache.org/jira/browse/HUDI-1626 Project: Apache Hudi Issue Type: New Feature

[jira] [Updated] (HUDI-1625) Support Range Partition Key Generator

2021-02-17 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1625: -- Status: In Progress (was: Open) > Support Range Partition Key Generator > -

[jira] [Created] (HUDI-1625) Support Range Partition Key Generator

2021-02-17 Thread Gary Li (Jira)
Gary Li created HUDI-1625: - Summary: Support Range Partition Key Generator Key: HUDI-1625 URL: https://issues.apache.org/jira/browse/HUDI-1625 Project: Apache Hudi Issue Type: New Feature

[GitHub] [hudi] nsivabalan commented on pull request #2188: [HUDI-1347]fix Hbase index partition changes cause data duplication p…

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2188: URL: https://github.com/apache/hudi/pull/2188#issuecomment-781096026 @hj2016 : If you are busy, I can look at addressing any pending feedbacks. Will push updates to the patch w/ any fixes if you are ok. -

[GitHub] [hudi] nsivabalan commented on pull request #2210: [HUDI-1348] Provide option to clean up DFS sources

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2210: URL: https://github.com/apache/hudi/pull/2210#issuecomment-781095795 @hotienvu : If you are busy, I can look at addressing any pending feedback and looking into the build failure. Will push updates to the patch w/ any fixes if you are ok. -

[GitHub] [hudi] xushiyan commented on a change in pull request #2582: [HUDI-393] Add azure pipelines configs

2021-02-17 Thread GitBox
xushiyan commented on a change in pull request #2582: URL: https://github.com/apache/hudi/pull/2582#discussion_r578162176 ## File path: pom.xml ## @@ -1218,6 +1218,40 @@ + + azp-unit-tests + +false +true +true +

[GitHub] [hudi] xushiyan opened a new pull request #2582: [HUDI-393] Add azure pipelines configs

2021-02-17 Thread GitBox
xushiyan opened a new pull request #2582: URL: https://github.com/apache/hudi/pull/2582 Use a separate maven profile `azp-unit-tests` to exclude some failing tests in Azure pipelines. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Co

[GitHub] [hudi] vinothchandar commented on pull request #2541: [HUDI-1587] Add latency and freshness support

2021-02-17 Thread GitBox
vinothchandar commented on pull request #2541: URL: https://github.com/apache/hudi/pull/2541#issuecomment-781088346 Actually, I think you are just emitting a metric? can we also add this to commit metadata? This is an auto

[GitHub] [hudi] vinothchandar commented on pull request #2541: [HUDI-1587] Add latency and freshness support

2021-02-17 Thread GitBox
vinothchandar commented on pull request #2541: URL: https://github.com/apache/hudi/pull/2541#issuecomment-781085404 @xushiyan Few things I want to confirm 1. We are just adding this to extraMetadata in the commit? So, no changes to the avro schema for commit metadata? 2. For now

[jira] [Updated] (HUDI-1624) The state based index should bootstrap from existing base files

2021-02-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1624: - Labels: pull-request-available (was: ) > The state based index should bootstrap from existing bas

[GitHub] [hudi] danny0405 opened a new pull request #2581: [HUDI-1624] The state based index should bootstrap from existing base…

2021-02-17 Thread GitBox
danny0405 opened a new pull request #2581: URL: https://github.com/apache/hudi/pull/2581 … files ## What is the purpose of the pull request The index should bootstrap from existing base files if there are, in the design, we load all the keys for one partition if we found tha

[GitHub] [hudi] vinothchandar commented on a change in pull request #2541: [HUDI-1587] Add latency and freshness support

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2541: URL: https://github.com/apache/hudi/pull/2541#discussion_r578152379 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java ## @@ -79,4 +88,13 @@ public DefaultHoodieRecordP

[hudi] branch release-0.5.1 updated: Removing spring repos from pom (#2481) (#2549)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch release-0.5.1 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.5.1 by this push: new 51a9421 Removing spring repos from

[GitHub] [hudi] vinothchandar merged pull request #2549: [HUDI-1597]remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar merged pull request #2549: URL: https://github.com/apache/hudi/pull/2549 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[hudi] branch release-0.5.2 updated: Removing spring repos from pom (#2481) (#2550)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch release-0.5.2 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.5.2 by this push: new 8729c44 Removing spring repos from

[GitHub] [hudi] vinothchandar merged pull request #2550: [HUDI-1597]remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar merged pull request #2550: URL: https://github.com/apache/hudi/pull/2550 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Created] (HUDI-1624) The state based index should bootstrap from existing base files

2021-02-17 Thread Danny Chen (Jira)
Danny Chen created HUDI-1624: Summary: The state based index should bootstrap from existing base files Key: HUDI-1624 URL: https://issues.apache.org/jira/browse/HUDI-1624 Project: Apache Hudi Is

[GitHub] [hudi] vinothchandar merged pull request #2551: [HUDI-1597]remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar merged pull request #2551: URL: https://github.com/apache/hudi/pull/2551 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[hudi] branch release-0.7.0 updated: Removing spring repos from pom (#2481) (#2551)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch release-0.7.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.7.0 by this push: new 3e71c91 Removing spring repos from

[GitHub] [hudi] vinothchandar merged pull request #2552: [HUDI-1597]remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar merged pull request #2552: URL: https://github.com/apache/hudi/pull/2552 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[hudi] branch release-0.6.0 updated: Removing spring repos from pom (#2481) (#2552)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch release-0.6.0 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.6.0 by this push: new e599764 Removing spring repos from

[hudi] branch release-0.5.3 updated: Removing spring repos from pom (#2481) (#2548)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch release-0.5.3 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/release-0.5.3 by this push: new be12de3 Removing spring repos from

[GitHub] [hudi] vinothchandar merged pull request #2548: [HUDI-1597] remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar merged pull request #2548: URL: https://github.com/apache/hudi/pull/2548 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] vinothchandar commented on pull request #2548: [HUDI-1597] remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar commented on pull request #2548: URL: https://github.com/apache/hudi/pull/2548#issuecomment-781055752 @caidezhi Thanks for your patience. It took us a while to sort this out, given this is the first time we faced somethign like this. We deeply appreciate your initiativ

[GitHub] [hudi] nsivabalan commented on issue #2564: Hoodie clean is not deleting old files

2021-02-17 Thread GitBox
nsivabalan commented on issue #2564: URL: https://github.com/apache/hudi/issues/2564#issuecomment-781038539 slightly unrelated comment. guess you might have to fix your config value for hoodie.memory.compaction.fraction. It is expected to be a fraction like 0.3 or 0.5 etc. From your desc,

[GitHub] [hudi] nsivabalan commented on issue #2533: [SUPPORT] Found in-flight commits after time :20210129225133, please rollback greater commits first

2021-02-17 Thread GitBox
nsivabalan commented on issue #2533: URL: https://github.com/apache/hudi/issues/2533#issuecomment-781035419 yeah, unfortunately we don't expose this as of today. We have a tracking ticket https://issues.apache.org/jira/browse/HUDI-1593. We will try to get it in next release. @abhijeetk

[jira] [Updated] (HUDI-1593) Add support for "show restore" in hudi-cli

2021-02-17 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1593: -- Labels: user-support-issues (was: ) > Add support for "show restore" in hudi-cli >

[GitHub] [hudi] nsivabalan commented on issue #2563: [Feature Request] Full Schema Evolution

2021-02-17 Thread GitBox
nsivabalan commented on issue #2563: URL: https://github.com/apache/hudi/issues/2563#issuecomment-781034857 yeah. we don't support full schema evolution as of today. But definitely its in our radar. https://issues.apache.org/jira/browse/HUDI-631 https://issues.apache.org/jira/browse/HUD

[GitHub] [hudi] nsivabalan commented on issue #2522: [SUPPORT] Avoid UPSERT unchanged records from source

2021-02-17 Thread GitBox
nsivabalan commented on issue #2522: URL: https://github.com/apache/hudi/issues/2522#issuecomment-781032755 Feel free to reach out to us if you need any more info. This is an automated message from the Apache Git Service. To

[GitHub] [hudi] nsivabalan closed issue #2522: [SUPPORT] Avoid UPSERT unchanged records from source

2021-02-17 Thread GitBox
nsivabalan closed issue #2522: URL: https://github.com/apache/hudi/issues/2522 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [hudi] nsivabalan commented on issue #2522: [SUPPORT] Avoid UPSERT unchanged records from source

2021-02-17 Thread GitBox
nsivabalan commented on issue #2522: URL: https://github.com/apache/hudi/issues/2522#issuecomment-781032648 @sleapfish : closing this out as we could achieve this using existing recordPayload with one column to determine source ordering. We have a tracking ticket for multiple preCombine ke

[GitHub] [hudi] codecov-io edited a comment on pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2359: URL: https://github.com/apache/hudi/pull/2359#issuecomment-751328228 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=h1) Report > Merging [#2359](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=desc) (ab25ff9) in

[GitHub] [hudi] codecov-io edited a comment on pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2359: URL: https://github.com/apache/hudi/pull/2359#issuecomment-751328228 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=h1) Report > Merging [#2359](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=desc) (ab25ff9) in

[GitHub] [hudi] codecov-io edited a comment on pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2359: URL: https://github.com/apache/hudi/pull/2359#issuecomment-751328228 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=h1) Report > Merging [#2359](https://codecov.io/gh/apache/hudi/pull/2359?src=pr&el=desc) (ab25ff9) in

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578081554 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -181,7 +171,7 @@ public boolean commit

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578081171 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -236,17 +226,16 @@ protected void sync

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578080521 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -598,6 +588,9 @@ public HoodieRestoreM

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578080521 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -598,6 +588,9 @@ public HoodieRestoreM

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578080260 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HeartbeatUtils.java ## @@ -0,0 +1,95 @@ +/* + * Licensed to the A

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578080111 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -912,5 +929,6 @@ public void close() {

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578079857 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -617,11 +610,8 @@ public HoodieCleanMe

[GitHub] [hudi] ZhangChaoming commented on a change in pull request #2540: [HUDI-1586] [Common Core] [Flink Integration] Reduce the coupling of hadoop.

2021-02-17 Thread GitBox
ZhangChaoming commented on a change in pull request #2540: URL: https://github.com/apache/hudi/pull/2540#discussion_r578079792 ## File path: hudi-flink/pom.xml ## @@ -157,7 +157,7 @@ org.apache.hadoop hadoop-common - compile + provided Review comm

[GitHub] [hudi] ZhangChaoming commented on a change in pull request #2540: [HUDI-1586] [Common Core] [Flink Integration] Reduce the coupling of hadoop.

2021-02-17 Thread GitBox
ZhangChaoming commented on a change in pull request #2540: URL: https://github.com/apache/hudi/pull/2540#discussion_r578078710 ## File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java ## @@ -80,9 +80,6 @@ private static final PathFilter ALLOW_ALL_FILTER

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578071340 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -424,15 +411,15 @@ protected void post

[GitHub] [hudi] n3nash commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
n3nash commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r578071271 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java ## @@ -236,17 +226,16 @@ protected void sync

[GitHub] [hudi] codecov-io edited a comment on pull request #2565: [HUDI-1611] Added a configuration to allow specific directories to be filtered out during Metadata Table bootstrap.

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2565: URL: https://github.com/apache/hudi/pull/2565#issuecomment-777120159 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2565?src=pr&el=h1) Report > Merging [#2565](https://codecov.io/gh/apache/hudi/pull/2565?src=pr&el=desc) (6cb664d) in

[GitHub] [hudi] yanghua commented on pull request #2534: [HUDI-1315] Adding builder for HoodieTableMetaClient initialization

2021-02-17 Thread GitBox
yanghua commented on pull request #2534: URL: https://github.com/apache/hudi/pull/2534#issuecomment-780978653 @nsivabalan There is a conflicting file. This is an automated message from the Apache Git Service. To respond to th

[hudi] branch master updated: [HUDI-1582] Throw an exception when syncHoodieTable() fails, with RuntimeException (#2536)

2021-02-17 Thread nagarwal
This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new b0010bf [HUDI-1582] Throw an exception when syn

[GitHub] [hudi] n3nash merged pull request #2536: [HUDI-1582] Throw an exception when syncHoodieTable() fails, with RuntimeException

2021-02-17 Thread GitBox
n3nash merged pull request #2536: URL: https://github.com/apache/hudi/pull/2536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] n3nash commented on pull request #2536: [HUDI-1582] Throw an exception when syncHoodieTable() fails, with RuntimeException

2021-02-17 Thread GitBox
n3nash commented on pull request #2536: URL: https://github.com/apache/hudi/pull/2536#issuecomment-780973665 LGTM, merging this. This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hudi] n3nash commented on pull request #2565: [HUDI-1611] Added a configuration to allow specific directories to be filtered out during Metadata Table bootstrap.

2021-02-17 Thread GitBox
n3nash commented on pull request #2565: URL: https://github.com/apache/hudi/pull/2565#issuecomment-780972942 @prashantwason can you take care of @nsivabalan comment and concern, we can merge then. This is an automated messag

[GitHub] [hudi] n3nash closed pull request #2565: [HUDI-1611] Added a configuration to allow specific directories to be filtered out during Metadata Table bootstrap.

2021-02-17 Thread GitBox
n3nash closed pull request #2565: URL: https://github.com/apache/hudi/pull/2565 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] rubenssoto commented on issue #2509: [SUPPORT]Hudi saves TimestampType as bigInt

2021-02-17 Thread GitBox
rubenssoto commented on issue #2509: URL: https://github.com/apache/hudi/issues/2509#issuecomment-780850528 @nsivabalan it worked but I think a view it is not a good solution, because we will have a maintenence problem. It is not a Hudi fault, so we need to wait for athena, but I thi

[GitHub] [hudi] vinothchandar commented on a change in pull request #2494: [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2494: URL: https://github.com/apache/hudi/pull/2494#discussion_r577798704 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java ## @@ -112,13 +113,59 @@ private void initIfNeeded() {

[GitHub] [hudi] nsivabalan commented on a change in pull request #2359: [HUDI-1486] Remove inflight rollback in hoodie writer

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2359: URL: https://github.com/apache/hudi/pull/2359#discussion_r577633594 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java ## @@ -48,6 +49,7 @@ protected final trans

[GitHub] [hudi] vinothchandar commented on a change in pull request #2475: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2475: URL: https://github.com/apache/hudi/pull/2475#discussion_r577755265 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -74,6 +78,19 @@ class DefaultSource extends Re

[hudi] branch master updated: [HUDI-1381] Schedule compaction based on time elapsed (#2260)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9431aab [HUDI-1381] Schedule compaction based on

[GitHub] [hudi] vinothchandar merged pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-02-17 Thread GitBox
vinothchandar merged pull request #2260: URL: https://github.com/apache/hudi/pull/2260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] konradwudkowski edited a comment on issue #2265: Arrays with nulls in them result in broken parquet files

2021-02-17 Thread GitBox
konradwudkowski edited a comment on issue #2265: URL: https://github.com/apache/hudi/issues/2265#issuecomment-780589054 Hi @nsivabalan I've not tried any previous version, I've used hudi for the first time now but we're in prod with it. I noticed the problem in a test where this scenario w

[GitHub] [hudi] konradwudkowski commented on issue #2265: Arrays with nulls in them result in broken parquet files

2021-02-17 Thread GitBox
konradwudkowski commented on issue #2265: URL: https://github.com/apache/hudi/issues/2265#issuecomment-780589054 Hi @nsivabalan I've not tried any previous version, I'm using hudi for the first time now but we're in prod with it. I noticed the problem in a test where this scenario was an e

[GitHub] [hudi] nsivabalan commented on pull request #2388: [HUDI-1353] add incremental timeline support for pending clustering ops

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2388: URL: https://github.com/apache/hudi/pull/2388#issuecomment-780561687 @vinothchandar @n3nash : gentle reminder to review :) its been few days. This is an automated message from the A

[GitHub] [hudi] nsivabalan edited a comment on pull request #2580: [HUDI 1623] Introduce start & end commit times to timeline

2021-02-17 Thread GitBox
nsivabalan edited a comment on pull request #2580: URL: https://github.com/apache/hudi/pull/2580#issuecomment-780559379 Can you go over steps required for migration if any. Also, guess we have to follow up with downgrade step if someone downgrades from 0.8.0 to 0.7.0 for eg. I assume no ad

[GitHub] [hudi] nsivabalan commented on pull request #2580: [HUDI 1623] Introduce start & end commit times to timeline

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2580: URL: https://github.com/apache/hudi/pull/2580#issuecomment-780559379 Can you go over steps required for migration if any. Also, guess we have to follow up with downgrade step if someone downgrades from 0.8.0 to 0.7.0 for eg. I assume no additiona

[GitHub] [hudi] nsivabalan commented on pull request #2580: [HUDI 1623] Introduce start & end commit times to timeline

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2580: URL: https://github.com/apache/hudi/pull/2580#issuecomment-780557022 Can you please update the description and the purpose. This is an automated message from the Apache Git Service

[GitHub] [hudi] nsivabalan commented on pull request #2565: [HUDI-1611] Added a configuration to allow specific directories to be filtered out during Metadata Table bootstrap.

2021-02-17 Thread GitBox
nsivabalan commented on pull request #2565: URL: https://github.com/apache/hudi/pull/2565#issuecomment-780556494 just curious. actual bootstrap should have some config on these lines right? while bootstrapping data to hudi, filter directories based on some predicate. can't we reuse the sam

[GitHub] [hudi] nsivabalan commented on a change in pull request #2565: [HUDI-1611] Added a configuration to allow specific directories to be filtered out during Metadata Table bootstrap.

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2565: URL: https://github.com/apache/hudi/pull/2565#discussion_r577606882 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java ## @@ -326,9 +327,15 @@ private

[GitHub] [hudi] nsivabalan commented on a change in pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2378: URL: https://github.com/apache/hudi/pull/2378#discussion_r577570005 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala ## @@ -77,18 +81,26 @@ object HoodieSparkUtils {

[GitHub] [hudi] nsivabalan commented on a change in pull request #2160: [HUDI-865] Improve Hive Syncing by directly translating avro schema to Hive types

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2160: URL: https://github.com/apache/hudi/pull/2160#discussion_r577552355 ## File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java ## @@ -155,113 +150,75 @@ private static boolean isFie

[GitHub] [hudi] yanghua commented on pull request #2548: [HUDI-1597] remove deprecated spring repos from pom

2021-02-17 Thread GitBox
yanghua commented on pull request #2548: URL: https://github.com/apache/hudi/pull/2548#issuecomment-780513062 > My vote is for just merging these PRs and not do the new versions. The existing artifacts are out there already. +1 for only merging the commit. -

[jira] [Closed] (HUDI-1621) Gets the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1621. -- Resolution: Fixed Fixed via master branch: c4bbcb7f0e83732b63ed7bb81c20470ae8d9a0dc > Gets the parallelism from

[jira] [Assigned] (HUDI-1621) Gets the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-1621: -- Assignee: lamber-ken > Gets the parallelism from context when init StreamWriteOperatorCoordinator > ---

[jira] [Updated] (HUDI-1621) Gets the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1621: --- Fix Version/s: 0.8.0 > Gets the parallelism from context when init StreamWriteOperatorCoordinator > --

[hudi] branch master updated: [HUDI-1621] Gets the parallelism from context when init StreamWriteOperatorCoordinator (#2579)

2021-02-17 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new c4bbcb7 [HUDI-1621] Gets the parallelism from c

[GitHub] [hudi] yanghua merged pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
yanghua merged pull request #2579: URL: https://github.com/apache/hudi/pull/2579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] yanghua commented on a change in pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
yanghua commented on a change in pull request #2579: URL: https://github.com/apache/hudi/pull/2579#discussion_r577556163 ## File path: hudi-flink/src/test/resources/log4j-surefire.properties ## @@ -15,7 +15,7 @@ # See the License for the specific language governing permissions

[GitHub] [hudi] yanghua commented on a change in pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
yanghua commented on a change in pull request #2579: URL: https://github.com/apache/hudi/pull/2579#discussion_r577556163 ## File path: hudi-flink/src/test/resources/log4j-surefire.properties ## @@ -15,7 +15,7 @@ # See the License for the specific language governing permissions

[GitHub] [hudi] nsivabalan commented on a change in pull request #2494: [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2494: URL: https://github.com/apache/hudi/pull/2494#discussion_r577550788 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java ## @@ -112,13 +113,59 @@ private void initIfNeeded() {

[GitHub] [hudi] lamber-ken commented on a change in pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
lamber-ken commented on a change in pull request #2579: URL: https://github.com/apache/hudi/pull/2579#discussion_r577543899 ## File path: hudi-flink/src/test/resources/log4j-surefire.properties ## @@ -15,7 +15,7 @@ # See the License for the specific language governing permissi

[GitHub] [hudi] lamber-ken commented on a change in pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
lamber-ken commented on a change in pull request #2579: URL: https://github.com/apache/hudi/pull/2579#discussion_r577548857 ## File path: hudi-flink/src/test/resources/log4j-surefire.properties ## @@ -15,7 +15,7 @@ # See the License for the specific language governing permissi

[GitHub] [hudi] nsivabalan commented on a change in pull request #2494: [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.

2021-02-17 Thread GitBox
nsivabalan commented on a change in pull request #2494: URL: https://github.com/apache/hudi/pull/2494#discussion_r577544617 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java ## @@ -188,41 +238,42 @@ private synchronized void openFi

[GitHub] [hudi] lamber-ken commented on a change in pull request #2579: [HUDI-1621] Get the parallelism from context when init StreamWriteOperatorCoordinator

2021-02-17 Thread GitBox
lamber-ken commented on a change in pull request #2579: URL: https://github.com/apache/hudi/pull/2579#discussion_r577543899 ## File path: hudi-flink/src/test/resources/log4j-surefire.properties ## @@ -15,7 +15,7 @@ # See the License for the specific language governing permissi

[hudi] branch master updated: [HUDI-1109] Support Spark Structured Streaming read from Hudi table (#2485)

2021-02-17 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3797207 [HUDI-1109] Support Spark Structured Stre

[GitHub] [hudi] vinothchandar merged pull request #2485: [HUDI-1109] Support Spark Structured Streaming read from Hudi table

2021-02-17 Thread GitBox
vinothchandar merged pull request #2485: URL: https://github.com/apache/hudi/pull/2485 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] vinothchandar commented on a change in pull request #2494: [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2494: URL: https://github.com/apache/hudi/pull/2494#discussion_r577537335 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java ## @@ -112,13 +113,59 @@ private void initIfNeeded() {

[GitHub] [hudi] vinothchandar commented on a change in pull request #2496: [HUDI-1554] Introduced buffering for streams in HUDI.

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2496: URL: https://github.com/apache/hudi/pull/2496#discussion_r577531939 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestCopyOnWriteActionExecutor.java ## @@ -375,8 +375,8 @

[GitHub] [hudi] vinothchandar commented on pull request #2496: [HUDI-1554] Introduced buffering for streams in HUDI.

2021-02-17 Thread GitBox
vinothchandar commented on pull request #2496: URL: https://github.com/apache/hudi/pull/2496#issuecomment-780489181 @prashantwason can we get the PR to pass tests? I can take a final pass for landing. it'd be good to get this in ---

[GitHub] [hudi] vinothchandar commented on a change in pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem

2021-02-17 Thread GitBox
vinothchandar commented on a change in pull request #2500: URL: https://github.com/apache/hudi/pull/2500#discussion_r577528143 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java ## @@ -74,20 +75,28 @@ public HoodieLogFileRead

[GitHub] [hudi] vinothchandar commented on pull request #2548: [HUDI-1597] remove deprecated spring repos from pom

2021-02-17 Thread GitBox
vinothchandar commented on pull request #2548: URL: https://github.com/apache/hudi/pull/2548#issuecomment-780483801 @yanghua lets make a call here and move on. My vote is for just merging these PRs and not do the new versions. The existing artifacts are out there already. --

[GitHub] [hudi] codecov-io edited a comment on pull request #2580: [HUDI 1623] Introduce start & end commit times to timeline

2021-02-17 Thread GitBox
codecov-io edited a comment on pull request #2580: URL: https://github.com/apache/hudi/pull/2580#issuecomment-780218110 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2580?src=pr&el=h1) Report > Merging [#2580](https://codecov.io/gh/apache/hudi/pull/2580?src=pr&el=desc) (ff582fe) in