[GitHub] [hudi] bvaradar commented on issue #2203: [SUPPORT] Hive query on HUDI table with partition column in where condition returning no results

2020-10-25 Thread GitBox
bvaradar commented on issue #2203: URL: https://github.com/apache/hudi/issues/2203#issuecomment-716322522 I have seen this issue when partitions are not registered correctly. Can you provide 1. desc formatted tbl_name 2. describe formatted tbl_name partition () 3. Full path of

[GitHub] [hudi] bvaradar commented on issue #2204: [SUPPORT] Hive count(*) query on _rt table failing with exception

2020-10-25 Thread GitBox
bvaradar commented on issue #2204: URL: https://github.com/apache/hudi/issues/2204#issuecomment-716320954 @BalaMahesh : Can you enable debug logging in hive and run the query and attach the full logs ? Thanks, Balaji.V

[GitHub] [hudi] bvaradar commented on issue #2207: Performance issue with Dataset write to S3

2020-10-25 Thread GitBox
bvaradar commented on issue #2207: URL: https://github.com/apache/hudi/issues/2207#issuecomment-716301612 @adnanhb : Concurrent writes are writes happening to the same dataset from different spark applications. In your case, without any additional information, I would guess increasing

[GitHub] [hudi] yanghua merged pull request #2194: [MINOR] Private the NoArgsConstructor of SparkMergeHelper and code clean

2020-10-25 Thread GitBox
yanghua merged pull request #2194: URL: https://github.com/apache/hudi/pull/2194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] codecov-io edited a comment on pull request #2092: [HUDI-1285] Fix merge on read DAG to make docker demo pass

2020-10-25 Thread GitBox
codecov-io edited a comment on pull request #2092: URL: https://github.com/apache/hudi/pull/2092#issuecomment-716293969 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2092?src=pr=h1) Report > Merging [#2092](https://codecov.io/gh/apache/hudi/pull/2092?src=pr=desc) into

[hudi] branch master updated: [MINOR] Private the NoArgsConstructor of SparkMergeHelper and code clean (#2194)

2020-10-25 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e206ddd [MINOR] Private the NoArgsConstructor

[GitHub] [hudi] codecov-io commented on pull request #2092: [HUDI-1285] Fix merge on read DAG to make docker demo pass

2020-10-25 Thread GitBox
codecov-io commented on pull request #2092: URL: https://github.com/apache/hudi/pull/2092#issuecomment-716293969 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2092?src=pr=h1) Report > Merging [#2092](https://codecov.io/gh/apache/hudi/pull/2092?src=pr=desc) into

[hudi] branch master updated: [HUDI-1118] Cleanup rollback files residing in .hoodie folder (#2205)

2020-10-25 Thread nagarwal
This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8545ea3 [HUDI-1118] Cleanup rollback files

[GitHub] [hudi] n3nash merged pull request #2205: [HUDI-1118] Cleanup rollback files residing in .hoodie folder

2020-10-25 Thread GitBox
n3nash merged pull request #2205: URL: https://github.com/apache/hudi/pull/2205 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] n3nash commented on pull request #2205: [HUDI-1118] Cleanup rollback files residing in .hoodie folder

2020-10-25 Thread GitBox
n3nash commented on pull request #2205: URL: https://github.com/apache/hudi/pull/2205#issuecomment-716290307 @vinothchandar the reason is basically what @lw309637554 mentioned. This is an automated message from the Apache

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511709134 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/writer/DFSDeltaWriterAdapter.java ## @@ -40,10 +40,12 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511708914 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordFullPayloadGenerator.java ## @@ -130,43 +132,60 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511708422 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -155,6 +156,42 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511708354 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -155,6 +156,42 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511707782 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/UpsertNode.java ## @@ -23,6 +23,7 @@ import

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511707633 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/configuration/DeltaConfig.java ## @@ -118,6 +125,10 @@ public int

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511707483 ## File path: docker/demo/config/test-suite/complex-dag-cow.yaml ## @@ -93,3 +93,50 @@ second_hive_query: result2: 11900 type: HiveQueryNode

[GitHub] [hudi] n3nash commented on a change in pull request #2172: [HUDI-1338] Adding Delete support to test suite framework

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2172: URL: https://github.com/apache/hudi/pull/2172#discussion_r511707521 ## File path: docker/demo/config/test-suite/complex-dag-cow.yaml ## @@ -93,3 +93,50 @@ second_hive_query: result2: 11900 type: HiveQueryNode

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511706536 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -95,11 +110,19 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511706536 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -95,11 +110,19 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511706427 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -77,6 +82,16 @@ public

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511706331 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/DeltaGenerator.java ## @@ -58,15 +63,15 @@ private static Logger

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511704839 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/configuration/DFSDeltaConfig.java ## @@ -36,15 +36,22 @@ private final Long

[GitHub] [hudi] n3nash commented on a change in pull request #2197: [HUDI-1351] Improvements to the hudi test suite for scalability and repeated testing.

2020-10-25 Thread GitBox
n3nash commented on a change in pull request #2197: URL: https://github.com/apache/hudi/pull/2197#discussion_r511704768 ## File path: hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/configuration/DFSDeltaConfig.java ## @@ -36,15 +36,22 @@ private final Long

[GitHub] [hudi] liujinhui1994 commented on pull request #2122: [HUDI-1274] Make hive synchronization supports hourly partition

2020-10-25 Thread GitBox
liujinhui1994 commented on pull request #2122: URL: https://github.com/apache/hudi/pull/2122#issuecomment-716262613 Sorry, I am late in reply, I will add unit tests soon @vinothchandar This is an automated message from

[GitHub] [hudi] adnanhb opened a new issue #2207: [SUPPORT]

2020-10-25 Thread GitBox
adnanhb opened a new issue #2207: URL: https://github.com/apache/hudi/issues/2207 Hello, this might be a basic question but I am not able to find any guidance anywhere. We are writing approx 8 million records (55 columns per reord) to a hudi dataset which is saved on s3. We are using copy

[GitHub] [hudi] nsivabalan commented on a change in pull request #2206: Adding dedup support for Bulk Insert w/ Rows

2020-10-25 Thread GitBox
nsivabalan commented on a change in pull request #2206: URL: https://github.com/apache/hudi/pull/2206#discussion_r511623934 ## File path: hudi-spark/src/main/java/org/apache/hudi/PreCombineRow.java ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] nsivabalan commented on a change in pull request #2206: Adding dedup support for Bulk Insert w/ Rows

2020-10-25 Thread GitBox
nsivabalan commented on a change in pull request #2206: URL: https://github.com/apache/hudi/pull/2206#discussion_r511623934 ## File path: hudi-spark/src/main/java/org/apache/hudi/PreCombineRow.java ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] nsivabalan opened a new pull request #2206: Adding dedup support for Bulk Insert w/ Rows

2020-10-25 Thread GitBox
nsivabalan opened a new pull request #2206: URL: https://github.com/apache/hudi/pull/2206 ## What is the purpose of the pull request Adding dedup support for Bulk Insert w/ Rows ## Brief change log - Adding dedup support for Bulk Insert w/ Rows - Introduced an

[GitHub] [hudi] lw309637554 commented on pull request #2190: [HUDI-892] RealtimeParquetInputFormat skip adding projection columns if there are no log files

2020-10-25 Thread GitBox
lw309637554 commented on pull request #2190: URL: https://github.com/apache/hudi/pull/2190#issuecomment-716167363 > Okay test fails again. @lw309637554 can you please investigate. > > ``` > [ERROR] Errors: > [ERROR]

[jira] [Commented] (HUDI-1281) deltacommit is not part of ActionType used in HoodieArchivedMetaEntry

2020-10-25 Thread liwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220295#comment-17220295 ] liwei commented on HUDI-1281: - [~satishkotha] hello , i am interested to take this work  :D > deltacommit is

[GitHub] [hudi] lw309637554 commented on a change in pull request #2136: [HUDI-37] Persist the HoodieIndex type in the hoodie.properties file

2020-10-25 Thread GitBox
lw309637554 commented on a change in pull request #2136: URL: https://github.com/apache/hudi/pull/2136#discussion_r511595604 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java ## @@ -124,9 +128,30 @@ public

[GitHub] [hudi] lw309637554 commented on a change in pull request #2136: [HUDI-37] Persist the HoodieIndex type in the hoodie.properties file

2020-10-25 Thread GitBox
lw309637554 commented on a change in pull request #2136: URL: https://github.com/apache/hudi/pull/2136#discussion_r511595506 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java ## @@ -124,9 +128,30 @@ public

[GitHub] [hudi] lw309637554 commented on a change in pull request #2136: [HUDI-37] Persist the HoodieIndex type in the hoodie.properties file

2020-10-25 Thread GitBox
lw309637554 commented on a change in pull request #2136: URL: https://github.com/apache/hudi/pull/2136#discussion_r511595506 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieClient.java ## @@ -124,9 +128,30 @@ public

[GitHub] [hudi] codecov-io edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and BulkSortModes to BulkInsert with Rows

2020-10-25 Thread GitBox
codecov-io edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-716133477 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=h1) Report > Merging [#2049](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=desc) into

[GitHub] [hudi] codecov-io commented on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and BulkSortModes to BulkInsert with Rows

2020-10-25 Thread GitBox
codecov-io commented on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-716133477 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=h1) Report > Merging [#2049](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=desc) into

[GitHub] [hudi] lw309637554 commented on a change in pull request #2136: [HUDI-37] Persist the HoodieIndex type in the hoodie.properties file

2020-10-25 Thread GitBox
lw309637554 commented on a change in pull request #2136: URL: https://github.com/apache/hudi/pull/2136#discussion_r511574134 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/AbstractUpgradeDowngrade.java ## @@ -132,6 +134,8 @@ protected