[GitHub] [incubator-hudi] codecov-io commented on issue #1543: [HUDI-821]:Fix the wrong annotation of JCommander IStringConverter

codecov-io commented on issue #1543: URL: https://github.com/apache/incubator-hudi/pull/1543#issuecomment-616990309 # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1543?src=pr&el=h1) Report > Merging [#1543](https://codecov.io/gh/apache/incubator-hudi/pull/1543?src=pr&el=d

[jira] [Closed] (HUDI-789) Adjust logic of upsert in HDFSParquetImporter

[ https://issues.apache.org/jira/browse/HUDI-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-789. - Resolution: Fixed Fixed via master branch: 84dd9047d3902650d7ff5bc95b9789d6880ca8e2 > Adjust logic of upsert in HD

[jira] [Updated] (HUDI-823) Typo in quick start guide

[ https://issues.apache.org/jira/browse/HUDI-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Wang updated HUDI-823: -- Status: In Progress (was: Open) > Typo in quick start guide > - > >

[jira] [Updated] (HUDI-823) Typo in quick start guide

[ https://issues.apache.org/jira/browse/HUDI-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Wang updated HUDI-823: -- Status: Open (was: New) > Typo in quick start guide > - > > Key:

[jira] [Updated] (HUDI-789) Adjust logic of upsert in HDFSParquetImporter

[ https://issues.apache.org/jira/browse/HUDI-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-789: -- Fix Version/s: 0.6.0 > Adjust logic of upsert in HDFSParquetImporter > --

[jira] [Updated] (HUDI-789) Adjust logic of upsert in HDFSParquetImporter

[ https://issues.apache.org/jira/browse/HUDI-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-789: -- Status: Open (was: New) > Adjust logic of upsert in HDFSParquetImporter > --

[jira] [Created] (HUDI-823) Typo in quick start guide

Lisheng Wang created HUDI-823: - Summary: Typo in quick start guide Key: HUDI-823 URL: https://issues.apache.org/jira/browse/HUDI-823 Project: Apache Hudi (incubating) Issue Type: Bug Co

[incubator-hudi] branch master updated: [HUDI-789]Adjust logic of upsert in HDFSParquetImporter (#1511)

This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/master by this push: new 84dd904 [HUDI-789]Adjust logic of ups

[GitHub] [incubator-hudi] jenu9417 commented on issue #1528: [SUPPORT] Issue while writing to HDFS via hudi. Only `/.hoodie` folder is written.

jenu9417 commented on issue #1528: URL: https://github.com/apache/incubator-hudi/issues/1528#issuecomment-616968792 @vinothchandar Thanks for replying in detail. As you pointed out, premature termination of job seems to be the problem. Since this was a POC and dry run, I was using a

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1511: [HUDI-789]Adjust logic of upsert in HDFSParquetImporter

codecov-io edited a comment on issue #1511: URL: https://github.com/apache/incubator-hudi/pull/1511#issuecomment-612848674 # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1511?src=pr&el=h1) Report > Merging [#1511](https://codecov.io/gh/apache/incubator-hudi/pull/1511?src=

[GitHub] [incubator-hudi] shenh062326 commented on issue #1544: [Minor] Update docs for oss_filesystem

shenh062326 commented on issue #1544: URL: https://github.com/apache/incubator-hudi/pull/1544#issuecomment-616952742 @leesf please help review this patch. This is an automated message from the Apache Git Service. To respond

[GitHub] [incubator-hudi] shenh062326 opened a new pull request #1544: [Minor] Update docs for oss_filesystem

shenh062326 opened a new pull request #1544: URL: https://github.com/apache/incubator-hudi/pull/1544 ## What is the purpose of the pull request * Update docs for oss_filesystem ## Brief change log - Modify docs/_docs/0_5_oss_filesystem.cn.md - Modify docs/_docs/0_

[GitHub] [incubator-hudi] bvaradar edited a comment on issue #1543: [HUDI-821]:Fix the wrong annotation of JCommander IStringConverter

bvaradar edited a comment on issue #1543: URL: https://github.com/apache/incubator-hudi/pull/1543#issuecomment-616939586 @dengziming : Thanks for your contribution. This is addressing the same issue as https://github.com/apache/incubator-hudi/pull/1525. Do you have any specific comments/co

[GitHub] [incubator-hudi] bvaradar commented on issue #1543: [HUDI-821]:Fix the wrong annotation of JCommander IStringConverter

bvaradar commented on issue #1543: URL: https://github.com/apache/incubator-hudi/pull/1543#issuecomment-616939586 @dengziming : THis is addressing the same issue as https://github.com/apache/incubator-hudi/pull/1525. Do you have any specific comments/concerns with https://github.com/apache

[GitHub] [incubator-hudi] yanghua commented on issue #1541: [Minor] Add ability to specify time unit for TimestampBasedKeyGenerator

yanghua commented on issue #1541: URL: https://github.com/apache/incubator-hudi/pull/1541#issuecomment-616938485 Hi @afilipchik thanks for your contribution. The correct prefix is "MINOR". IMHO, the change of this PR should be filed in JIRA. ---

[jira] [Updated] (HUDI-821) Fix the wrong annotation of JCommander IStringConverter

[ https://issues.apache.org/jira/browse/HUDI-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-821: Labels: pull-request-available (was: ) > Fix the wrong annotation of JCommander IStringConverter > -

[GitHub] [incubator-hudi] dengziming opened a new pull request #1543: HUDI-821:Fix the wrong annotation of JCommander IStringConverter

dengziming opened a new pull request #1543: URL: https://github.com/apache/incubator-hudi/pull/1543 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpo

[jira] [Updated] (HUDI-766) Update Apache Hudi website with usage info about HoodieMultiTableDeltaStreamer

[ https://issues.apache.org/jira/browse/HUDI-766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-766: -- Status: In Progress (was: Open) > Update Apache Hudi website with usage info about HoodieMultiTa

[jira] [Updated] (HUDI-803) Improve Unit test coverage of HoodieAvroUtils around default values

[ https://issues.apache.org/jira/browse/HUDI-803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-803: -- Status: In Progress (was: Open) > Improve Unit test coverage of HoodieAvroUtils around default v

[jira] [Updated] (HUDI-769) Write blog about HoodieMultiTableDeltaStreamer in cwiki

[ https://issues.apache.org/jira/browse/HUDI-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-769: -- Status: In Progress (was: Open) > Write blog about HoodieMultiTableDeltaStreamer in cwiki >

[jira] [Updated] (HUDI-796) Rewrite DedupeSparkJob.scala without considering the _hoodie_commit_time

[ https://issues.apache.org/jira/browse/HUDI-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-796: -- Status: In Progress (was: Open) > Rewrite DedupeSparkJob.scala without considering the _hoodie_c

[jira] [Commented] (HUDI-480) Support a querying delete data methond in incremental view

[ https://issues.apache.org/jira/browse/HUDI-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088263#comment-17088263 ] vinoyang commented on HUDI-480: --- [~chenxiang] Glad to hear this. We are busy with other thing

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #254

See Changes: -- [...truncated 2.38 KB...] /home/jenkins/tools/maven/apache-maven-3.5.4/conf: logging settings.xml toolchains.xml /home/jenkins/tools/maven/apache-maven-3.5.

[incubator-hudi] branch master updated: [HUDI-371] Supporting hive combine input format for realtime tables (#1503)

This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/master by this push: new 332072b [HUDI-371] Supporting hive com

[jira] [Commented] (HUDI-83) Map Timestamp type in spark to corresponding Timestamp type in Hive during Hive sync

[ https://issues.apache.org/jira/browse/HUDI-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088258#comment-17088258 ] cdmikechen commented on HUDI-83: [~vinoth] [~arw357] [~uditme] [~xleesf] I have custom a new

[jira] [Assigned] (HUDI-83) Map Timestamp type in spark to corresponding Timestamp type in Hive during Hive sync

[ https://issues.apache.org/jira/browse/HUDI-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cdmikechen reassigned HUDI-83: -- Assignee: cdmikechen > Map Timestamp type in spark to corresponding Timestamp type in Hive during > Hive

[jira] [Created] (HUDI-822) Decouple hoodie related methods with Hoodie Input Formats

Yanjia Gary Li created HUDI-822: --- Summary: Decouple hoodie related methods with Hoodie Input Formats Key: HUDI-822 URL: https://issues.apache.org/jira/browse/HUDI-822 Project: Apache Hudi (incubating)

[GitHub] [incubator-hudi] pratyakshsharma commented on issue #765: [WIP] Fix KafkaAvroSource to use the latest schema

pratyakshsharma commented on issue #765: URL: https://github.com/apache/incubator-hudi/pull/765#issuecomment-616917937 @haiminh87 Still working on this? This is an automated message from the Apache Git Service. To respond to

[jira] [Created] (HUDI-821) Fix the wrong annotation of JCommander IStringConverter

dengziming created HUDI-821: --- Summary: Fix the wrong annotation of JCommander IStringConverter Key: HUDI-821 URL: https://issues.apache.org/jira/browse/HUDI-821 Project: Apache Hudi (incubating) Is

[jira] [Commented] (HUDI-351) Implement Range + Bloom Filter checking in one go to improve speed of index

[ https://issues.apache.org/jira/browse/HUDI-351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088208#comment-17088208 ] sivabalan narayanan commented on HUDI-351: -- [~vinoth]: Do you think we still need

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class

codecov-io edited a comment on issue #1538: URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616904151 # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report > Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=

[GitHub] [incubator-hudi] codecov-io commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class

codecov-io commented on issue #1538: URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616904151 # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report > Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=d

[incubator-hudi] branch master updated (ddd105b -> 2a2f31d)

This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from ddd105b [HUDI-772] Make UserDefinedBulkInsertPartitioner configurable for DataSource (#1500) add 2a2

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

vinothchandar commented on a change in pull request #1512: URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411783820 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileFormat.java ## @@ -22,7 +22,7 @@ * Hoodie file format. */

[jira] [Updated] (HUDI-820) Fix bug in repair corrupted clean files command

[ https://issues.apache.org/jira/browse/HUDI-820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-820: Labels: pull-request-available (was: ) > Fix bug in repair corrupted clean files command > -

[GitHub] [incubator-hudi] bvaradar opened a new pull request #1542: [HUDI-820] cleaner repair command should only inspect clean metadata files

bvaradar opened a new pull request #1542: URL: https://github.com/apache/incubator-hudi/pull/1542 @lamber-ken : This is something I missed when reviewing cleaner repair code changes. The repair command has a serious bug in that it might delete inflight instants of other actions. c

[jira] [Commented] (HUDI-480) Support a querying delete data methond in incremental view

[ https://issues.apache.org/jira/browse/HUDI-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088183#comment-17088183 ] cdmikechen commented on HUDI-480: - [~vinoth] [~yanghua] Maybe I can open a RFC and write do

[GitHub] [incubator-hudi] vinothchandar commented on issue #1528: [SUPPORT] Issue while writing to HDFS via hudi. Only `/.hoodie` folder is written.

vinothchandar commented on issue #1528: URL: https://github.com/apache/incubator-hudi/issues/1528#issuecomment-616877781 @jenu9417 Thanks for taking the time to report this. a) is weird.. The logs do indicate that tasks got scheduled atleast.. but I think the job died before getting

[jira] [Created] (HUDI-820) Fix bug in repair corrupted clean files command

Balaji Varadarajan created HUDI-820: --- Summary: Fix bug in repair corrupted clean files command Key: HUDI-820 URL: https://issues.apache.org/jira/browse/HUDI-820 Project: Apache Hudi (incubating)

[GitHub] [incubator-hudi] afilipchik opened a new pull request #1541: [Minor] Add ability to specify time unit for TimestampBasedKeyGenerator

afilipchik opened a new pull request #1541: URL: https://github.com/apache/incubator-hudi/pull/1541 ## What is the purpose of the pull request Adding a way to specify any source time unit for TimestampBasedKeyGenerator. Properties probably need some refactoring, kept unix timestam

[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer

[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088169#comment-17088169 ] renyi.bao commented on HUDI-760: [~vbalaji] thanks for your guidance, if I understand it co

[incubator-hudi] branch asf-site updated: Travis CI build asf-site

This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 9b2b8d4 Travis CI build asf-site 9b

[jira] [Updated] (HUDI-316) Improve performance of HbaseIndex puts by repartitioning WriteStatus and using rate limiter instead of sleep()

[ https://issues.apache.org/jira/browse/HUDI-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-316: Labels: pull-request-available (was: ) > Improve performance of HbaseIndex puts by repartitioning Wr

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1484: [HUDI-316] : Hbase qps repartition writestatus

satishkotha commented on a change in pull request #1484: URL: https://github.com/apache/incubator-hudi/pull/1484#discussion_r411749174 ## File path: hudi-client/src/main/java/org/apache/hudi/index/hbase/HBaseIndex.java ## @@ -322,66 +347,94 @@ private boolean checkIfValidCommit

[jira] [Assigned] (HUDI-819) missing write status in MergeOnReadLazyInsertIterable

[ https://issues.apache.org/jira/browse/HUDI-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish reassigned HUDI-819: --- Assignee: satish > missing write status in MergeOnReadLazyInsertIterable > ---

[jira] [Updated] (HUDI-819) missing write status in MergeOnReadLazyInsertIterable

[ https://issues.apache.org/jira/browse/HUDI-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-819: Labels: pull-request-available (was: ) > missing write status in MergeOnReadLazyInsertIterable > ---

[GitHub] [incubator-hudi] satishkotha opened a new pull request #1540: [HUDI-819] Fix a bug with MergeOnReadLazyInsertIterable.

satishkotha opened a new pull request #1540: URL: https://github.com/apache/incubator-hudi/pull/1540 ## What is the purpose of the pull request Variable declared here [1] masks protected statuses variable. So although hoodie writes data, will not include WriteStatus in the completed

[jira] [Created] (HUDI-819) missing write status in MergeOnReadLazyInsertIterable

satish created HUDI-819: --- Summary: missing write status in MergeOnReadLazyInsertIterable Key: HUDI-819 URL: https://issues.apache.org/jira/browse/HUDI-819 Project: Apache Hudi (incubating) Issue Type:

[GitHub] [incubator-hudi] vingov commented on issue #1526: [HUDI-1526] Add pyspark example in quickstart

vingov commented on issue #1526: URL: https://github.com/apache/incubator-hudi/pull/1526#issuecomment-616768714 @vinothchandar - This is similar to the blog post draft I have prepared, which explains the usage of the hudi reader/writer with pyspark. I will review the example code. @

[GitHub] [incubator-hudi] lamber-ken commented on issue #1491: [SUPPORT] OutOfMemoryError during upsert 53M records

lamber-ken commented on issue #1491: URL: https://github.com/apache/incubator-hudi/issues/1491#issuecomment-616729550 > @lamber-ken since this has come up a few times, worth tracking a jira for 0.6 that can help get a better default for this? Agree, https://issues.apache.org/jira/bro

[jira] [Updated] (HUDI-818) Optimize the default value of hoodie.memory.merge.max.size option

[ https://issues.apache.org/jira/browse/HUDI-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lamber-ken updated HUDI-818: Fix Version/s: 0.6.0 > Optimize the default value of hoodie.memory.merge.max.size option > --

[jira] [Updated] (HUDI-818) Optimize the default value of hoodie.memory.merge.max.size option

[ https://issues.apache.org/jira/browse/HUDI-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lamber-ken updated HUDI-818: Status: Open (was: New) > Optimize the default value of hoodie.memory.merge.max.size option > --

[jira] [Created] (HUDI-818) Optimize the default value of hoodie.memory.merge.max.size option

lamber-ken created HUDI-818: --- Summary: Optimize the default value of hoodie.memory.merge.max.size option Key: HUDI-818 URL: https://issues.apache.org/jira/browse/HUDI-818 Project: Apache Hudi (incubating)

[jira] [Commented] (HUDI-773) Hudi On Azure Data Lake Storage V2

[ https://issues.apache.org/jira/browse/HUDI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087994#comment-17087994 ] Yanjia Gary Li commented on HUDI-773: - [~sasikumar.venkat] I haven't tried Databricks S

[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1515: [HUDI-795] Ignoring missing aux folder

pratyakshsharma commented on a change in pull request #1515: URL: https://github.com/apache/incubator-hudi/pull/1515#discussion_r411592119 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieCommitArchiveLog.java ## @@ -219,14 +220,29 @@ private boolean delete

[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

lamber-ken commented on a change in pull request #1512: URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411589627 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java ## @@ -131,6 +131,9 @@ public static void createHoo

[GitHub] [incubator-hudi] afilipchik commented on a change in pull request #1516: [HUDI-784] Adressing issue with log reader on GCS

afilipchik commented on a change in pull request #1516: URL: https://github.com/apache/incubator-hudi/pull/1516#discussion_r411560192 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java ## @@ -79,6 +79,7 @@ this.inputStream

[jira] [Updated] (HUDI-801) Add a way to postprocess schema after it is loaded from the schema provider

[ https://issues.apache.org/jira/browse/HUDI-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-801: Labels: pull-request-available (was: ) > Add a way to postprocess schema after it is loaded from the

[GitHub] [incubator-hudi] afilipchik commented on a change in pull request #1524: [HUDI-801] Adding a way to post process schema after it is fetched

afilipchik commented on a change in pull request #1524: URL: https://github.com/apache/incubator-hudi/pull/1524#discussion_r411546036 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/TestSchemaPostProcessor.java ## @@ -0,0 +1,61 @@ +package org.apache.hudi.

[jira] [Updated] (HUDI-817) Wrong index filter condition check in HoodieGlobalBloomIndex

[ https://issues.apache.org/jira/browse/HUDI-817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-817: Labels: pull-request-available (was: ) > Wrong index filter condition check in HoodieGlobalBloomInde

[GitHub] [incubator-hudi] nsivabalan commented on issue #1537: [HUDI-817] fixed building IndexFileFilter with a wrong condition in Hood…

nsivabalan commented on issue #1537: URL: https://github.com/apache/incubator-hudi/pull/1537#issuecomment-616687122 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-hudi] vinothchandar commented on issue #1526: [HUDI-1526] Add pyspark example in quickstart

vinothchandar commented on issue #1526: URL: https://github.com/apache/incubator-hudi/pull/1526#issuecomment-616686836 @vingov does this supercede your work? Or you could add more on top? Trying to understand how’d these two are related.. In any case, do you mind reviewing this sinc

[jira] [Created] (HUDI-817) Wrong index filter condition check in HoodieGlobalBloomIndex

sivabalan narayanan created HUDI-817: Summary: Wrong index filter condition check in HoodieGlobalBloomIndex Key: HUDI-817 URL: https://issues.apache.org/jira/browse/HUDI-817 Project: Apache Hudi (

[GitHub] [incubator-hudi] vinothchandar commented on issue #1491: [SUPPORT] OutOfMemoryError during upsert 53M records

vinothchandar commented on issue #1491: URL: https://github.com/apache/incubator-hudi/issues/1491#issuecomment-616684234 @lamber-ken since this has come up a few times, worth tracking a jira for 0.6 that can help get a better default for this?

[GitHub] [incubator-hudi] n3nash commented on issue #1503: [HUDI-371] : Supporting combine input format RT tables

n3nash commented on issue #1503: URL: https://github.com/apache/incubator-hudi/pull/1503#issuecomment-616684125 @vinothchandar no new class, for the existing class -> https://github.com/apache/incubator-hudi/blob/master/LICENSE#L206

[GitHub] [incubator-hudi] tooptoop4 opened a new issue #1539: [SUPPORT] Migration new inputformat for hive?

tooptoop4 opened a new issue #1539: URL: https://github.com/apache/incubator-hudi/issues/1539 https://cwiki.apache.org/confluence/display/HUDI/Migration+Guide+From+com.uber.hoodie+to+org.apache.hudi mentions 2 conflicting names for Read Optimized: View Type | Pre v0.5.0 Input Fo

[GitHub] [incubator-hudi] afilipchik commented on a change in pull request #1515: [HUDI-795] Ignoring missing aux folder

afilipchik commented on a change in pull request #1515: URL: https://github.com/apache/incubator-hudi/pull/1515#discussion_r411536924 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieCommitArchiveLog.java ## @@ -219,14 +220,23 @@ private boolean deleteArchi

[incubator-hudi] branch hudi_test_suite_refactor updated (d29e41e -> 6465dc4)

This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a change to branch hudi_test_suite_refactor in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. discard d29e41e [HUDI-394] Provide a basic implementation of test suite add 6465dc4 [HUDI-

[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1513: [HUDI-793] Adding proper default to hudi metadata fields and proper handling to rewrite routine

pratyakshsharma commented on issue #1513: URL: https://github.com/apache/incubator-hudi/pull/1513#issuecomment-616657717 > Sure. https://issues.apache.org/jira/browse/HUDI-803 tracks this. https://github.com/apache/incubator-hudi/pull/1538 is raised for this. ---

[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class

pratyakshsharma commented on issue #1538: URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616650031 @vinothchandar Please take a pass. This is an automated message from the Apache Git Service. To respond

[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class

pratyakshsharma commented on issue #1538: URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616649362 Few observations related to issues we have faced recently: 1. If we specify \"default\": null in string schema for a field or specify NullNode.getInstance() for d

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1516: [HUDI-784] Adressing issue with log reader on GCS

bvaradar commented on a change in pull request #1516: URL: https://github.com/apache/incubator-hudi/pull/1516#discussion_r411494903 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java ## @@ -59,7 +59,7 @@ private final FSDataI

[GitHub] [incubator-hudi] pratyakshsharma opened a new pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class

pratyakshsharma opened a new pull request #1538: URL: https://github.com/apache/incubator-hudi/pull/1538 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the

[jira] [Updated] (HUDI-803) Improve Unit test coverage of HoodieAvroUtils around default values

[ https://issues.apache.org/jira/browse/HUDI-803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-803: Labels: pull-request-available (was: ) > Improve Unit test coverage of HoodieAvroUtils around defaul

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1515: [HUDI-795] Ignoring missing aux folder

bvaradar commented on a change in pull request #1515: URL: https://github.com/apache/incubator-hudi/pull/1515#discussion_r411490140 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieCommitArchiveLog.java ## @@ -219,14 +220,23 @@ private boolean deleteArchive

[jira] [Commented] (HUDI-773) Hudi On Azure Data Lake Storage V2

[ https://issues.apache.org/jira/browse/HUDI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087869#comment-17087869 ] Vinoth Chandar commented on HUDI-773: - [~sasikumar.venkat] Happy to work with you and g

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

bvaradar commented on a change in pull request #1512: URL: https://github.com/apache/incubator-hudi/pull/1512#discussion_r411480163 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java ## @@ -131,6 +131,9 @@ public static void createHoodi

[GitHub] [incubator-hudi] vinothchandar commented on issue #1537: [MINOR] fixed building IndexFileFilter with a wrong condition in Hood…

vinothchandar commented on issue #1537: URL: https://github.com/apache/incubator-hudi/pull/1537#issuecomment-616637515 thanks for the catch @Jecarm .. @nsivabalan can you please review this. This deserves a JIRA since its an actual bug fix.. (performance should improve, correctness s

[GitHub] [incubator-hudi] vinothchandar commented on issue #1536: [HUDI-816] Fix MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

vinothchandar commented on issue #1536: URL: https://github.com/apache/incubator-hudi/pull/1536#issuecomment-616635927 an accompanying test case would be great! This is an automated message from the Apache Git Service. To res

[GitHub] [incubator-hudi] vinothchandar commented on issue #1503: [HUDI-371] : Supporting combine input format RT tables

vinothchandar commented on issue #1503: URL: https://github.com/apache/incubator-hudi/pull/1503#issuecomment-616634319 @bvaradar @n3nash any code that is reused from other projects here? (asking since this is Hive and combine input splits).. ---

[incubator-hudi] branch master updated: [HUDI-772] Make UserDefinedBulkInsertPartitioner configurable for DataSource (#1500)

This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/master by this push: new ddd105b [HUDI-772] Make UserDefinedBulk

[GitHub] [incubator-hudi] bvaradar commented on issue #1512: [HUDI-763] Add hoodie.table.base.file.format option to hoodie.properties file

bvaradar commented on issue #1512: URL: https://github.com/apache/incubator-hudi/pull/1512#issuecomment-616631319 @lamber-ken : Storing per-partition specific metadata in hoodie.properties wont work as we are not versioning hoodie.properties. There is no atomicity guarantees across differe

[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer

[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087835#comment-17087835 ] Balaji Varadarajan commented on HUDI-760: -  Hi [~baobaoyeye] : Sorry for the delay.

[jira] [Updated] (HUDI-760) Remove Rolling Stat management from Hudi Writer

[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Varadarajan updated HUDI-760: Status: Open (was: New) > Remove Rolling Stat management from Hudi Writer >

[GitHub] [incubator-hudi] Jecarm opened a new pull request #1537: [MINOR] fixed building IndexFileFilter with a wrong condition in Hood…

Jecarm opened a new pull request #1537: URL: https://github.com/apache/incubator-hudi/pull/1537 #672 What is the purpose of the pull request fixed bug, when building IndexFileFilter with a wrong condition in HoodieGlobalBloomIndex class ## Brief change log use a wrong config p

[jira] [Assigned] (HUDI-815) Typo in Demo's document

[ https://issues.apache.org/jira/browse/HUDI-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Wang reassigned HUDI-815: - Assignee: Lisheng Wang > Typo in Demo's document > --- > > Key

[jira] [Updated] (HUDI-816) Fix MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

[ https://issues.apache.org/jira/browse/HUDI-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-816: --- Summary: Fix MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678 (was: Fixed MAX

[jira] [Resolved] (HUDI-815) Typo in Demo's document

[ https://issues.apache.org/jira/browse/HUDI-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf resolved HUDI-815. Fix Version/s: 0.6.0 Resolution: Fixed > Typo in Demo's document > --- > >

[jira] [Updated] (HUDI-816) Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

[ https://issues.apache.org/jira/browse/HUDI-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-816: --- Status: Open (was: New) > Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not > work due to HU

[jira] [Updated] (HUDI-816) Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

[ https://issues.apache.org/jira/browse/HUDI-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-816: --- Fix Version/s: 0.6.0 > Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not > work due to HUDI-6

[GitHub] [incubator-hudi] leesf commented on issue #1536: [HUDI-816] Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

leesf commented on issue #1536: URL: https://github.com/apache/incubator-hudi/pull/1536#issuecomment-616533827 @lamber-ken Thanks for reporting this, please take a look when you are free. Thanks This is an automated message

[jira] [Updated] (HUDI-816) Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

[ https://issues.apache.org/jira/browse/HUDI-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-816: Labels: pull-request-available (was: ) > Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTI

[GitHub] [incubator-hudi] leesf opened a new pull request #1536: [HUDI-816] Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

leesf opened a new pull request #1536: URL: https://github.com/apache/incubator-hudi/pull/1536 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpos

[jira] [Updated] (HUDI-816) Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

[ https://issues.apache.org/jira/browse/HUDI-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-816: --- Component/s: (was: Writer Core) > Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not > work

[jira] [Created] (HUDI-816) Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678

leesf created HUDI-816: -- Summary: Fixed MAX_MEMORY_FOR_MERGE_PROP and MAX_MEMORY_FOR_COMPACTION_PROP do not work due to HUDI-678 Key: HUDI-816 URL: https://issues.apache.org/jira/browse/HUDI-816 Project: Apache

[GitHub] [incubator-hudi] wangxianghu commented on issue #1535: [MINOR]Remove reduntant code and fix typo in HoodieDefaultTimeline

wangxianghu commented on issue #1535: URL: https://github.com/apache/incubator-hudi/pull/1535#issuecomment-616526186 hi @yanghua, could you please take a look, thanks This is an automated message from the Apache Git Service.

[jira] [Commented] (HUDI-773) Hudi On Azure Data Lake Storage V2

[ https://issues.apache.org/jira/browse/HUDI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087672#comment-17087672 ] Sasikumar Venkatesh commented on HUDI-773: -- My Cluster is setup on Databricks. I h

[GitHub] [incubator-hudi] wangxianghu opened a new pull request #1535: [MINOR]Remove reduntant code and fix typo in HoodieDefaultTimeline

wangxianghu opened a new pull request #1535: URL: https://github.com/apache/incubator-hudi/pull/1535 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purp

[incubator-hudi] branch asf-site updated: [HUDI-815] Fix typo(Kakfa -> Kafka) (#1534)

This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 2bba463 [HUDI-815] Fix typo(Kakfa ->

[jira] [Updated] (HUDI-815) Typo in Demo's document

[ https://issues.apache.org/jira/browse/HUDI-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-815: Labels: pull-request-available (was: ) > Typo in Demo's document > --- > >

  1   2   >