[GitHub] [hudi] satishkotha commented on a change in pull request #3869: [HUDI-1937] Rollback unfinished replace commit to allow updates

2021-10-29 Thread GitBox
satishkotha commented on a change in pull request #3869: URL: https://github.com/apache/hudi/pull/3869#discussion_r739605652 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java ## @@ -117,7 +119,24 @@

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * d022aa7a5bd94492c7c3e96dc5b1288268520087 UNKNOWN * 9add65c8a2bc32aa62be2b0f0f8f711b3471a422 Azure: [CANCELED](https://dev.azure.com/apache-hudi

[GitHub] [hudi] hudi-bot edited a comment on pull request #3519: [DO NOT MERGE] 0.9.0 release patch for flink

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3519: URL: https://github.com/apache/hudi/pull/3519#issuecomment-903204631 ## CI report: * d022aa7a5bd94492c7c3e96dc5b1288268520087 UNKNOWN * 53132e5a5914a3ed00cf75706ed19e62ee2056b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi

[GitHub] [hudi] nsivabalan commented on pull request #3889: [HUDI-2443] Hudi KVComparator for all HFile writer usages

2021-10-29 Thread GitBox
nsivabalan commented on pull request #3889: URL: https://github.com/apache/hudi/pull/3889#issuecomment-955137258 @umehrot2 : Can you loop in Ryan to review the patch. May I know how to test this out ? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][RFC-28]support z-order for hudi

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 133379deca564ca42f10a1f3e59bb4aa17d80964 UNKNOWN * 42fb7aa4d82b8925ed7643c7ab9deae96f558145 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-

[GitHub] [hudi] hudi-bot edited a comment on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-954503596 ## CI report: * cefebbf684ef4f5e4f5075422ac4407b3a4dcd65 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] nsivabalan commented on a change in pull request #3889: [HUDI-2443] Hudi KVComparator for all HFile writer usages

2021-10-29 Thread GitBox
nsivabalan commented on a change in pull request #3889: URL: https://github.com/apache/hudi/pull/3889#discussion_r739601400 ## File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HoodieKVComparator.java ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] hudi-bot edited a comment on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-954503596 ## CI report: * cefebbf684ef4f5e4f5075422ac4407b3a4dcd65 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] yuzhaojing commented on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-10-29 Thread GitBox
yuzhaojing commented on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-955128692 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][RFC-28]support z-order for hudi

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 133379deca564ca42f10a1f3e59bb4aa17d80964 UNKNOWN * e891f941b3359bc675cc4288ee608e026ee03b2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-

[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][RFC-28]support z-order for hudi

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 133379deca564ca42f10a1f3e59bb4aa17d80964 UNKNOWN * e891f941b3359bc675cc4288ee608e026ee03b2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-

[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][RFC-28]support z-order for hudi

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 133379deca564ca42f10a1f3e59bb4aa17d80964 UNKNOWN * e891f941b3359bc675cc4288ee608e026ee03b2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-

[GitHub] [hudi] xiarixiaoyao commented on pull request #3330: [HUDI-2101][RFC-28]support z-order for hudi

2021-10-29 Thread GitBox
xiarixiaoyao commented on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-955120690 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [hudi] aharbunou-branch opened a new issue #3894: [SUPPORT] Property hoodie.datasource.write.recordkey.field not found during version ONE to TWO migration

2021-10-29 Thread GitBox
aharbunou-branch opened a new issue #3894: URL: https://github.com/apache/hudi/issues/3894 **Describe the problem you faced** I'm migrating Hudi from 0.8.0 to 0.9.0. I'm testing it with simple workflow that gets data from s3 and puts to a Hudi table via Spark. This workflow is ru

[GitHub] [hudi] pranotishanbhag commented on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
pranotishanbhag commented on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955117043 From the logs: ``` 21/10/30 00:15:00 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3://ums-source-m

[GitHub] [hudi] umehrot2 edited a comment on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
umehrot2 edited a comment on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955116742 Removing from `blocked-on-user` and marking as a `release blocker`. This has been reported in slack as well https://apache-hudi.slack.com/archives/C4D716NPQ/p1635490536147600. A

[GitHub] [hudi] umehrot2 commented on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
umehrot2 commented on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955116742 Removing from `blocked-on-user` and marking as a `release blocker`. This has been reported in slack as well https://apache-hudi.slack.com/archives/C4D716NPQ/p1635490536147600. Although

[GitHub] [hudi] pranotishanbhag edited a comment on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
pranotishanbhag edited a comment on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955083608 Hi, I am facing the same issue with 0.9. My schema is as below ``` root |-- _hoodie_commit_time: string (nullable = true) |-- _hoodie_commit_seqno: str

[jira] [Resolved] (HUDI-2654) Schedules the compaction from earliest for flink

2021-10-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2654. -- Resolution: Fixed Fixed via master branch: 92a3c458bde7ca4d2bb72f4dbe486073f6a5ec4f > Schedules the com

[hudi] branch master updated: [HUDI-2654] Schedules the compaction from earliest for flink (#3891)

2021-10-29 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 92a3c45 [HUDI-2654] Schedules the compaction f

[GitHub] [hudi] danny0405 merged pull request #3891: [HUDI-2654] Schedules the compaction from earliest for flink

2021-10-29 Thread GitBox
danny0405 merged pull request #3891: URL: https://github.com/apache/hudi/pull/3891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubsc

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

2021-10-29 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436219#comment-17436219 ] Prashant Wason commented on HUDI-2151: -- >> Is file listing parallelism too high? In

[jira] [Commented] (HUDI-2443) KVComparator in HFile for metadata table is tied to HBase version and shading

2021-10-29 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436216#comment-17436216 ] Prashant Wason commented on HUDI-2443: -- The metadata table is being upgraded to a new

[jira] [Assigned] (HUDI-2637) Triage all bugs around Multi-writer and certify the tested flows

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2637: - Assignee: sivabalan narayanan > Triage all bugs around Multi-writer and certify t

[jira] [Updated] (HUDI-2637) Triage all bugs around Multi-writer and certify the tested flows

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2637: -- Fix Version/s: 0.10.0 > Triage all bugs around Multi-writer and certify the tested flows

[jira] [Assigned] (HUDI-1839) FSUtils getAllPartitions broken by NotSerializableException: org.apache.hadoop.fs.Path

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1839: - Assignee: sivabalan narayanan > FSUtils getAllPartitions broken by NotSerializabl

[jira] [Resolved] (HUDI-2489) Tuning HoodieROTablePathFilter by caching, aiming to reduce unnecessary list/get requests

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2489. --- Resolution: Fixed > Tuning HoodieROTablePathFilter by caching, aiming to reduce unnece

[jira] [Reopened] (HUDI-2489) Tuning HoodieROTablePathFilter by caching, aiming to reduce unnecessary list/get requests

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-2489: --- Assignee: sivabalan narayanan > Tuning HoodieROTablePathFilter by caching, aiming to r

[jira] [Updated] (HUDI-2489) Tuning HoodieROTablePathFilter by caching, aiming to reduce unnecessary list/get requests

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2489: -- Status: Closed (was: Patch Available) > Tuning HoodieROTablePathFilter by caching, aimi

[jira] [Updated] (HUDI-2489) Tuning HoodieROTablePathFilter by caching, aiming to reduce unnecessary list/get requests

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2489: -- Status: Patch Available (was: In Progress) > Tuning HoodieROTablePathFilter by caching,

[jira] [Commented] (HUDI-1839) FSUtils getAllPartitions broken by NotSerializableException: org.apache.hadoop.fs.Path

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436212#comment-17436212 ] sivabalan narayanan commented on HUDI-1839: --- [~nishith29] [~satishkotha] [~uditm

[jira] [Resolved] (HUDI-2552) Metadata validation causes test failures and CI is all failing

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2552. --- Fix Version/s: 0.10.0 Resolution: Fixed > Metadata validation causes test failu

[jira] [Updated] (HUDI-2552) Metadata validation causes test failures and CI is all failing

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2552: -- Status: In Progress (was: Open) > Metadata validation causes test failures and CI is al

[jira] [Commented] (HUDI-2655) Non partitioned dataset with metadata fails

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436211#comment-17436211 ] sivabalan narayanan commented on HUDI-2655: --- Related ticket https://issues.apach

[jira] [Updated] (HUDI-2567) Verify synchronous metadata patch w/ multi writers end to end

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2567: -- Fix Version/s: 0.10.0 > Verify synchronous metadata patch w/ multi writers end to end >

[jira] [Assigned] (HUDI-2567) Verify synchronous metadata patch w/ multi writers end to end

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2567: - Assignee: sivabalan narayanan > Verify synchronous metadata patch w/ multi writer

[jira] [Updated] (HUDI-2567) Verify synchronous metadata patch w/ multi writers end to end

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2567: -- Priority: Blocker (was: Major) > Verify synchronous metadata patch w/ multi writers end

[jira] [Updated] (HUDI-2553) Re-enable max delta commits for metadata table to 10

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2553: -- Fix Version/s: 0.10.0 > Re-enable max delta commits for metadata table to 10 > -

[jira] [Reopened] (HUDI-2553) Re-enable max delta commits for metadata table to 10

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-2553: --- > Re-enable max delta commits for metadata table to 10 > -

[jira] [Commented] (HUDI-2303) TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436210#comment-17436210 ] sivabalan narayanan commented on HUDI-2303: --- [~pwason]: can we close this one ou

[jira] [Resolved] (HUDI-2553) Re-enable max delta commits for metadata table to 10

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2553. --- Resolution: Fixed > Re-enable max delta commits for metadata table to 10 > ---

[jira] [Updated] (HUDI-2553) Re-enable max delta commits for metadata table to 10

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2553: -- Status: Closed (was: Patch Available) > Re-enable max delta commits for metadata table

[jira] [Commented] (HUDI-1401) Presto use of Metadata Table for file listings

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436208#comment-17436208 ] sivabalan narayanan commented on HUDI-1401: --- [~bhavanisudha] [~vinoth] [~uditme]

[jira] [Updated] (HUDI-1401) Presto use of Metadata Table for file listings

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1401: -- Fix Version/s: (was: 0.7.0) 0.10.0 > Presto use of Metadata Table

[jira] [Updated] (HUDI-2303) TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2303: -- Fix Version/s: 0.10.0 > TestMereIntoLogOnlyTable with metadata enabled surfaces likely b

[jira] [Updated] (HUDI-2303) TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2303: -- Priority: Blocker (was: Major) > TestMereIntoLogOnlyTable with metadata enabled surface

[hudi] branch master updated (5b1992a -> f632669)

2021-10-29 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5b1992a [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved (#3802) add f632669 [

[GitHub] [hudi] nsivabalan merged pull request #3884: [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files

2021-10-29 Thread GitBox
nsivabalan merged pull request #3884: URL: https://github.com/apache/hudi/pull/3884 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[GitHub] [hudi] pranotishanbhag commented on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
pranotishanbhag commented on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955083864 Stack trace: ``` User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 195.0 failed 4 times, most recent failure:

[GitHub] [hudi] pranotishanbhag commented on issue #3841: Schema evolution improvement in 0.9.0 brakes existing applications

2021-10-29 Thread GitBox
pranotishanbhag commented on issue #3841: URL: https://github.com/apache/hudi/issues/3841#issuecomment-955083608 Hi, I am facing the same issue with 0.9. My schema is as below ``` root |-- _hoodie_commit_time: string (nullable = true) |-- _hoodie_commit_seqno: string

[GitHub] [hudi] kywe665 commented on pull request #3855: [HUDI-2607] Reorganize Hudi Docs

2021-10-29 Thread GitBox
kywe665 commented on pull request #3855: URL: https://github.com/apache/hudi/pull/3855#issuecomment-955080484 Good suggestion, I moved it down -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Updated] (HUDI-2641) One inflight commit rolling back other concurrent inflight commits causing them to fail

2021-10-29 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2641: - Priority: Blocker (was: Critical) > One inflight commit rolling back other concurrent inflight co

[GitHub] [hudi] manojpec commented on a change in pull request #3884: [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files

2021-10-29 Thread GitBox
manojpec commented on a change in pull request #3884: URL: https://github.com/apache/hudi/pull/3884#discussion_r739498221 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/hash/HashID.java ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundatio

[GitHub] [hudi] nsivabalan commented on a change in pull request #3884: [HUDI-1295] Hash ID generator util for Hudi table columns, partition and files

2021-10-29 Thread GitBox
nsivabalan commented on a change in pull request #3884: URL: https://github.com/apache/hudi/pull/3884#discussion_r739487446 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/hash/HashID.java ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] [hudi] hudi-bot edited a comment on pull request #3889: [HUDI-2443] Hudi KVComparator for all HFile writer usages

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3889: URL: https://github.com/apache/hudi/pull/3889#issuecomment-954508326 ## CI report: * 62c235d87ecd3cd04d847a1ca69b32eb24a1af61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 3c3560d52e3e3632294c4e30c6d67ad99730c8ec UNKNOWN * 47eaf38e01fb23599f2f1a8e5964177a2610121d Azure: [FAILURE](https://dev.azure.com/apache-hudi-

[jira] [Reopened] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-2573: --- > Deadlock w/ multi writer due to double locking > ---

[jira] [Resolved] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2573. --- Resolution: Fixed > Deadlock w/ multi writer due to double locking > -

[GitHub] [hudi] prashantwason commented on pull request #3871: [HUDI-2593][WIP] Enabling virtual keys for the metadata table

2021-10-29 Thread GitBox
prashantwason commented on pull request #3871: URL: https://github.com/apache/hudi/pull/3871#issuecomment-954973982 @manojpec Can you please give more details of why virtual keys dont work? Is this a limitation of the metadata table schema or of the way virtual key support is implemented?

[GitHub] [hudi] hudi-bot edited a comment on pull request #3889: [HUDI-2443] Hudi KVComparator for all HFile writer usages

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3889: URL: https://github.com/apache/hudi/pull/3889#issuecomment-954508326 ## CI report: * 658ed16a52becdf55e17193ace2a380834a3b7f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] hudi-bot edited a comment on pull request #3889: [HUDI-2443] Hudi KVComparator for all HFile writer usages

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3889: URL: https://github.com/apache/hudi/pull/3889#issuecomment-954508326 ## CI report: * 658ed16a52becdf55e17193ace2a380834a3b7f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] bkosuru commented on issue #3892: Insert produces 44764 files with ~50MB each

2021-10-29 Thread GitBox
bkosuru commented on issue #3892: URL: https://github.com/apache/hudi/issues/3892#issuecomment-954943267 Any idea why INSERT created 44764 files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 3c3560d52e3e3632294c4e30c6d67ad99730c8ec UNKNOWN * b7989f4971a7df795386e351ca4089cdfabbfda2 Azure: [CANCELED](https://dev.azure.com/apache-hudi

[GitHub] [hudi] hudi-bot edited a comment on pull request #3803: [HUDI-2472] Enabling Metadata table for TestCleaner unit tests

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3803: URL: https://github.com/apache/hudi/pull/3803#issuecomment-943565382 ## CI report: * 155da1e91c4aae82abcada285d8a382e2b1c23fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] bkosuru commented on issue #3892: Insert produces 44764 files with ~50MB each

2021-10-29 Thread GitBox
bkosuru commented on issue #3892: URL: https://github.com/apache/hudi/issues/3892#issuecomment-954914700 BULK_INSERT created 9639 files. PARALLELISM value I specified is 9338. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 3c3560d52e3e3632294c4e30c6d67ad99730c8ec UNKNOWN * b7989f4971a7df795386e351ca4089cdfabbfda2 Azure: [CANCELED](https://dev.azure.com/apache-hudi

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 207ad6f754e05af4345991f643479cd8964b93c7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[hudi] branch master updated (29574af -> 5b1992a)

2021-10-29 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 29574af [HUDI-2573] Fixing double locking with multi-writers (#3827) add 5b1992a [HUDI-1500] Support replace

[GitHub] [hudi] nsivabalan merged pull request #3802: [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved

2021-10-29 Thread GitBox
nsivabalan merged pull request #3802: URL: https://github.com/apache/hudi/pull/3802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[jira] [Reopened] (HUDI-2472) Tests failure follow up when metadata is enabled by default

2021-10-29 Thread Manoj Govindassamy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy reopened HUDI-2472: -- [https://github.com/apache/hudi/pull/3803] is not yet merged. Will close this after the mer

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 207ad6f754e05af4345991f643479cd8964b93c7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 207ad6f754e05af4345991f643479cd8964b93c7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot edited a comment on pull request #3866: [HUDI-1430] SparkDataFrameWriteClient

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3866: URL: https://github.com/apache/hudi/pull/3866#issuecomment-952070379 ## CI report: * 8144fcd5285a5f53f4a76c4327e0bb8c90b46c97 UNKNOWN * 01cb7594fc6b49dcdde255269d43f4b97d5193ce UNKNOWN * 7d3e9053f159b07c3266e4eef1dc0c17bb850b5

[GitHub] [hudi] hudi-bot edited a comment on pull request #3866: [HUDI-1430] SparkDataFrameWriteClient

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3866: URL: https://github.com/apache/hudi/pull/3866#issuecomment-952070379 ## CI report: * 8144fcd5285a5f53f4a76c4327e0bb8c90b46c97 UNKNOWN * 01cb7594fc6b49dcdde255269d43f4b97d5193ce UNKNOWN * 306086e46b2053e51b6378b03de209a972d0a71

[GitHub] [hudi] hudi-bot edited a comment on pull request #3891: [HUDI-2654] Schedules the compaction from earliest for flink

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3891: URL: https://github.com/apache/hudi/pull/3891#issuecomment-954691119 ## CI report: * 6006ac61c40fbb8fff0055407500e5b96e3b37e2 UNKNOWN * 22cf1b1e0d6a96029ea2933ccff9551b52d538ab Azure: [SUCCESS](https://dev.azure.com/apache-hudi-

[GitHub] [hudi] hudi-bot edited a comment on pull request #3803: [HUDI-2472] Enabling Metadata table for TestCleaner unit tests

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3803: URL: https://github.com/apache/hudi/pull/3803#issuecomment-943565382 ## CI report: * 155da1e91c4aae82abcada285d8a382e2b1c23fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/res

[GitHub] [hudi] hudi-bot edited a comment on pull request #3799: [HUDI-2491] hoodie.datasource.hive_sync.mode=hms mode is supported in…

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3799: URL: https://github.com/apache/hudi/pull/3799#issuecomment-943176004 ## CI report: * aa02b3508fee06bf0f3fd03b65d016eaeb9e4a65 UNKNOWN * e98d19ea99ead03b9360e04b1d006a67cf68a285 Azure: [FAILURE](https://dev.azure.com/apache-hudi-

[jira] [Updated] (HUDI-2332) Implement scheduling of compaction/ clustering for Kafka Connect

2021-10-29 Thread Rajesh Mahindra (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Mahindra updated HUDI-2332: -- Status: Patch Available (was: In Progress) > Implement scheduling of compaction/ clustering for

[jira] [Updated] (HUDI-2502) Refactor index in hudi-client module

2021-10-29 Thread Rajesh Mahindra (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Mahindra updated HUDI-2502: -- Status: Closed (was: Patch Available) > Refactor index in hudi-client module >

[jira] [Updated] (HUDI-2616) Implement BloomIndex for Dataset

2021-10-29 Thread Rajesh Mahindra (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Mahindra updated HUDI-2616: -- Status: In Progress (was: Open) > Implement BloomIndex for Dataset > --

[GitHub] [hudi] codope commented on a change in pull request #3802: [HUDI-1500] Support replace commit in DeltaSync with commit metadata preserved

2021-10-29 Thread GitBox
codope commented on a change in pull request #3802: URL: https://github.com/apache/hudi/pull/3802#discussion_r739386456 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1137,8 +1144,9 @@ public void testHoodieA

[jira] [Updated] (HUDI-2472) Tests failure follow up when metadata is enabled by default

2021-10-29 Thread Rajesh Mahindra (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Mahindra updated HUDI-2472: -- Status: Closed (was: Patch Available) > Tests failure follow up when metadata is enabled by def

[jira] [Updated] (HUDI-2655) Non partitioned dataset with metadata fails

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2655: -- Fix Version/s: 0.10.0 > Non partitioned dataset with metadata fails > --

[jira] [Assigned] (HUDI-2655) Non partitioned dataset with metadata fails

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2655: - Assignee: sivabalan narayanan (was: Manoj Govindassamy) > Non partitioned datase

[jira] [Updated] (HUDI-2655) Non partitioned dataset with metadata fails

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2655: -- Description: likely when compaction kicks in within metadata table, record key is empty.

[GitHub] [hudi] manojpec commented on pull request #3803: [HUDI-2472] Enabling Metadata table for TestCleaner unit tests

2021-10-29 Thread GitBox
manojpec commented on pull request #3803: URL: https://github.com/apache/hudi/pull/3803#issuecomment-954874732 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [hudi] manojpec commented on pull request #3803: [HUDI-2472] Enabling Metadata table for TestCleaner unit tests

2021-10-29 Thread GitBox
manojpec commented on pull request #3803: URL: https://github.com/apache/hudi/pull/3803#issuecomment-954874613 Last CI failure is in TestHiveSyncGlobalCommitTool.testBasicGlobalCommit:105 which is not related to the test fixes done here. -- This is an automated message from the Apache G

[jira] [Updated] (HUDI-2589) Write RFC for metadata based bloom index

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2589: -- Status: In Progress (was: Open) > Write RFC for metadata based bloom index > --

[jira] [Updated] (HUDI-2573) Deadlock w/ multi writer due to double locking

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2573: -- Status: Closed (was: Patch Available) > Deadlock w/ multi writer due to double locking

[jira] [Updated] (HUDI-1294) Implement inlining of HFile Data Blocks in metadata table log

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1294: -- Status: Closed (was: Patch Available) > Implement inlining of HFile Data Blocks in meta

[GitHub] [hudi] nsivabalan merged pull request #3827: [HUDI-2573] Fixing double locking with multi-writers

2021-10-29 Thread GitBox
nsivabalan merged pull request #3827: URL: https://github.com/apache/hudi/pull/3827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[hudi] branch master updated: [HUDI-2573] Fixing double locking with multi-writers (#3827)

2021-10-29 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 29574af [HUDI-2573] Fixing double locking with

[hudi] branch master updated: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table (#3762)

2021-10-29 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 69ee790 [HUDI-1294] Adding inline read and see

[GitHub] [hudi] nsivabalan commented on a change in pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-29 Thread GitBox
nsivabalan commented on a change in pull request #3762: URL: https://github.com/apache/hudi/pull/3762#discussion_r739371195 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java ## @@ -120,65 +120,114 @@ private void initIfNeeded() {

[GitHub] [hudi] nsivabalan merged pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-29 Thread GitBox
nsivabalan merged pull request #3762: URL: https://github.com/apache/hudi/pull/3762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[jira] [Updated] (HUDI-2657) Make inlining configurable based on diff use-case.

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2657: -- Parent: HUDI-1822 Issue Type: Sub-task (was: Improvement) > Make inlining confi

[jira] [Created] (HUDI-2657) Make inlining configurable based on diff use-case.

2021-10-29 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-2657: - Summary: Make inlining configurable based on diff use-case. Key: HUDI-2657 URL: https://issues.apache.org/jira/browse/HUDI-2657 Project: Apache Hudi

[jira] [Assigned] (HUDI-2657) Make inlining configurable based on diff use-case.

2021-10-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2657: - Assignee: Prashant Wason > Make inlining configurable based on diff use-case. >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3893: [WIP][HUDI-2656] Generalize HoodieIndex for flexible record data type

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3893: URL: https://github.com/apache/hudi/pull/3893#issuecomment-954849759 ## CI report: * 207ad6f754e05af4345991f643479cd8964b93c7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot edited a comment on pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-29 Thread GitBox
hudi-bot edited a comment on pull request #3762: URL: https://github.com/apache/hudi/pull/3762#issuecomment-938271221 ## CI report: * 5fb7a2afa196fd75ada005d26a0fb9fce5472545 UNKNOWN * 2b369a695c04138759613ab0acc03eec24484c47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-

  1   2   3   >