[GitHub] [hudi] zhangyue19921010 opened a new pull request #3785: [Minor][asf-site] Modify hoodieDeltaStreamer user docs, adding more merged source-class

2021-10-11 Thread GitBox
zhangyue19921010 opened a new pull request #3785: URL: https://github.com/apache/hudi/pull/3785 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is

[GitHub] [hudi] hudi-bot edited a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938488397 ## CI report: * 742efa960a3472bb3bb6d505dabe5bf800b03fec Azure:

[GitHub] [hudi] Rap70r closed issue #3697: [SUPPORT] Performance Tuning: How to speed up stages?

2021-10-11 Thread GitBox
Rap70r closed issue #3697: URL: https://github.com/apache/hudi/issues/3697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-11 Thread GitBox
xiarixiaoyao edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-940624808 @codope yes, it will be ok after compact. but consider that case: read incremental datas which before replacecommit, origin read logical cannot resolve this

[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-11 Thread GitBox
xiarixiaoyao commented on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-940624808 @codope yes, it will be ok after compact. but consider that case: read incremental datas which before replacecommit, origin read logical cannot resolve this problem.

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-11 Thread GitBox
xiarixiaoyao commented on a change in pull request #3203: URL: https://github.com/apache/hudi/pull/3203#discussion_r726732125 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-11 Thread GitBox
xiarixiaoyao commented on a change in pull request #3203: URL: https://github.com/apache/hudi/pull/3203#discussion_r726731332 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache

[GitHub] [hudi] hudi-bot edited a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938488397 ## CI report: * aa79155eef2d731dcdf5377afa5b02127996310b Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938488397 ## CI report: * aa79155eef2d731dcdf5377afa5b02127996310b Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938488397 ## CI report: * a215aa25074d52c2002a32b68c02b4429902ab3d Azure:

[GitHub] [hudi] xiaogaozi commented on a change in pull request #3780: [DOCS] Update JuiceFS doc

2021-10-11 Thread GitBox
xiaogaozi commented on a change in pull request #3780: URL: https://github.com/apache/hudi/pull/3780#discussion_r726713961 ## File path: website/docs/jfs_hoodie.md ## @@ -1,59 +1,65 @@ --- -title: JuiceFS -keywords: [ hudi, hive, jfs, spark, flink] -summary: On this page, we

[GitHub] [hudi] garyli1019 commented on a change in pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-11 Thread GitBox
garyli1019 commented on a change in pull request #3768: URL: https://github.com/apache/hudi/pull/3768#discussion_r726711740 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala ## @@ -64,8 +64,24 @@ object HoodieSparkUtils extends

[GitHub] [hudi] YannByron commented on pull request #3754: [HUDI-2482] support 'drop partition' sql

2021-10-11 Thread GitBox
YannByron commented on pull request #3754: URL: https://github.com/apache/hudi/pull/3754#issuecomment-940595957 @nsivabalan this pr is supporting just the (b), has the same semantic with hive drop-partitions. The "delete_partitions" operation in hudi currently still has some spaces

[GitHub] [hudi] hudi-bot edited a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938488397 ## CI report: * a215aa25074d52c2002a32b68c02b4429902ab3d Azure:

[GitHub] [hudi] zhangyue19921010 removed a comment on pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-11 Thread GitBox
zhangyue19921010 removed a comment on pull request #3765: URL: https://github.com/apache/hudi/pull/3765#issuecomment-938627449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution

2021-10-11 Thread GitBox
xiarixiaoyao commented on pull request #3668: URL: https://github.com/apache/hudi/pull/3668#issuecomment-940584211 @codope thanks for your try this pr.notice that we should not use --conf "hoodie.schema.evolution.enable=true", this conf is not start with spark/hadoop/hive spark

[jira] [Updated] (HUDI-2540) Wrong validation for MetadataTableEnabled

2021-10-11 Thread Roc Marshal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roc Marshal updated HUDI-2540: -- Status: In Progress (was: Open) > Wrong validation for MetadataTableEnabled >

[jira] [Resolved] (HUDI-2540) Wrong validation for MetadataTableEnabled

2021-10-11 Thread Roc Marshal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roc Marshal resolved HUDI-2540. --- Resolution: Fixed > Wrong validation for MetadataTableEnabled >

[jira] [Commented] (HUDI-2540) Wrong validation for MetadataTableEnabled

2021-10-11 Thread Roc Marshal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427405#comment-17427405 ] Roc Marshal commented on HUDI-2540: --- Fixed in f14d4e65e7edab58aa86c495e4bd4caa79086d6b of master >

[jira] [Commented] (HUDI-2472) Tests failure follow up when metadata is enabled by default

2021-10-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427399#comment-17427399 ] sivabalan narayanan commented on HUDI-2472: ---

[GitHub] [hudi] xushiyan commented on a change in pull request #3744: [HUDI-2108] Fix flakiness in TestHoodieBackedMetadata

2021-10-11 Thread GitBox
xushiyan commented on a change in pull request #3744: URL: https://github.com/apache/hudi/pull/3744#discussion_r726681254 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java ## @@ -315,12 +314,13 @@ public

[GitHub] [hudi] hudi-bot edited a comment on pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3784: URL: https://github.com/apache/hudi/pull/3784#issuecomment-940516222 ## CI report: * 609307b5b594eddd4c789d91ffd23dbf20128b58 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3784: URL: https://github.com/apache/hudi/pull/3784#issuecomment-940516222 ## CI report: * 609307b5b594eddd4c789d91ffd23dbf20128b58 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-11 Thread GitBox
hudi-bot commented on pull request #3784: URL: https://github.com/apache/hudi/pull/3784#issuecomment-940516222 ## CI report: * 609307b5b594eddd4c789d91ffd23dbf20128b58 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Updated] (HUDI-2532) Set right default value for max delta commits for compaction in metadata table

2021-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2532: - Labels: pull-request-available (was: ) > Set right default value for max delta commits for

[GitHub] [hudi] manojpec opened a new pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-11 Thread GitBox
manojpec opened a new pull request #3784: URL: https://github.com/apache/hudi/pull/3784 ## What is the purpose of the pull request Setting the max delta commits default config to 10 (previously it was 24) to trigger the compaction in metadata table quicker than before. ##

[jira] [Updated] (HUDI-2532) Set right default value for max delta commits for compaction in metadata table

2021-10-11 Thread Manoj Govindassamy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HUDI-2532: - Status: In Progress (was: Open) > Set right default value for max delta commits for

[GitHub] [hudi] nsivabalan commented on a change in pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-11 Thread GitBox
nsivabalan commented on a change in pull request #3768: URL: https://github.com/apache/hudi/pull/3768#discussion_r726622650 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala ## @@ -64,8 +64,24 @@ object HoodieSparkUtils extends

[GitHub] [hudi] hudi-bot edited a comment on pull request #3783: [WIP] Adding parquet data block with inline read support

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3783: URL: https://github.com/apache/hudi/pull/3783#issuecomment-940438535 ## CI report: * 761ed1bd177bec5385ea5e652d7e8fcb802ac59f Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3783: [WIP] Adding parquet data block with inline read support

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3783: URL: https://github.com/apache/hudi/pull/3783#issuecomment-940438535 ## CI report: * 761ed1bd177bec5385ea5e652d7e8fcb802ac59f Azure:

[GitHub] [hudi] ChiehFu commented on issue #3782: [SUPPORT] Hudi Concurrent write (OCC) with upsert tables random errors

2021-10-11 Thread GitBox
ChiehFu commented on issue #3782: URL: https://github.com/apache/hudi/issues/3782#issuecomment-940439919 In addition, looking at how a Hudi instantTime is created, it seems that there is no additional mechanism to prevent two concurrent jobs from having a same instantTime. Cloud

[GitHub] [hudi] hudi-bot commented on pull request #3783: [WIP] Adding parquet data block with inline read support

2021-10-11 Thread GitBox
hudi-bot commented on pull request #3783: URL: https://github.com/apache/hudi/pull/3783#issuecomment-940438535 ## CI report: * 761ed1bd177bec5385ea5e652d7e8fcb802ac59f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Assigned] (HUDI-2472) Tests failure follow up when metadata is enabled by default

2021-10-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2472: - Assignee: Manoj Govindassamy (was: sivabalan narayanan) > Tests failure follow

[jira] [Assigned] (HUDI-2468) Fix rollback of first commit after being synced to metadata table

2021-10-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2468: - Assignee: Manoj Govindassamy (was: sivabalan narayanan) > Fix rollback of first

[GitHub] [hudi] nsivabalan opened a new pull request #3783: [WIP] Adding parquet data block with inline read support

2021-10-11 Thread GitBox
nsivabalan opened a new pull request #3783: URL: https://github.com/apache/hudi/pull/3783 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[hudi] branch master updated (f14d4e6 -> 48a3906)

2021-10-11 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from f14d4e6 [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable (#3781) add 48a3906

[GitHub] [hudi] nsivabalan merged pull request #3764: [MINOR] Fix typo,'paritition' corrected to 'partition'

2021-10-11 Thread GitBox
nsivabalan merged pull request #3764: URL: https://github.com/apache/hudi/pull/3764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on pull request #3752: [HUDI-1294][WIP] Adding inline read for Hfile log blocks

2021-10-11 Thread GitBox
nsivabalan commented on pull request #3752: URL: https://github.com/apache/hudi/pull/3752#issuecomment-940269721 Closing in favor of https://github.com/apache/hudi/pull/3762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan closed pull request #3752: [HUDI-1294][WIP] Adding inline read for Hfile log blocks

2021-10-11 Thread GitBox
nsivabalan closed pull request #3752: URL: https://github.com/apache/hudi/pull/3752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] pratyakshsharma commented on pull request #3776: [HUDI-2543]: Added guides section

2021-10-11 Thread GitBox
pratyakshsharma commented on pull request #3776: URL: https://github.com/apache/hudi/pull/3776#issuecomment-940247146 @vinothchandar please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[hudi] branch master updated: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable (#3781)

2021-10-11 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f14d4e6 [HUDI-2540] Fixed wrong validation

[GitHub] [hudi] nsivabalan merged pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-11 Thread GitBox
nsivabalan merged pull request #3781: URL: https://github.com/apache/hudi/pull/3781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] refset commented on issue #3756: [SUPPORT] Can we use Hudi to build Temporal Datastore?

2021-10-11 Thread GitBox
refset commented on issue #3756: URL: https://github.com/apache/hudi/issues/3756#issuecomment-940232664 Hi @govorunov, I was just taking a look at Hudi myself, so I'm certainly no expert, but I think you are looking for "bitemporal" as-of queries where `commit time` (aka `transaction

[GitHub] [hudi] nsivabalan commented on a change in pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-11 Thread GitBox
nsivabalan commented on a change in pull request #3741: URL: https://github.com/apache/hudi/pull/3741#discussion_r726211865 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/RunCompactionActionExecutor.java ## @@ -19,65 +19,65 @@

[GitHub] [hudi] ChiehFu opened a new issue #3782: [SUPPORT] Hudi Concurrent write (OCC) with upsert tables random errors

2021-10-11 Thread GitBox
ChiehFu opened a new issue #3782: URL: https://github.com/apache/hudi/issues/3782 Hello, We are running jobs on AWS EMR to compact tables stored in S3 and maintaining Athena tables through Hudi Hive sync. Recently we started exploring Hudi multi writer and we were

[jira] [Commented] (HUDI-2275) HoodieDeltaStreamerException when using OCC and a second concurrent writer

2021-10-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427252#comment-17427252 ] sivabalan narayanan commented on HUDI-2275: --- coordinated w/ [~dave_hagman] late last week. Here

[GitHub] [hudi] fengjian428 edited a comment on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running

2021-10-11 Thread GitBox
fengjian428 edited a comment on issue #3755: URL: https://github.com/apache/hudi/issues/3755#issuecomment-940179806 > It seems that the file left in reconcile stage is different with commit meta. Could you kindly share relevant logs and file status about marker file? before I got

[GitHub] [hudi] fengjian428 commented on issue #3755: [Delta Streamer] file name mismatch with meta when compaction running

2021-10-11 Thread GitBox
fengjian428 commented on issue #3755: URL: https://github.com/apache/hudi/issues/3755#issuecomment-940179806 > It seems that the file left in reconcile stage is different with commit meta. Could you kindly share relevant logs and file status about marker file? before I got error

[jira] [Assigned] (HUDI-2525) Test prometheus metrics with hudi (both spark ds and deltastreamer)

2021-10-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2525: - Assignee: Sagar Sumit (was: sivabalan narayanan) > Test prometheus metrics with

[jira] [Updated] (HUDI-2546) Validate Full Schema Evolution in Hudi including Rename

2021-10-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2546: -- Description: Test [PR#3668|https://github.com/apache/hudi/pull/3668] which implements RFC-33.

[jira] [Created] (HUDI-2546) Validate Full Schema Evolution in Hudi including Rename

2021-10-11 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-2546: - Summary: Validate Full Schema Evolution in Hudi including Rename Key: HUDI-2546 URL: https://issues.apache.org/jira/browse/HUDI-2546 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-2546) Validate Full Schema Evolution in Hudi including Rename

2021-10-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2546: -- Fix Version/s: (was: 0.9.0) 0.10.0 > Validate Full Schema Evolution in Hudi

[jira] [Updated] (HUDI-2023) Validate Schema evolution in hudi

2021-10-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2023: -- Fix Version/s: (was: 0.10.0) 0.9.0 > Validate Schema evolution in hudi >

[jira] [Closed] (HUDI-2023) Validate Schema evolution in hudi

2021-10-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-2023. - Resolution: Done > Validate Schema evolution in hudi > - > >

[GitHub] [hudi] stym06 edited a comment on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit

2021-10-11 Thread GitBox
stym06 edited a comment on issue #3747: URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218 1. Apache Hive 2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2) 3. I will check this and revert.

[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit

2021-10-11 Thread GitBox
stym06 commented on issue #3747: URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218 1. Apache Hive 2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2) 3. I will check this and revert.

[jira] [Updated] (HUDI-2531) [UMBRELLA] Support Dataset APIs in writer paths

2021-10-11 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2531: - Labels: hudi-umbrellas (was: hudi-umbrellas sev:critical user-support-issues) > [UMBRELLA]

[jira] [Updated] (HUDI-1887) Make schema post processor's default as disabled

2021-10-11 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1887: - Labels: pull-request-available release-blocker sev:high triaged (was: pull-request-available

[GitHub] [hudi] leesf commented on a change in pull request #3780: [DOCS] Update JuiceFS doc

2021-10-11 Thread GitBox
leesf commented on a change in pull request #3780: URL: https://github.com/apache/hudi/pull/3780#discussion_r726120713 ## File path: website/docs/jfs_hoodie.md ## @@ -1,59 +1,65 @@ --- -title: JuiceFS -keywords: [ hudi, hive, jfs, spark, flink] -summary: On this page, we go

[GitHub] [hudi] hudi-bot edited a comment on pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3781: URL: https://github.com/apache/hudi/pull/3781#issuecomment-939962084 ## CI report: * 732b46a47324b368c36688997ad7286a10cc7d99 Azure:

[GitHub] [hudi] RocMarshal closed pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-11 Thread GitBox
RocMarshal closed pull request #3781: URL: https://github.com/apache/hudi/pull/3781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] codope commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution

2021-10-11 Thread GitBox
codope commented on pull request #3668: URL: https://github.com/apache/hudi/pull/3668#issuecomment-939993169 @xiarixiaoyao It would really help if you could share a gist showing the schema evolution steps. For example, earlier I tried

[jira] [Assigned] (HUDI-2545) Flink compaction source supports the Source interface based on FLIP-27

2021-10-11 Thread Nicholas Jiang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Jiang reassigned HUDI-2545: Assignee: Nicholas Jiang > Flink compaction source supports the Source interface based on

[jira] [Created] (HUDI-2545) Flink compaction source supports the Source interface based on FLIP-27

2021-10-11 Thread Nicholas Jiang (Jira)
Nicholas Jiang created HUDI-2545: Summary: Flink compaction source supports the Source interface based on FLIP-27 Key: HUDI-2545 URL: https://issues.apache.org/jira/browse/HUDI-2545 Project: Apache

[GitHub] [hudi] hudi-bot edited a comment on pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3781: URL: https://github.com/apache/hudi/pull/3781#issuecomment-939962084 ## CI report: * 732b46a47324b368c36688997ad7286a10cc7d99 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in HoodieTable

2021-10-11 Thread GitBox
hudi-bot commented on pull request #3781: URL: https://github.com/apache/hudi/pull/3781#issuecomment-939962084 ## CI report: * 732b46a47324b368c36688997ad7286a10cc7d99 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Assigned] (HUDI-2503) HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread Nicholas Jiang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Jiang reassigned HUDI-2503: Assignee: Nicholas Jiang > HoodieFlinkWriteClient supports to allow parallel writing to

[jira] [Updated] (HUDI-2540) Wrong validation for MetadataTableEnabled

2021-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2540: - Labels: pull-request-available (was: ) > Wrong validation for MetadataTableEnabled >

[GitHub] [hudi] RocMarshal opened a new pull request #3781: [HUDI-2540] Fixed wrong validation for metadataTableEnabled in Hoodie…

2021-10-11 Thread GitBox
RocMarshal opened a new pull request #3781: URL: https://github.com/apache/hudi/pull/3781 …Table ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ##

[jira] [Assigned] (HUDI-2540) Wrong validation for MetadataTableEnabled

2021-10-11 Thread Roc Marshal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roc Marshal reassigned HUDI-2540: - Assignee: Roc Marshal > Wrong validation for MetadataTableEnabled >

[GitHub] [hudi] codope commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-11 Thread GitBox
codope commented on a change in pull request #3203: URL: https://github.com/apache/hudi/pull/3203#discussion_r726012801 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] codope commented on a change in pull request #3722: HUDI-2491 hoodie.datasource.hive_sync.mode=hms mode is supported in s…

2021-10-11 Thread GitBox
codope commented on a change in pull request #3722: URL: https://github.com/apache/hudi/pull/3722#discussion_r725986918 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala ## @@ -381,6 +381,11 @@ object

[GitHub] [hudi] codope commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit

2021-10-11 Thread GitBox
codope commented on issue #3747: URL: https://github.com/apache/hudi/issues/3747#issuecomment-939890785 @stym06 I am not able to reproduce this issue locally or with S3 storage. Can you please clarify a couple of things: 1. Is this with Apache Hive and Apache Spark? 2. Can you share

[GitHub] [hudi] hudi-bot edited a comment on pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3779: URL: https://github.com/apache/hudi/pull/3779#issuecomment-939811204 ## CI report: * 3629ead17fe47a09bb849a76b201200e07f1072c Azure:

[jira] [Commented] (HUDI-1139) Add support for JuiceFS

2021-10-11 Thread Changjian Gao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427014#comment-17427014 ] Changjian Gao commented on HUDI-1139: - The [#3729|https://github.com/apache/hudi/pull/3729] already

[GitHub] [hudi] xiaogaozi commented on pull request #3729: Support JuiceFileSystem

2021-10-11 Thread GitBox
xiaogaozi commented on pull request #3729: URL: https://github.com/apache/hudi/pull/3729#issuecomment-939831312 @leesf https://issues.apache.org/jira/browse/HUDI-1139 Maybe link this pull request with this JIRA issue? -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] xiaogaozi opened a new pull request #3780: [DOCS] Update JuiceFS doc

2021-10-11 Thread GitBox
xiaogaozi opened a new pull request #3780: URL: https://github.com/apache/hudi/pull/3780 ## What is the purpose of the pull request Update JuiceFS document ## Brief change log - Update JuiceFS document ## Verify this pull request This pull request just

[jira] [Created] (HUDI-2544) Use standard builder pattern to refactor ConfigProperty

2021-10-11 Thread Yann Byron (Jira)
Yann Byron created HUDI-2544: Summary: Use standard builder pattern to refactor ConfigProperty Key: HUDI-2544 URL: https://issues.apache.org/jira/browse/HUDI-2544 Project: Apache Hudi Issue

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #3646: [HUDI-349]: Added new cleaning policy based on number of hours

2021-10-11 Thread GitBox
pratyakshsharma commented on a change in pull request #3646: URL: https://github.com/apache/hudi/pull/3646#discussion_r725913854 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java ## @@ -402,9 +478,16 @@ private

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #3646: [HUDI-349]: Added new cleaning policy based on number of hours

2021-10-11 Thread GitBox
pratyakshsharma commented on a change in pull request #3646: URL: https://github.com/apache/hudi/pull/3646#discussion_r725913037 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java ## @@ -512,6 +517,11 @@ public

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #3646: [HUDI-349]: Added new cleaning policy based on number of hours

2021-10-11 Thread GitBox
pratyakshsharma commented on a change in pull request #3646: URL: https://github.com/apache/hudi/pull/3646#discussion_r725912662 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java ## @@ -69,6 +69,11 @@

[GitHub] [hudi] hudi-bot edited a comment on pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3779: URL: https://github.com/apache/hudi/pull/3779#issuecomment-939811204 ## CI report: * 3629ead17fe47a09bb849a76b201200e07f1072c Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread GitBox
hudi-bot commented on pull request #3779: URL: https://github.com/apache/hudi/pull/3779#issuecomment-939811204 ## CI report: * 3629ead17fe47a09bb849a76b201200e07f1072c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] hudi-bot edited a comment on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * ce28ca85cd2f8edecf808560538a5c5f5e4de962 Azure:

[GitHub] [hudi] pratyakshsharma commented on pull request #3646: [HUDI-349]: Added new cleaning policy based on number of hours

2021-10-11 Thread GitBox
pratyakshsharma commented on pull request #3646: URL: https://github.com/apache/hudi/pull/3646#issuecomment-939803093 @nsivabalan ack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Resolved] (HUDI-2542) AppendWriteFunction throws NPE when checkpointing without written data

2021-10-11 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2542. -- Assignee: Danny Chen Resolution: Fixed Fixed via master branch:

[hudi] branch master updated: [HUDI-2542] AppendWriteFunction throws NPE when checkpointing without written data (#3777)

2021-10-11 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 5b8bc66 [HUDI-2542] AppendWriteFunction

[GitHub] [hudi] danny0405 merged pull request #3777: [HUDI-2542] AppendWriteFunction throws NPE when checkpointing without…

2021-10-11 Thread GitBox
danny0405 merged pull request #3777: URL: https://github.com/apache/hudi/pull/3777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-2543) Introduce guides section on website

2021-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2543: - Labels: pull-request-available (was: ) > Introduce guides section on website >

[GitHub] [hudi] pratyakshsharma commented on pull request #3776: [HUDI-2543]: Added guides section

2021-10-11 Thread GitBox
pratyakshsharma commented on pull request #3776: URL: https://github.com/apache/hudi/pull/3776#issuecomment-939794284 Thank you @vingov , that helped fix the issue! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Created] (HUDI-2543) Introduce guides section on website

2021-10-11 Thread Pratyaksh Sharma (Jira)
Pratyaksh Sharma created HUDI-2543: -- Summary: Introduce guides section on website Key: HUDI-2543 URL: https://issues.apache.org/jira/browse/HUDI-2543 Project: Apache Hudi Issue Type: Task

[jira] [Updated] (HUDI-2543) Introduce guides section on website

2021-10-11 Thread Pratyaksh Sharma (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratyaksh Sharma updated HUDI-2543: --- Status: In Progress (was: Open) > Introduce guides section on website >

[jira] [Updated] (HUDI-2503) HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2503: - Labels: pull-request-available (was: ) > HoodieFlinkWriteClient supports to allow parallel

[GitHub] [hudi] SteNicholas opened a new pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-11 Thread GitBox
SteNicholas opened a new pull request #3779: URL: https://github.com/apache/hudi/pull/3779 ## What is the purpose of the pull request *The strategy interface for conflict resolution with multiple writers is introduced and the `SparkRDDWriteClient` has integrated with the

[GitHub] [hudi] hudi-bot edited a comment on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * ce28ca85cd2f8edecf808560538a5c5f5e4de962 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-11 Thread GitBox
hudi-bot commented on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * ce28ca85cd2f8edecf808560538a5c5f5e4de962 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] hudi-bot edited a comment on pull request #3777: [HUDI-2542] AppendWriteFunction throws NPE when checkpointing without…

2021-10-11 Thread GitBox
hudi-bot edited a comment on pull request #3777: URL: https://github.com/apache/hudi/pull/3777#issuecomment-939716923 ## CI report: * a2f88f17bdc775cfafdd5a2cb453fabc52f01148 Azure:

[jira] [Updated] (HUDI-2502) Refactor index in hudi-client module

2021-10-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2502: - Labels: pull-request-available (was: ) > Refactor index in hudi-client module >

[GitHub] [hudi] novakov-alexey commented on issue #3759: [SUPPORT] HoodieKeyException: recordKey value: "null"

2021-10-11 Thread GitBox
novakov-alexey commented on issue #3759: URL: https://github.com/apache/hudi/issues/3759#issuecomment-939752701 thanks @Carl-Zhou-CN , I also tried to debug Hudi code and noticed the same. That exception is thrown when key column is missing in dataframe. So far I have no idea why it can

[GitHub] [hudi] yihua opened a new pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-11 Thread GitBox
yihua opened a new pull request #3778: URL: https://github.com/apache/hudi/pull/3778 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose

  1   2   >