[GitHub] [hudi] nsivabalan edited a comment on issue #3676: MOR table rolls out new parquet files at 10MB for new inserts - even though max file size set as 128MB

2021-10-12 Thread GitBox
nsivabalan edited a comment on issue #3676: URL: https://github.com/apache/hudi/issues/3676#issuecomment-941946294 @FelixKJose : If you are interested in working on a fix, I have filed a tracking jira https://issues.apache.org/jira/browse/HUDI-2550. I can help guide you if you are

[jira] [Updated] (HUDI-2550) Add support to configure no of small files to consider with MOR

2021-10-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2550: -- Labels: sev:critical user-support-issues (was: ) > Add support to configure no of

[GitHub] [hudi] nsivabalan commented on issue #3676: MOR table rolls out new parquet files at 10MB for new inserts - even though max file size set as 128MB

2021-10-12 Thread GitBox
nsivabalan commented on issue #3676: URL: https://github.com/apache/hudi/issues/3676#issuecomment-941946294 @FelixKJose : If you are interested in working on a fix, I have filed a tracking jira https://issues.apache.org/jira/browse/HUDI-2550 -- This is an automated message from the

[jira] [Assigned] (HUDI-2550) Add support to configure no of small files to consider with MOR

2021-10-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2550: - Assignee: sivabalan narayanan > Add support to configure no of small files to

[jira] [Created] (HUDI-2550) Add support to configure no of small files to consider with MOR

2021-10-12 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-2550: - Summary: Add support to configure no of small files to consider with MOR Key: HUDI-2550 URL: https://issues.apache.org/jira/browse/HUDI-2550 Project:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d1f847d4bc54f9ee4d2d4ae075dba3279bbf53f0 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d1f847d4bc54f9ee4d2d4ae075dba3279bbf53f0 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d1f847d4bc54f9ee4d2d4ae075dba3279bbf53f0 Azure:

[GitHub] [hudi] danny0405 commented on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
danny0405 commented on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-941915874 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3746: URL: https://github.com/apache/hudi/pull/3746#issuecomment-941904370 @yihua : Can you review this patch please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] xiaogaozi commented on pull request #3780: [DOCS] Update JuiceFS doc

2021-10-12 Thread GitBox
xiaogaozi commented on pull request #3780: URL: https://github.com/apache/hudi/pull/3780#issuecomment-941892619 @leesf Hi, may I ask is there anything else that needs to be modified? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] hudi-bot edited a comment on pull request #3754: [HUDI-2482] support 'drop partition' sql

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3754: URL: https://github.com/apache/hudi/pull/3754#issuecomment-935480787 ## CI report: * 42ea9882efc540dfe36610a5b343b672b1eeaee8 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * b5649459e5e8a5ebf5a140418c16951294a29689 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * b5649459e5e8a5ebf5a140418c16951294a29689 Azure:

[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
xiarixiaoyao commented on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-941865271 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] hudi-bot edited a comment on pull request #3754: [HUDI-2482] support 'drop partition' sql

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3754: URL: https://github.com/apache/hudi/pull/3754#issuecomment-935480787 ## CI report: * 42ea9882efc540dfe36610a5b343b672b1eeaee8 Azure:

[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426

2021-10-12 Thread GitBox
prashantwason commented on a change in pull request #3590: URL: https://github.com/apache/hudi/pull/3590#discussion_r727634907 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java ## @@ -206,6 +209,19 @@

[GitHub] [hudi] YannByron commented on pull request #3754: [HUDI-2482] support 'drop partition' sql

2021-10-12 Thread GitBox
YannByron commented on pull request #3754: URL: https://github.com/apache/hudi/pull/3754#issuecomment-941850211 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] SteNicholas commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
SteNicholas commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727643723 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -257,6 +260,11 @@ protected void

[GitHub] [hudi] yihua commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
yihua commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727640936 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -257,6 +260,11 @@ protected void

[GitHub] [hudi] nsivabalan commented on pull request #3275: HUDI-1827 Add ORC Support in Bootstrap Op

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3275: URL: https://github.com/apache/hudi/pull/3275#issuecomment-941838417 @manasaks : Are you sure patch's commit history is in good shape? I see 28 commits in total and 272 files have been changed. Can you please check that out and clean up the PR.

[GitHub] [hudi] nsivabalan commented on pull request #3312: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-941837466 @liujinhui1994 : when you get a chance, can you respond to Vinoth's clarifications. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on pull request #3420: [HUDI-2283] Support Clustering Command For Spark Sql

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3420: URL: https://github.com/apache/hudi/pull/3420#issuecomment-941836764 @yihua : Can you take a stab at reviewing this patch. If you wanna jam or need some headsup, I can fill you in. -- This is an automated message from the Apache Git Service.

[GitHub] [hudi] nsivabalan commented on pull request #3540: [HUDI-2364] Run compaction without user schema file provided

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3540: URL: https://github.com/apache/hudi/pull/3540#issuecomment-941836349 @umehrot2 : Can you please review this patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] nsivabalan commented on a change in pull request #3765: [HUDI-2533] New option for hoodieClusteringJob to check, rollback and re-execute the last failed clustering job

2021-10-12 Thread GitBox
nsivabalan commented on a change in pull request #3765: URL: https://github.com/apache/hudi/pull/3765#discussion_r727633410 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1096,12 +1100,21 @@ public void

[GitHub] [hudi] hudi-bot edited a comment on pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3762: URL: https://github.com/apache/hudi/pull/3762#issuecomment-938271221 ## CI report: * 5fb7a2afa196fd75ada005d26a0fb9fce5472545 UNKNOWN * 1140119d054ac3bad6e982f35ad72fa788fa9a70 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3762: URL: https://github.com/apache/hudi/pull/3762#issuecomment-938271221 ## CI report: * 5fb7a2afa196fd75ada005d26a0fb9fce5472545 UNKNOWN * cb7e9cea8fa966437a892be1e0917443c034e21e Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3762: [HUDI-1294] Adding inline read and seek based read(batch get) for hfile log blocks in metadata table

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3762: URL: https://github.com/apache/hudi/pull/3762#issuecomment-938271221 ## CI report: * 5fb7a2afa196fd75ada005d26a0fb9fce5472545 UNKNOWN * cb7e9cea8fa966437a892be1e0917443c034e21e Azure:

[GitHub] [hudi] nsivabalan commented on pull request #3783: [WIP] Adding parquet data block with inline read support

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3783: URL: https://github.com/apache/hudi/pull/3783#issuecomment-941747075 @rmahindra123 : can you please review this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] satishkotha commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors

2021-10-12 Thread GitBox
satishkotha commented on pull request #3666: URL: https://github.com/apache/hudi/pull/3666#issuecomment-941670666 @zhangyue19921010 Sorry for delay. its merged now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[hudi] branch master updated (8a487ea -> e6711b1)

2021-10-12 Thread satish
This is an automated email from the ASF dual-hosted git repository. satish pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 8a487ea [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths (#3768) add e6711b1 [HUDI-2435][BUG]Fix

[GitHub] [hudi] satishkotha merged pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors

2021-10-12 Thread GitBox
satishkotha merged pull request #3666: URL: https://github.com/apache/hudi/pull/3666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3719: URL: https://github.com/apache/hudi/pull/3719#issuecomment-941259434 @zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when

[jira] [Updated] (HUDI-2549) Exceptions when using second writer into Hudi table managed by DeltaStreamer

2021-10-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2549: -- Labels: multi-writer sev:critical (was: multi-writer) > Exceptions when using second

[hudi] branch master updated: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths (#3768)

2021-10-12 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8a487ea [HUDI-2494] Fixing glob pattern to

[GitHub] [hudi] nsivabalan merged pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-12 Thread GitBox
nsivabalan merged pull request #3768: URL: https://github.com/apache/hudi/pull/3768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3768: URL: https://github.com/apache/hudi/pull/3768#issuecomment-938643768 ## CI report: * 87885b79753568ed685fa0d29d44cdcb7fe06d69 Azure:

[jira] [Closed] (HUDI-2275) HoodieDeltaStreamerException when using OCC and a second concurrent writer

2021-10-12 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Hagman closed HUDI-2275. - Fix Version/s: (was: 0.10.0) 0.9.0 Resolution: Fixed >

[jira] [Commented] (HUDI-2275) HoodieDeltaStreamerException when using OCC and a second concurrent writer

2021-10-12 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427809#comment-17427809 ] Dave Hagman commented on HUDI-2275: --- We are experiencing new issues when migrating to version 0.9 so I

[jira] [Commented] (HUDI-2549) Exceptions when using second writer into Hudi table managed by DeltaStreamer

2021-10-12 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427807#comment-17427807 ] Dave Hagman commented on HUDI-2549: --- In order to try and validate my hypothesis about race conditions I

[jira] [Created] (HUDI-2549) Exceptions when using second writer into Hudi table managed by DeltaStreamer

2021-10-12 Thread Dave Hagman (Jira)
Dave Hagman created HUDI-2549: - Summary: Exceptions when using second writer into Hudi table managed by DeltaStreamer Key: HUDI-2549 URL: https://issues.apache.org/jira/browse/HUDI-2549 Project: Apache

[jira] [Updated] (HUDI-2549) Exceptions when using second writer into Hudi table managed by DeltaStreamer

2021-10-12 Thread Dave Hagman (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Hagman updated HUDI-2549: -- Description: When running the DeltaStreamer along with a second spark datasource writer (with

[GitHub] [hudi] hudi-bot edited a comment on pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3768: URL: https://github.com/apache/hudi/pull/3768#issuecomment-938643768 ## CI report: * d08cac085906b121ab6e72e822add6d38a3c6034 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3768: URL: https://github.com/apache/hudi/pull/3768#issuecomment-938643768 ## CI report: * d08cac085906b121ab6e72e822add6d38a3c6034 Azure:

[GitHub] [hudi] nsivabalan commented on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-941163867 @codope: Can you review this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] nsivabalan commented on issue #3737: how can we migrate a legacy COW table into MOR table

2021-10-12 Thread GitBox
nsivabalan commented on issue #3737: URL: https://github.com/apache/hudi/issues/3737#issuecomment-941148645 I guess it should work w/o issues in general. with latest spark-sql dml support, not sure if we added validation to check if table type can't be changed. Technically speaking, COW

[hudi] branch asf-site updated: [Minor][asf-site] Modify hoodieDeltaStreamer user docs, adding more merged source-class (#3785)

2021-10-12 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 52c721c [Minor][asf-site] Modify

[GitHub] [hudi] nsivabalan merged pull request #3785: [Minor][asf-site] Modify hoodieDeltaStreamer user docs, adding more merged source-class

2021-10-12 Thread GitBox
nsivabalan merged pull request #3785: URL: https://github.com/apache/hudi/pull/3785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d1f847d4bc54f9ee4d2d4ae075dba3279bbf53f0 Azure:

[hudi] branch master updated: [HUDI-2532] Metadata table compaction trigger max delta commits (#3784)

2021-10-12 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 252c4ed [HUDI-2532] Metadata table compaction

[GitHub] [hudi] nsivabalan merged pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-12 Thread GitBox
nsivabalan merged pull request #3784: URL: https://github.com/apache/hudi/pull/3784 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on pull request #3784: [HUDI-2532] Metadata table compaction trigger max delta commits default config

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3784: URL: https://github.com/apache/hudi/pull/3784#issuecomment-941031091 CC @n3nash @prashantwason Gonna land this. just wanted to keep you folks informed. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on pull request #3757: [HUDI-2005][WIP] Avoiding direct fs calls in HoodieLogFileReader and AbstractTableFileSystemView

2021-10-12 Thread GitBox
nsivabalan commented on pull request #3757: URL: https://github.com/apache/hudi/pull/3757#issuecomment-941025806 @vinothchandar : wrt fix in AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(), lets sync up sometime. I [tried moving](https://github.com/apache/hudi/pull/3769) the

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * 2e13ee5b2758d0c37ea731219ff453196afa5868 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * 2e13ee5b2758d0c37ea731219ff453196afa5868 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * 2e13ee5b2758d0c37ea731219ff453196afa5868 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * 57ad6441990bef9ffec760928b561a4f3bf9f1f5 Azure:

[GitHub] [hudi] SteNicholas commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
SteNicholas commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727107543 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -412,8 +421,21 @@ private void

[GitHub] [hudi] SteNicholas commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
SteNicholas commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727103868 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -257,6 +260,11 @@ protected void

[GitHub] [hudi] leesf commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
leesf commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727089132 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -412,8 +421,21 @@ private void

[GitHub] [hudi] leesf commented on a change in pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
leesf commented on a change in pull request #3779: URL: https://github.com/apache/hudi/pull/3779#discussion_r727088305 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -257,6 +260,11 @@ protected void

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d80bc60e3943da3aa0c9309e8bb63d38edf0cd04 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d80bc60e3943da3aa0c9309e8bb63d38edf0cd04 Azure:

[jira] [Updated] (HUDI-2548) Flink streaming reader misses the rolling over file handles on object storage

2021-10-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-2548: - Summary: Flink streaming reader misses the rolling over file handles on object storage (was: Flink

[jira] [Updated] (HUDI-2548) Flink streaming reader misses the rolling over file handles

2021-10-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-2548: - Summary: Flink streaming reader misses the rolling over file handles (was: Flink streaming reader misses

[GitHub] [hudi] hudi-bot edited a comment on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * a8cfca40fc0419928c3061f65ec6cc7b162a193e Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * a8cfca40fc0419928c3061f65ec6cc7b162a193e Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d80bc60e3943da3aa0c9309e8bb63d38edf0cd04 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d80bc60e3943da3aa0c9309e8bb63d38edf0cd04 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
hudi-bot commented on pull request #3787: URL: https://github.com/apache/hudi/pull/3787#issuecomment-940888966 ## CI report: * d80bc60e3943da3aa0c9309e8bb63d38edf0cd04 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Updated] (HUDI-2548) Flink streaming reader misses the rolling over file handles

2021-10-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2548: - Labels: pull-request-available (was: ) > Flink streaming reader misses the rolling over file

[GitHub] [hudi] danny0405 opened a new pull request #3787: [HUDI-2548] Flink streaming reader misses the rolling over file handles

2021-10-12 Thread GitBox
danny0405 opened a new pull request #3787: URL: https://github.com/apache/hudi/pull/3787 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[jira] [Created] (HUDI-2548) Flink streaming reader misses the rolling over file handles

2021-10-12 Thread Danny Chen (Jira)
Danny Chen created HUDI-2548: Summary: Flink streaming reader misses the rolling over file handles Key: HUDI-2548 URL: https://issues.apache.org/jira/browse/HUDI-2548 Project: Apache Hudi Issue

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * b5649459e5e8a5ebf5a140418c16951294a29689 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * bc4493a2b866b0b08d31d0f79f91cbeceb9c0ddf Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 7c97e48c93c985ba989e83b7716aef87a73a5773 Azure:

[GitHub] [hudi] govorunov commented on issue #3756: [SUPPORT] Can we use Hudi to build Temporal Datastore?

2021-10-12 Thread GitBox
govorunov commented on issue #3756: URL: https://github.com/apache/hudi/issues/3756#issuecomment-940832604 Sorry, I'm quite new to big data so may ask some stupid questions. Let's forget about temporal storage, database backups etc. for a minute. Can we use Hudi to store all database

[GitHub] [hudi] hudi-bot edited a comment on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * a8cfca40fc0419928c3061f65ec6cc7b162a193e Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 Azure:

[jira] [Updated] (HUDI-2547) Schedule Flink compaction in service

2021-10-12 Thread yuzhaojing (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuzhaojing updated HUDI-2547: - Summary: Schedule Flink compaction in service (was: Schedule Flink Compaction in service) > Schedule

[jira] [Created] (HUDI-2547) Schedule Flink Compaction in service

2021-10-12 Thread yuzhaojing (Jira)
yuzhaojing created HUDI-2547: Summary: Schedule Flink Compaction in service Key: HUDI-2547 URL: https://issues.apache.org/jira/browse/HUDI-2547 Project: Apache Hudi Issue Type: New Feature

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * a8cfca40fc0419928c3061f65ec6cc7b162a193e Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * bc4493a2b866b0b08d31d0f79f91cbeceb9c0ddf Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 7c97e48c93c985ba989e83b7716aef87a73a5773 Azure:

[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
xiarixiaoyao commented on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-940777833 @danny0405 @nsivabalan @leesf update the code. addressed all comments。 1. recover HUDI-1969, and remove RealTimeMergedRecordReader.java which is no needed. 2. add

[jira] [Commented] (HUDI-864) parquet schema conflict: optional binary (UTF8) is not a group

2021-10-12 Thread Sebastian Bernauer (Jira)
[ https://issues.apache.org/jira/browse/HUDI-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427536#comment-17427536 ] Sebastian Bernauer commented on HUDI-864: - Hi [~rolandjohann] can you please try the following

[GitHub] [hudi] yihua commented on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
yihua commented on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-940757505 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] yihua commented on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-12 Thread GitBox
yihua commented on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-940756599 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] hudi-bot commented on pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
hudi-bot commented on pull request #3786: URL: https://github.com/apache/hudi/pull/3786#issuecomment-940753752 ## CI report: * a8cfca40fc0419928c3061f65ec6cc7b162a193e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[jira] [Updated] (HUDI-2541) Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2541: - Labels: pull-request-available (was: ) > Eliminate the unnecessary usage of write client for

[GitHub] [hudi] SteNicholas opened a new pull request #3786: [HUDI-2541] Eliminate the unnecessary usage of write client for flink

2021-10-12 Thread GitBox
SteNicholas opened a new pull request #3786: URL: https://github.com/apache/hudi/pull/3786 ## What is the purpose of the pull request *`HoodieFlinkWriteClient` defaults to start a embedded timeline server. If there is any dataset to write, the embedded server is definitely

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
xiarixiaoyao commented on a change in pull request #3203: URL: https://github.com/apache/hudi/pull/3203#discussion_r726838452 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java ## @@ -99,7 +101,32 @@ } }

[GitHub] [hudi] hudi-bot edited a comment on pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3741: URL: https://github.com/apache/hudi/pull/3741#issuecomment-931660346 ## CI report: * 7c97e48c93c985ba989e83b7716aef87a73a5773 Azure:

[GitHub] [hudi] SteNicholas commented on pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
SteNicholas commented on pull request #3779: URL: https://github.com/apache/hudi/pull/3779#issuecomment-940736007 > @SteNicholas Check the azure CI? @yanghua , I have checked the result of the azure CI and the Flink client tests are passed. -- This is an automated message from

[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3203: URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745 ## CI report: * 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN * 6eebb1a711e3655061d471c2a96b54e205dac630 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3778: [WIP][HUDI-2502] Refactor index in hudi-client module

2021-10-12 Thread GitBox
hudi-bot edited a comment on pull request #3778: URL: https://github.com/apache/hudi/pull/3778#issuecomment-939762523 ## CI report: * bc4493a2b866b0b08d31d0f79f91cbeceb9c0ddf Azure:

[GitHub] [hudi] yanghua commented on pull request #3779: [HUDI-2503] HoodieFlinkWriteClient supports to allow parallel writing to tables using Locking service

2021-10-12 Thread GitBox
yanghua commented on pull request #3779: URL: https://github.com/apache/hudi/pull/3779#issuecomment-940722695 @SteNicholas Check the azure CI? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] yihua commented on a change in pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
yihua commented on a change in pull request #3741: URL: https://github.com/apache/hudi/pull/3741#discussion_r726803858 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java ## @@ -366,12 +367,13 @@ public HoodieActiveTimeline

[GitHub] [hudi] yihua commented on a change in pull request #3741: [HUDI-2501] Add HoodieData abstraction and refactor compaction actions in hudi-client module

2021-10-12 Thread GitBox
yihua commented on a change in pull request #3741: URL: https://github.com/apache/hudi/pull/3741#discussion_r726802924 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieCopyOnWriteTableOperation.java ## @@ -0,0 +1,43 @@ +/* + * Licensed to

  1   2   >