[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time

2023-01-12 Thread GitBox
hudi-bot commented on PR #7664: URL: https://github.com/apache/hudi/pull/7664#issuecomment-1381433123 ## CI report: * 2f4ee14477c6868151f3d14eb1f3535d3eafb11d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1430

[GitHub] [hudi] hudi-bot commented on pull request #7664: [HUDI-5551] support seconds unit on event_time

2023-01-12 Thread GitBox
hudi-bot commented on PR #7664: URL: https://github.com/apache/hudi/pull/7664#issuecomment-1381427203 ## CI report: * 2f4ee14477c6868151f3d14eb1f3535d3eafb11d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7645: [HUDI-5529] Ensure that consistent hashing metadata is purged when dropping a partition

2023-01-12 Thread GitBox
hudi-bot commented on PR #7645: URL: https://github.com/apache/hudi/pull/7645#issuecomment-1381427069 ## CI report: * b4353d0e64b67fb6516df2b86aba695f68f71f17 UNKNOWN * 30f38f8635dccd6c202e72467874f73f4f6ee696 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] BruceKellan commented on issue #7643: [SUPPORT] Too slow while using trino-hudi connector while querying partitioned tables.

2023-01-12 Thread GitBox
BruceKellan commented on issue #7643: URL: https://github.com/apache/hudi/issues/7643#issuecomment-1381426977 The entire query took 15 seconds, and the step of fetching the partitions took 12 seconds, it seem to not be the expected behavior. -- This is an automated message from the Apache

[jira] [Created] (HUDI-5551) support seconds unit on event_time

2023-01-12 Thread scx (Jira)
scx created HUDI-5551: - Summary: support seconds unit on event_time Key: HUDI-5551 URL: https://issues.apache.org/jira/browse/HUDI-5551 Project: Apache Hudi Issue Type: New Feature Reporter:

[GitHub] [hudi] scxwhite opened a new pull request, #7664: [HUDI-5551] support seconds unit on event_time

2023-01-12 Thread GitBox
scxwhite opened a new pull request, #7664: URL: https://github.com/apache/hudi/pull/7664 ### Change Logs https://user-images.githubusercontent.com/23207189/212252797-fcca4d17-d49b-48c0-970c-a576eaeed8ea.png";> When I used grafana to draw the metrics of commitFreshnessInMs, I found t

[jira] [Updated] (HUDI-5551) support seconds unit on event_time

2023-01-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5551: - Labels: pull-request-available (was: ) > support seconds unit on event_time > --

[jira] [Resolved] (HUDI-5528) HiveSyncProcedure & HiveSyncTool also needs to add HIVE_SYNC_TABLE_STRATEGY

2023-01-12 Thread HunterXHunter (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HunterXHunter resolved HUDI-5528. - > HiveSyncProcedure & HiveSyncTool also needs to add HIVE_SYNC_TABLE_STRATEGY > --

[GitHub] [hudi] LinMingQiang commented on pull request #7460: [HUDI-5391] Modify the default value of parameter `hoodie.write.lock.…

2023-01-12 Thread GitBox
LinMingQiang commented on PR #7460: URL: https://github.com/apache/hudi/pull/7460#issuecomment-1381364632 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] hudi-bot commented on pull request #7661: [DO NOT MERGE] Release testing record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7661: URL: https://github.com/apache/hudi/pull/7661#issuecomment-1381351399 ## CI report: * f698f26db2314cbbbee30d37df0d6fd343317796 UNKNOWN * 7c73d0f50fc521a0171d26c09e1f47fc658527b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7662: [DO NOT MERGE] Release 0130 test record merger with unified names

2023-01-12 Thread GitBox
hudi-bot commented on PR #7662: URL: https://github.com/apache/hudi/pull/7662#issuecomment-1381351445 ## CI report: * 8450c1d373a4e3c440390e622b6d720feb13fecc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1429

[jira] [Closed] (HUDI-5538) Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support

2023-01-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-5538. Resolution: Fixed Fixed via master branch: 6aa8e9681b2f24975d1c6a216d3b6de53f6b5702 > Fix ContinuousFileSou

[hudi] branch master updated (b6db9b1ff87 -> 6aa8e9681b2)

2023-01-12 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from b6db9b1ff87 [MINOR] Fix flaky testStructuredStreamingWithCompaction (#7485) add 6aa8e9681b2 [HUDI-5538] Fix Con

[GitHub] [hudi] danny0405 merged pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support

2023-01-12 Thread GitBox
danny0405 merged PR #7637: URL: https://github.com/apache/hudi/pull/7637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[jira] [Updated] (HUDI-5550) TestHoodieDeltaStreamerWithMultiWriter#testUpsertsContinuousModeWithMultipleWritersForConflicts is flaky

2023-01-12 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5550: - Description: A raw log: https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_api

[jira] [Created] (HUDI-5550) TestHoodieDeltaStreamerWithMultiWriter#testUpsertsContinuousModeWithMultipleWritersForConflicts is flaky

2023-01-12 Thread Danny Chen (Jira)
Danny Chen created HUDI-5550: Summary: TestHoodieDeltaStreamerWithMultiWriter#testUpsertsContinuousModeWithMultipleWritersForConflicts is flaky Key: HUDI-5550 URL: https://issues.apache.org/jira/browse/HUDI-5550

[GitHub] [hudi] pushpavanthar commented on issue #7657: [SUPPORT] Invalid number of file groups for partition:column_stats

2023-01-12 Thread GitBox
pushpavanthar commented on issue #7657: URL: https://github.com/apache/hudi/issues/7657#issuecomment-1381331765 Thanks for the work around @BalaMahesh. Will try this out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] BalaMahesh commented on issue #7657: [SUPPORT] Invalid number of file groups for partition:column_stats

2023-01-12 Thread GitBox
BalaMahesh commented on issue #7657: URL: https://github.com/apache/hudi/issues/7657#issuecomment-1381325931 I have came across the same problem using 0.12.0 version. I have set hoodie.metadata.index.bloom.filter.enable=false hoodie.metadata.index.column.stats.enable=false t

[GitHub] [hudi] BalaMahesh commented on issue #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

2023-01-12 Thread GitBox
BalaMahesh commented on issue #7595: URL: https://github.com/apache/hudi/issues/7595#issuecomment-1381317993 I have changed the index type to simple and then restarted the application. Index look up duration has come down and uniform now. https://user-images.githubusercontent.com/2

[GitHub] [hudi] hudi-bot commented on pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7655: URL: https://github.com/apache/hudi/pull/7655#issuecomment-1381303467 ## CI report: * 66e1a0bc35a484c20d7ba871359ed3424db23af9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1427

[GitHub] [hudi] hudi-bot commented on pull request #7660: [MINOR] unify naming for record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7660: URL: https://github.com/apache/hudi/pull/7660#issuecomment-1381303509 ## CI report: * cbbcde078cfd2653710905439861fd4188e06943 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1429

[GitHub] [hudi] KevinyhZou commented on pull request #7189: [HUDI-5159]Add support write success file to finished partition in flink streaming write

2023-01-12 Thread GitBox
KevinyhZou commented on PR #7189: URL: https://github.com/apache/hudi/pull/7189#issuecomment-1381297335 cc @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

2023-01-12 Thread GitBox
nsivabalan commented on code in PR #7612: URL: https://github.com/apache/hudi/pull/7612#discussion_r1068908725 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -358,23 +364,27 @@ public static String createNewFileId(String idPfx, int id) { * Get th

[GitHub] [hudi] alexeykudinkin commented on pull request #6782: [HUDI-4911][HUDI-3301] Fixing `HoodieMetadataLogRecordReader` to avoid flushing cache for every lookup

2023-01-12 Thread GitBox
alexeykudinkin commented on PR #6782: URL: https://github.com/apache/hudi/pull/6782#issuecomment-1381295059 CI is green: https://user-images.githubusercontent.com/428277/212235746-acc1da03-e8ce-443e-9d66-738446a8a568.png";> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_b

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7619: [MINOR] Optimizing schema validation in Metadata table

2023-01-12 Thread GitBox
nsivabalan commented on code in PR #7619: URL: https://github.com/apache/hudi/pull/7619#discussion_r1068907789 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -814,18 +820,22 @@ private void validateSchema() throws HoodieUpsertExcep

[GitHub] [hudi] yihua merged pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
yihua merged PR #7485: URL: https://github.com/apache/hudi/pull/7485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[hudi] branch master updated (669e5676725 -> b6db9b1ff87)

2023-01-12 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 669e5676725 [HUDI-5543] Description of clustering.plan.partition.filter.mode supports DAY_ROLLING strategy (#7656)

[GitHub] [hudi] hudi-bot commented on pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7655: URL: https://github.com/apache/hudi/pull/7655#issuecomment-1381265626 ## CI report: * 66e1a0bc35a484c20d7ba871359ed3424db23af9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1427

[GitHub] [hudi] hudi-bot commented on pull request #7645: [HUDI-5529] Ensure that consistent hashing metadata is purged when dropping a partition

2023-01-12 Thread GitBox
hudi-bot commented on PR #7645: URL: https://github.com/apache/hudi/pull/7645#issuecomment-1381265545 ## CI report: * b4353d0e64b67fb6516df2b86aba695f68f71f17 UNKNOWN * 63b413649a394d956da5f05e1c65356d12d2b356 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1381265247 ## CI report: * 8a2e23787b5117afec3e8fabf098e1908615c366 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1429

[GitHub] [hudi] hudi-bot commented on pull request #7362: [HUDI-5315] The record size is dynamically estimated when the table i…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7362: URL: https://github.com/apache/hudi/pull/7362#issuecomment-1381265077 ## CI report: * 4cbb5a5ee787dd44ee3b29d422cfbb39f6560351 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354

[GitHub] [hudi] voonhous commented on issue #7663: [SUPPORT] ALTER TABLE DROP PARTITION DDL may cause data inconsistencies when table service actions are performed

2023-01-12 Thread GitBox
voonhous commented on issue #7663: URL: https://github.com/apache/hudi/issues/7663#issuecomment-1381263841 Not sure if this is the correct approach , but should we prevent users from dropping a partition if there's a pending table service action on partition path? ```java // ensur

[GitHub] [hudi] hudi-bot commented on pull request #7362: [HUDI-5315] The record size is dynamically estimated when the table i…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7362: URL: https://github.com/apache/hudi/pull/7362#issuecomment-1381260448 ## CI report: * 4cbb5a5ee787dd44ee3b29d422cfbb39f6560351 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354

[GitHub] [hudi] hudi-bot commented on pull request #7645: [HUDI-5529] Ensure that consistent hashing metadata is purged when dropping a partition

2023-01-12 Thread GitBox
hudi-bot commented on PR #7645: URL: https://github.com/apache/hudi/pull/7645#issuecomment-1381260896 ## CI report: * b4353d0e64b67fb6516df2b86aba695f68f71f17 UNKNOWN * 63b413649a394d956da5f05e1c65356d12d2b356 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] stream2000 commented on a diff in pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
stream2000 commented on code in PR #7655: URL: https://github.com/apache/hudi/pull/7655#discussion_r1068878268 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/DeleteMarkerProcedure.scala: ## @@ -60,6 +62,9 @@ class DeleteMarkerProc

[GitHub] [hudi] voonhous opened a new issue, #7663: [SUPPORT] ALTER TABLE DROP PARTITION DDL may cause data inconsistencies when table service actions are performed

2023-01-12 Thread GitBox
voonhous opened a new issue, #7663: URL: https://github.com/apache/hudi/issues/7663 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr.

[GitHub] [hudi] leesf commented on a diff in pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
leesf commented on code in PR #7655: URL: https://github.com/apache/hudi/pull/7655#discussion_r1068873943 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/DeleteMarkerProcedure.scala: ## @@ -60,6 +62,9 @@ class DeleteMarkerProcedure

[jira] [Updated] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5549: -- Description: HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT needs to be set to "parquet" wheneve

[jira] [Updated] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5549: -- Priority: Critical (was: Major) > LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parque

[jira] [Updated] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5549: -- Fix Version/s: 0.13.0 > LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when usin

[GitHub] [hudi] leesf merged pull request #7656: [HUDI-5543] Description of clustering.plan.partition.filter.mode supports DAY_ROLLING strategy

2023-01-12 Thread GitBox
leesf merged PR #7656: URL: https://github.com/apache/hudi/pull/7656 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[jira] [Assigned] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-5549: - Assignee: Frank Wong (was: Alexey Kudinkin) > LOGFILE_DATA_BLOCK_FORMAT Should not need

[hudi] branch master updated (70d450e8389 -> 669e5676725)

2023-01-12 Thread leesf
This is an automated email from the ASF dual-hosted git repository. leesf pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 70d450e8389 [HUDI-5545] Extending support to other special characters (#7585) add 669e5676725 [HUDI-5543] Descripti

[GitHub] [hudi] leesf commented on pull request #7656: [HUDI-5543] Description of clustering.plan.partition.filter.mode supports DAY_ROLLING strategy

2023-01-12 Thread GitBox
leesf commented on PR #7656: URL: https://github.com/apache/hudi/pull/7656#issuecomment-1381251689 Merging this PR as it only modified docs and the flink related CI passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[jira] [Assigned] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-5549: - Assignee: Alexey Kudinkin > LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parque

[jira] [Updated] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5549: -- Description: HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT needs to be set to "parquet" whenever

[jira] [Updated] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordManager

2023-01-12 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5549: -- Summary: LOGFILE_DATA_BLOCK_FORMAT Should not need to be set to parquet when using SparkRecordMa

[jira] [Created] (HUDI-5549) LOGFILE_DATA_BLOCK_FORMAT Should not need to be setting parquet when using SparkRecordManager

2023-01-12 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-5549: - Summary: LOGFILE_DATA_BLOCK_FORMAT Should not need to be setting parquet when using SparkRecordManager Key: HUDI-5549 URL: https://issues.apache.org/jira/browse/HUDI-5549

[jira] [Updated] (HUDI-5520) Fail MDT when list of log files grows unboundedly

2023-01-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5520: -- Summary: Fail MDT when list of log files grows unboundedly (was: Fail MDT when list of

[GitHub] [hudi] yihua commented on pull request #7607: [HUDI-5499] Fixing Spark SQL configs not being properly propagated for CTAS and other commands

2023-01-12 Thread GitBox
yihua commented on PR #7607: URL: https://github.com/apache/hudi/pull/7607#issuecomment-1381219405 @alexeykudinkin could you check the CI failure? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[hudi] branch master updated: [HUDI-5545] Extending support to other special characters (#7585)

2023-01-12 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 70d450e8389 [HUDI-5545] Extending support to other

[GitHub] [hudi] yihua merged pull request #7585: [HUDI-5545] Extending support to other special characters

2023-01-12 Thread GitBox
yihua merged PR #7585: URL: https://github.com/apache/hudi/pull/7585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7655: URL: https://github.com/apache/hudi/pull/7655#issuecomment-1381209350 ## CI report: * 66e1a0bc35a484c20d7ba871359ed3424db23af9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1427

[GitHub] [hudi] stream2000 commented on pull request #7655: [HUDI-5540] Close write client after usage of DeleteMarker/RollbackTo…

2023-01-12 Thread GitBox
stream2000 commented on PR #7655: URL: https://github.com/apache/hudi/pull/7655#issuecomment-1381209268 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[jira] [Updated] (HUDI-5541) Disable precombine in bootstrap

2023-01-12 Thread Luning Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luning Wang updated HUDI-5541: -- Description: When I run a bootstrap to convert a hive table to Hudi in the 0.12.2 version, it throws th

[GitHub] [hudi] hudi-bot commented on pull request #7585: [HUDI-5545] Extending support to other special characters

2023-01-12 Thread GitBox
hudi-bot commented on PR #7585: URL: https://github.com/apache/hudi/pull/7585#issuecomment-1381203996 ## CI report: * 85370798131dd4272a55fdea95195c0dd880a2ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1415

[GitHub] [hudi] hudi-bot commented on pull request #7661: [DO NOT MERGE] Release testing record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7661: URL: https://github.com/apache/hudi/pull/7661#issuecomment-1381198297 ## CI report: * f698f26db2314cbbbee30d37df0d6fd343317796 UNKNOWN * 7c73d0f50fc521a0171d26c09e1f47fc658527b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[jira] [Created] (HUDI-5548) Spark Sql update hudi's table properties

2023-01-12 Thread Forward Xu (Jira)
Forward Xu created HUDI-5548: Summary: Spark Sql update hudi's table properties Key: HUDI-5548 URL: https://issues.apache.org/jira/browse/HUDI-5548 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] hudi-bot commented on pull request #7662: [DO NOT MERGE] Release 0130 test record merger with unified names

2023-01-12 Thread GitBox
hudi-bot commented on PR #7662: URL: https://github.com/apache/hudi/pull/7662#issuecomment-1381149724 ## CI report: * 8450c1d373a4e3c440390e622b6d720feb13fecc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1429

[GitHub] [hudi] hudi-bot commented on pull request #7661: [DO NOT MERGE] Release testing record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7661: URL: https://github.com/apache/hudi/pull/7661#issuecomment-1381149691 ## CI report: * f698f26db2314cbbbee30d37df0d6fd343317796 UNKNOWN * 7c73d0f50fc521a0171d26c09e1f47fc658527b2 UNKNOWN Bot commands @hudi-bot supports the following

[GitHub] [hudi] hudi-bot commented on pull request #7660: [MINOR] unify naming for record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7660: URL: https://github.com/apache/hudi/pull/7660#issuecomment-1381149659 ## CI report: * cbbcde078cfd2653710905439861fd4188e06943 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1429

[GitHub] [hudi] hudi-bot commented on pull request #7662: [DO NOT MERGE] Release 0130 test record merger with unified names

2023-01-12 Thread GitBox
hudi-bot commented on PR #7662: URL: https://github.com/apache/hudi/pull/7662#issuecomment-1381142086 ## CI report: * 8450c1d373a4e3c440390e622b6d720feb13fecc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7661: [DO NOT MERGE] Release testing record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7661: URL: https://github.com/apache/hudi/pull/7661#issuecomment-1381142012 ## CI report: * f698f26db2314cbbbee30d37df0d6fd343317796 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7660: [MINOR] unify naming for record merger

2023-01-12 Thread GitBox
hudi-bot commented on PR #7660: URL: https://github.com/apache/hudi/pull/7660#issuecomment-1381141946 ## CI report: * cbbcde078cfd2653710905439861fd4188e06943 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7658: [HUDI-5544] Improve log msgs during bulk insert

2023-01-12 Thread GitBox
hudi-bot commented on PR #7658: URL: https://github.com/apache/hudi/pull/7658#issuecomment-1381141870 ## CI report: * bad3777fde1d006cad0d2eeddbf7ded179ab9e0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[GitHub] [hudi] hudi-bot commented on pull request #7490: [HUDI-5407][HUDI-5408] Fixing rollback in MDT to be eager

2023-01-12 Thread GitBox
hudi-bot commented on PR #7490: URL: https://github.com/apache/hudi/pull/7490#issuecomment-1381133389 ## CI report: * edddc18a49977834315dcee0b13465a2f0b622f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[GitHub] [hudi] jonvex opened a new pull request, #7662: [DO NOT MERGE] Release 0130 test record merger with unified names

2023-01-12 Thread GitBox
jonvex opened a new pull request, #7662: URL: https://github.com/apache/hudi/pull/7662 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[GitHub] [hudi] jonvex opened a new pull request, #7661: [DO NOT MERGE] Release testing record merger

2023-01-12 Thread GitBox
jonvex opened a new pull request, #7661: URL: https://github.com/apache/hudi/pull/7661 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[GitHub] [hudi] jonvex opened a new pull request, #7660: [MINOR] unify naming for record merger

2023-01-12 Thread GitBox
jonvex opened a new pull request, #7660: URL: https://github.com/apache/hudi/pull/7660 ### Change Logs Inconsistent usage of "record merger impls"/"record merger strategy" and "merger impls"/"merger strategy" ### Impact Consistent naming is easier to read and use

[jira] [Updated] (HUDI-5547) Add support to refresh FileSystem based schema provider for every batch w/ deltastreamer in continuous mode

2023-01-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5547: -- Fix Version/s: 0.13.0 > Add support to refresh FileSystem based schema provider for ever

[jira] [Created] (HUDI-5547) Add support to refresh FileSystem based schema provider for every batch w/ deltastreamer in continuous mode

2023-01-12 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-5547: - Summary: Add support to refresh FileSystem based schema provider for every batch w/ deltastreamer in continuous mode Key: HUDI-5547 URL: https://issues.apache.org/jira/b

[GitHub] [hudi] hudi-bot commented on pull request #7544: [HUDI-5433] Fix the way we deduce the pending instants for MDT writes

2023-01-12 Thread GitBox
hudi-bot commented on PR #7544: URL: https://github.com/apache/hudi/pull/7544#issuecomment-1381072918 ## CI report: * be019e3773773c2639b79376a6d0262e80a45650 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support

2023-01-12 Thread GitBox
hudi-bot commented on PR #7637: URL: https://github.com/apache/hudi/pull/7637#issuecomment-1381019092 ## CI report: * 5a3df6098e7b2867080d2cacde328380c977b144 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[jira] [Closed] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin closed HUDI-5023. - Resolution: Fixed > Add new Executor avoiding Queueing in the write-path > ---

[jira] [Updated] (HUDI-5546) Use bulk insert for the first write to an empty Hudi table

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-5546: -- Priority: Critical (was: Major) > Use bulk insert for the first write to an empty Hudi table >

[jira] [Assigned] (HUDI-5023) Add new Executor avoiding Queueing in the write-path

2023-01-12 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-5023: - Assignee: Yue Zhang (was: Alexey Kudinkin) > Add new Executor avoiding Queueing in the w

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1380935862 ## CI report: * 6f68c27960c024bed805fdb1a409ee46872d8d8d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=142

[GitHub] [hudi] hudi-bot commented on pull request #7656: [HUDI-5543] Description of clustering.plan.partition.filter.mode supports DAY_ROLLING strategy

2023-01-12 Thread GitBox
hudi-bot commented on PR #7656: URL: https://github.com/apache/hudi/pull/7656#issuecomment-1380929380 ## CI report: * 9753a3a804529ce76af1e1a4a762487dfa8307cf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1380928977 ## CI report: * d2d97c5492bfe4c6210028f0ce52d2872e0d1397 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1380921690 ## CI report: * d2d97c5492bfe4c6210028f0ce52d2872e0d1397 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385

[GitHub] [hudi] hudi-bot commented on pull request #7460: [HUDI-5391] Modify the default value of parameter `hoodie.write.lock.…

2023-01-12 Thread GitBox
hudi-bot commented on PR #7460: URL: https://github.com/apache/hudi/pull/7460#issuecomment-1380921585 ## CI report: * 2f475cc3c3a8c19e1c56c5f230a5aee002435eac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1428

[GitHub] [hudi] yihua commented on a diff in pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

2023-01-12 Thread GitBox
yihua commented on code in PR #7612: URL: https://github.com/apache/hudi/pull/7612#discussion_r1068539565 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -358,23 +364,27 @@ public static String createNewFileId(String idPfx, int id) { * Get the fil

[jira] [Updated] (HUDI-5546) Use bulk insert for the first write to an empty Hudi table

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5546: Description: For the first batch to an empty table, given no tagging/indexing is needed, we can always use b

[jira] [Updated] (HUDI-5546) Use bulk insert for the first write to an empty Hudi table

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5546: Description: For the first batch to an empty table, given no tagging/indexing is needed, we can always use b

[jira] [Created] (HUDI-5546) Use bulk insert for the first write to an empty Hudi table

2023-01-12 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5546: --- Summary: Use bulk insert for the first write to an empty Hudi table Key: HUDI-5546 URL: https://issues.apache.org/jira/browse/HUDI-5546 Project: Apache Hudi Issue Type

[jira] [Updated] (HUDI-5546) Use bulk insert for the first write to an empty Hudi table

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5546: Fix Version/s: 0.13.0 > Use bulk insert for the first write to an empty Hudi table > ---

[GitHub] [hudi] kazdy commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow

2023-01-12 Thread GitBox
kazdy commented on PR #7640: URL: https://github.com/apache/hudi/pull/7640#issuecomment-1380861070 Hi @the-other-tim-brown I'm interested in this functionality and have some questions, if I understand correctly the UUID will be the same for the same set of values in columns that it's ba

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1380852604 ## CI report: * d2d97c5492bfe4c6210028f0ce52d2872e0d1397 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385

[GitHub] [hudi] hudi-bot commented on pull request #7645: [HUDI-5529] Ensure that consistent hashing metadata is purged when dropping a partition

2023-01-12 Thread GitBox
hudi-bot commented on PR #7645: URL: https://github.com/apache/hudi/pull/7645#issuecomment-1380840284 ## CI report: * b4353d0e64b67fb6516df2b86aba695f68f71f17 UNKNOWN * 63b413649a394d956da5f05e1c65356d12d2b356 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #7585: [HUDI-5545] Extending support to other special characters

2023-01-12 Thread GitBox
hudi-bot commented on PR #7585: URL: https://github.com/apache/hudi/pull/7585#issuecomment-1380839837 ## CI report: * 85370798131dd4272a55fdea95195c0dd880a2ef Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1415

[GitHub] [hudi] hudi-bot commented on pull request #7485: [MINOR] Fix flaky testStructuredStreamingWithCompaction

2023-01-12 Thread GitBox
hudi-bot commented on PR #7485: URL: https://github.com/apache/hudi/pull/7485#issuecomment-1380839293 ## CI report: * d2d97c5492bfe4c6210028f0ce52d2872e0d1397 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-5545) Extending support to other special characters for S3EventsMetaSelector

2023-01-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5545: - Labels: pull-request-available (was: ) > Extending support to other special characters for S3Even

[GitHub] [hudi] hudi-bot commented on pull request #7585: [HUDI-5545] Extending support to other special characters

2023-01-12 Thread GitBox
hudi-bot commented on PR #7585: URL: https://github.com/apache/hudi/pull/7585#issuecomment-1380824478 ## CI report: * 85370798131dd4272a55fdea95195c0dd880a2ef UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Created] (HUDI-5545) Extending support to other special characters for S3EventsMetaSelector

2023-01-12 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5545: --- Summary: Extending support to other special characters for S3EventsMetaSelector Key: HUDI-5545 URL: https://issues.apache.org/jira/browse/HUDI-5545 Project: Apache Hudi

[jira] [Updated] (HUDI-5545) Extending support to other special characters for S3EventsMetaSelector

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5545: Priority: Critical (was: Major) > Extending support to other special characters for S3EventsMetaSelector >

[jira] [Updated] (HUDI-5545) Extending support to other special characters for S3EventsMetaSelector

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5545: Description: This fix is to cover issue as follows. I am working on ingestion with S3 as source by followin

[jira] [Updated] (HUDI-5545) Extending support to other special characters for S3EventsMetaSelector

2023-01-12 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5545: Fix Version/s: 0.13.0 > Extending support to other special characters for S3EventsMetaSelector > ---

[GitHub] [hudi] yihua commented on pull request #7585: [MINOR] Extending support to other special characters

2023-01-12 Thread GitBox
yihua commented on PR #7585: URL: https://github.com/apache/hudi/pull/7585#issuecomment-1380814769 @srikanthjaggari Thanks for the fix! Since the Slack message is not going to be visible after 90 days, I updated the description with your message for clarity. -- This is an automated mess

[GitHub] [hudi] hudi-bot commented on pull request #7445: [HUDI-5380] Fixing change table path but table location in metastore …

2023-01-12 Thread GitBox
hudi-bot commented on PR #7445: URL: https://github.com/apache/hudi/pull/7445#issuecomment-1380808068 ## CI report: * 21c809bf3d1303d3a731454002a1fb1b0ce920a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1427

[GitHub] [hudi] yihua commented on a diff in pull request #7619: [MINOR] Optimizing schema validation in Metadata table

2023-01-12 Thread GitBox
yihua commented on code in PR #7619: URL: https://github.com/apache/hudi/pull/7619#discussion_r1068463661 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/SparkRDDMetadataWriteClient.java: ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

  1   2   >