[GitHub] [hudi] nsivabalan commented on pull request #5760: [HUDI-4196]support index alignment

2022-11-01 Thread GitBox
nsivabalan commented on PR #5760: URL: https://github.com/apache/hudi/pull/5760#issuecomment-1299668428 @danny0405 @yuzhaojing : can you assign someone to review this patch. been there for 5 months ish. -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [hudi] complone commented on pull request #7103: [HUDI-5126] Delete duplicate configuration items PAYLOAD_CLASS_NAME

2022-11-01 Thread GitBox
complone commented on PR #7103: URL: https://github.com/apache/hudi/pull/7103#issuecomment-1299667995 > 为了让它通过验证公关检查,你必须写一些受到影响的东西,并在风险级别部分写一个风险级别。我会四处打听,看看是否有人可以查看代码。 > > 此外,如果您有兴趣参与 Hudi 社区,请查看此页面https://hudi.apache.org/community/get-involved以获取指向我们的 slack 和电子邮件列表的一些链接 ok i

[GitHub] [hudi] nsivabalan commented on pull request #6957: fix(sec): upgrade com.google.protobuf:protobuf-java to 3.18.2

2022-11-01 Thread GitBox
nsivabalan commented on PR #6957: URL: https://github.com/apache/hudi/pull/6957#issuecomment-1299666067 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[hudi] branch master updated: [MINOR] update commons-codec:commons-codec 1.4 to 1.13 (#6959)

2022-11-01 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 6fea6fc36b [MINOR] update commons-codec:commons

[GitHub] [hudi] nsivabalan merged pull request #6959: [MINOR] fix(sec): upgrade commons-codec:commons-codec to 1.13

2022-11-01 Thread GitBox
nsivabalan merged PR #6959: URL: https://github.com/apache/hudi/pull/6959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

[GitHub] [hudi] complone commented on a diff in pull request #7103: [HUDI-5126] Delete duplicate configuration items PAYLOAD_CLASS_NAME

2022-11-01 Thread GitBox
complone commented on code in PR #7103: URL: https://github.com/apache/hudi/pull/7103#discussion_r1011258331 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java: ## @@ -106,7 +106,7 @@ private FlinkOptions() { + "determine

[jira] [Updated] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5148: Story Points: 8 > Write RFC for index on functions and logical partitioning > --

[GitHub] [hudi] hudi-bot commented on pull request #7063: [HUDI-5094]Remove partition fields from table schema,if the DROP_PARTITION_COLUMNS is enabled.

2022-11-01 Thread GitBox
hudi-bot commented on PR #7063: URL: https://github.com/apache/hudi/pull/7063#issuecomment-1299658629 ## CI report: * b474805bf9deeb27944189575b7c25fa03d0bf5f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[jira] [Commented] (HUDI-512) Support Logical Partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627488#comment-17627488 ] Ethan Guo commented on HUDI-512: [~flashJd] Thanks for raising this.  Do you have any concr

[jira] [Updated] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5148: Priority: Blocker (was: Major) > Write RFC for index on functions and logical partitioning > --

[jira] [Updated] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5148: Epic Link: HUDI-512 > Write RFC for index on functions and logical partitioning > --

[jira] [Assigned] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-5148: --- Assignee: Ethan Guo > Write RFC for index on functions and logical partitioning > ---

[jira] [Updated] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5148: Fix Version/s: 0.13.0 > Write RFC for index on functions and logical partitioning >

[jira] [Created] (HUDI-5148) Write RFC for index on functions and logical partitioning

2022-11-01 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-5148: --- Summary: Write RFC for index on functions and logical partitioning Key: HUDI-5148 URL: https://issues.apache.org/jira/browse/HUDI-5148 Project: Apache Hudi Issue Type:

[GitHub] [hudi] hudi-bot commented on pull request #7063: [HUDI-5094]Remove partition fields from table schema,if the DROP_PARTITION_COLUMNS is enabled.

2022-11-01 Thread GitBox
hudi-bot commented on PR #7063: URL: https://github.com/apache/hudi/pull/7063#issuecomment-1299654055 ## CI report: * b474805bf9deeb27944189575b7c25fa03d0bf5f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…

2022-11-01 Thread GitBox
hudi-bot commented on PR #7113: URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299649595 ## CI report: * 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] hudi-bot commented on pull request #7063: [HUDI-5094]Remove partition fields from table schema,if the DROP_PARTITION_COLUMNS is enabled.

2022-11-01 Thread GitBox
hudi-bot commented on PR #7063: URL: https://github.com/apache/hudi/pull/7063#issuecomment-1299649435 ## CI report: * b474805bf9deeb27944189575b7c25fa03d0bf5f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…

2022-11-01 Thread GitBox
hudi-bot commented on PR #7113: URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299644745 ## CI report: * 6f7af524ee4693252495a4ef100bc940b44b3599 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7086: [HUDI-4624] Implement Closable for S3EventsSource

2022-11-01 Thread GitBox
hudi-bot commented on PR #7086: URL: https://github.com/apache/hudi/pull/7086#issuecomment-1299639400 ## CI report: * 3fdf983b772e3797f03a2c6fd25d9a1147351a32 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] skadooshhhh commented on pull request #7093: [WIP] Hudi platoform spark 3.0

2022-11-01 Thread GitBox
skadoos commented on PR #7093: URL: https://github.com/apache/hudi/pull/7093#issuecomment-1299612308 > @skadoos what is this PR intended for? @xushiyan Sorry closing this, this link was mentioned somewhere to make hudi cli work but this doesnt seem useful anymore. cc: @nsivab

[GitHub] [hudi] skadooshhhh closed pull request #7093: [WIP] Hudi platoform spark 3.0

2022-11-01 Thread GitBox
skadoos closed pull request #7093: [WIP] Hudi platoform spark 3.0 URL: https://github.com/apache/hudi/pull/7093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

[GitHub] [hudi] CrazyBeeline commented on issue #7102: [SUPPORT] FileNotFoundException when read from mor table

2022-11-01 Thread GitBox
CrazyBeeline commented on issue #7102: URL: https://github.com/apache/hudi/issues/7102#issuecomment-1299606850 @nsivabalan I disable metadata table and unset hoodie.write.concurrency.mode now ! but why error find when spark-shell read hive table which is synced by hudi sp

[jira] [Updated] (HUDI-5147) Flink data skipping doesn't work when HepPlanner calls copy() on HoodieTableSource

2022-11-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5147: - Labels: pull-request-available (was: ) > Flink data skipping doesn't work when HepPlanner calls c

[GitHub] [hudi] trushev opened a new pull request, #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…

2022-11-01 Thread GitBox
trushev opened a new pull request, #7113: URL: https://github.com/apache/hudi/pull/7113 … on HoodieTableSource ### Change Logs When `HepPlanner` applies `PushFilterIntoSourceScanRuleBase` it copyies `HoodieTableSource` which leads to the loss of `List filters` in `FileIndex`.

[jira] [Updated] (HUDI-5147) Flink data skipping doesn't work when HepPlanner calls copy() on HoodieTableSource

2022-11-01 Thread Alexander Trushev (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Trushev updated HUDI-5147: Description: When HepPlanner applies PushFilterIntoSourceScanRuleBase it copies HoodieTable

[jira] [Created] (HUDI-5147) Flink data skipping doesn't work when HepPlanner calls copy() on HoodieTableSource

2022-11-01 Thread Alexander Trushev (Jira)
Alexander Trushev created HUDI-5147: --- Summary: Flink data skipping doesn't work when HepPlanner calls copy() on HoodieTableSource Key: HUDI-5147 URL: https://issues.apache.org/jira/browse/HUDI-5147

[GitHub] [hudi] hudi-bot commented on pull request #7112: [MINOR] Removing spark2 scala12 combinations from readme

2022-11-01 Thread GitBox
hudi-bot commented on PR #7112: URL: https://github.com/apache/hudi/pull/7112#issuecomment-1299587851 ## CI report: * 2666f2c276781e0d840565bf1488ba685be9ed8b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] hudi-bot commented on pull request #7112: [MINOR] Removing spark2 scala12 combinations from readme

2022-11-01 Thread GitBox
hudi-bot commented on PR #7112: URL: https://github.com/apache/hudi/pull/7112#issuecomment-1299583502 ## CI report: * 2666f2c276781e0d840565bf1488ba685be9ed8b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6986: [HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
hudi-bot commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299579002 ## CI report: * cd21d9804de19bc035e6bd8f1665bcd88ffa342f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] codope commented on a diff in pull request #7111: [DOCS][MINOR] Adding useful maven commands to website

2022-11-01 Thread GitBox
codope commented on code in PR #7111: URL: https://github.com/apache/hudi/pull/7111#discussion_r1011193279 ## website/contribute/developer-setup.md: ## @@ -245,6 +245,49 @@ to effect change and source feedback, start a new email thread with the `[DISCUS by a [vote](https://www

[GitHub] [hudi] nsivabalan opened a new pull request, #7112: [MINOR] Removing spark2 scala12 combinations from readme

2022-11-01 Thread GitBox
nsivabalan opened a new pull request, #7112: URL: https://github.com/apache/hudi/pull/7112 ### Change Logs Removing spark2 scala12 combinations from readme ### Impact Removing spark2 scala12 combinations from readme ### Risk level (write none, low medium or high be

[GitHub] [hudi] nsivabalan commented on issue #7102: [SUPPORT] FileNotFoundException when read from mor table

2022-11-01 Thread GitBox
nsivabalan commented on issue #7102: URL: https://github.com/apache/hudi/issues/7102#issuecomment-1299568216 Just a note on using FileSystemBasedLockProvider. Its not recommended to be used in production. for local testing, may be you can use it. but for production, would recommend producti

[GitHub] [hudi] nsivabalan commented on issue #7102: [SUPPORT] FileNotFoundException when read from mor table

2022-11-01 Thread GitBox
nsivabalan commented on issue #7102: URL: https://github.com/apache/hudi/issues/7102#issuecomment-1299566933 may I know, is the write failing or read from MOR table is failing? bcoz, on the read side, by default metadata table is disable unless you explicitly enable it. Was just curious.

[GitHub] [hudi] nsivabalan commented on issue #7100: [SUPPORT] Custom HoodieRecordPayload for use in flink sql

2022-11-01 Thread GitBox
nsivabalan commented on issue #7100: URL: https://github.com/apache/hudi/issues/7100#issuecomment-1299564461 @yuzhaojing : can you assist here please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] nsivabalan commented on issue #7081: [SUPPORT] optimistic_concurrency_control

2022-11-01 Thread GitBox
nsivabalan commented on issue #7081: URL: https://github.com/apache/hudi/issues/7081#issuecomment-1299563321 @mandnhdaiyudfaio : gentle ping. let us know after you give it a try w/ the lock provider config. -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [hudi] nsivabalan commented on pull request #7111: [DOCS][MINOR] Adding useful maven commands to website

2022-11-01 Thread GitBox
nsivabalan commented on PR #7111: URL: https://github.com/apache/hudi/pull/7111#issuecomment-1299560582 ![Screen Shot 2022-11-01 at 9 41 26 PM](https://user-images.githubusercontent.com/513218/199400256-fdd0f65d-2998-4a02-a783-19923be595fc.png) ![Screen Shot 2022-11-01 at 9 41 35 PM](htt

[GitHub] [hudi] nsivabalan opened a new pull request, #7111: [DOCS][MINOR] Adding useful maven commands to website

2022-11-01 Thread GitBox
nsivabalan opened a new pull request, #7111: URL: https://github.com/apache/hudi/pull/7111 ### Change Logs Adding useful maven commands to our website. ### Impact Improves productivity for devs. ### Risk level (write none, low medium or high below) low.

[GitHub] [hudi] hudi-bot commented on pull request #7110: [HUDI-5146] support payload preCombine between base file and log

2022-11-01 Thread GitBox
hudi-bot commented on PR #7110: URL: https://github.com/apache/hudi/pull/7110#issuecomment-1299530779 ## CI report: * 7ba6c2f2860f0fb6e7096933bb00bbb780c61c36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] hudi-bot commented on pull request #7110: [HUDI-5146] support payload preCombine between base file and log

2022-11-01 Thread GitBox
hudi-bot commented on PR #7110: URL: https://github.com/apache/hudi/pull/7110#issuecomment-1299528284 ## CI report: * 7ba6c2f2860f0fb6e7096933bb00bbb780c61c36 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7109: [MINOR] Adding tags to assist in filtering tests from maven command line

2022-11-01 Thread GitBox
hudi-bot commented on PR #7109: URL: https://github.com/apache/hudi/pull/7109#issuecomment-1299528265 ## CI report: * d30d7c50600fe776995460702f24ba62888d64e1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] nsivabalan commented on pull request #6612: [RFC-58][HUDI-4790] a more effective HoodieMergeHandler for COW table with parquet

2022-11-01 Thread GitBox
nsivabalan commented on PR #6612: URL: https://github.com/apache/hudi/pull/6612#issuecomment-1299519063 @loukey-lj : can you respond to @guanziyue 's comment above. I will review this patch by this week. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [hudi] nsivabalan commented on pull request #7041: [HUDI-5053] Create clean complete commit when there is none to clean in order to leverage incremental cleaning

2022-11-01 Thread GitBox
nsivabalan commented on PR #7041: URL: https://github.com/apache/hudi/pull/7041#issuecomment-1299516581 hey @hussein-awala : thanks for the patch. we can definitely take up this patch. But would prefer to guard it using a new flag. reason is, for those who are running clean after every comm

[GitHub] [hudi] nsivabalan commented on pull request #6793: 【HUDI-4917】Optimized the way to get HoodieBaseFile of loadColumnRange…

2022-11-01 Thread GitBox
nsivabalan commented on PR #6793: URL: https://github.com/apache/hudi/pull/6793#issuecomment-1299514410 @boneanxs : once you are good, we can go ahead and land the patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [hudi] danny0405 closed pull request #7011: [HUDI-5102] source operator(monitor and reader) support user uid

2022-11-01 Thread GitBox
danny0405 closed pull request #7011: [HUDI-5102] source operator(monitor and reader) support user uid URL: https://github.com/apache/hudi/pull/7011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [hudi] danny0405 commented on pull request #7011: [HUDI-5102] source operator(monitor and reader) support user uid

2022-11-01 Thread GitBox
danny0405 commented on PR #7011: URL: https://github.com/apache/hudi/pull/7011#issuecomment-1299500047 Close because it is fixed in #7085. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [hudi] TJX2014 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue

2022-11-01 Thread GitBox
TJX2014 commented on code in PR #7105: URL: https://github.com/apache/hudi/pull/7105#discussion_r1011105643 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java: ## @@ -79,7 +79,7 @@ public static List>> getAllLeafFoldersWit

[GitHub] [hudi] xicm commented on issue #7058: [SUPPORT]java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.AssertionError: assertion failed: Required select columns count:

2022-11-01 Thread GitBox
xicm commented on issue #7058: URL: https://github.com/apache/hudi/issues/7058#issuecomment-1299495331 I have encountered similar error logs when I query hive, you can set ```hoodie.datasource.write.drop.partition.columns=true```, and try again. -- This is an automated message fro

[jira] [Updated] (HUDI-5146) support payload precombine between base file and log

2022-11-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5146: - Labels: pull-request-available (was: ) > support payload precombine between base file and log > -

[GitHub] [hudi] fengjian428 opened a new pull request, #7110: [HUDI-5146] support payload preCombine between base file and log

2022-11-01 Thread GitBox
fengjian428 opened a new pull request, #7110: URL: https://github.com/apache/hudi/pull/7110 ### Change Logs For now, RealtimeRecordReader doesn't support payload pre-combine between base file and log, this may cause wrong query result on Presto/Trino side ### Impact

[GitHub] [hudi] hudi-bot commented on pull request #6725: [HUDI-4881] Push down filters if possible when syncing partitions to Hive

2022-11-01 Thread GitBox
hudi-bot commented on PR #6725: URL: https://github.com/apache/hudi/pull/6725#issuecomment-1299488895 ## CI report: * 0997396d2cde3e3faba38ab15c5ae227de4f20d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1270

[jira] [Created] (HUDI-5146) support payload precombine between base file and log

2022-11-01 Thread Jian Feng (Jira)
Jian Feng created HUDI-5146: --- Summary: support payload precombine between base file and log Key: HUDI-5146 URL: https://issues.apache.org/jira/browse/HUDI-5146 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] hudi-bot commented on pull request #7086: [HUDI-4624] Implement Closable for S3EventsSource

2022-11-01 Thread GitBox
hudi-bot commented on PR #7086: URL: https://github.com/apache/hudi/pull/7086#issuecomment-1299485761 ## CI report: * 937ebe8fa4f817021a36f06b136b271dbbbd8e86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1269

[GitHub] [hudi] hudi-bot commented on pull request #6986: [HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
hudi-bot commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299485540 ## CI report: * 1d34a26c34ced9fead6af9725e68d8bb86cc934e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[GitHub] [hudi] hudi-bot commented on pull request #6725: [HUDI-4881] Push down filters if possible when syncing partitions to Hive

2022-11-01 Thread GitBox
hudi-bot commented on PR #6725: URL: https://github.com/apache/hudi/pull/6725#issuecomment-1299485239 ## CI report: * 0997396d2cde3e3faba38ab15c5ae227de4f20d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1270

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7067: [HUDI-5054] Ensure archival also considers actual clean timeline files before archiving

2022-11-01 Thread GitBox
nsivabalan commented on code in PR #7067: URL: https://github.com/apache/hudi/pull/7067#discussion_r1011093632 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java: ## @@ -459,7 +467,10 @@ private Stream getCommitInstantsToArchive()

[GitHub] [hudi] hudi-bot commented on pull request #6986: [HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
hudi-bot commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299481483 ## CI report: * 1d34a26c34ced9fead6af9725e68d8bb86cc934e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[GitHub] [hudi] hudi-bot commented on pull request #7086: [HUDI-4624] Implement Closable for S3EventsSource

2022-11-01 Thread GitBox
hudi-bot commented on PR #7086: URL: https://github.com/apache/hudi/pull/7086#issuecomment-1299481683 ## CI report: * 937ebe8fa4f817021a36f06b136b271dbbbd8e86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1269

[jira] [Commented] (HUDI-5124) Fix HoodieInternalRowFileWriter#canWrite error return tag

2022-11-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627422#comment-17627422 ] Danny Chen commented on HUDI-5124: -- Fixed via master branch: 8ce9f9c5742f0b6706e2d9ada73a

[jira] [Updated] (HUDI-5124) Fix HoodieInternalRowFileWriter#canWrite error return tag

2022-11-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5124: - Fix Version/s: 0.13.0 > Fix HoodieInternalRowFileWriter#canWrite error return tag > --

[jira] [Updated] (HUDI-5124) Fix HoodieInternalRowFileWriter#canWrite error return tag

2022-11-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-5124: - Issue Type: Bug (was: Improvement) > Fix HoodieInternalRowFileWriter#canWrite error return tag >

[jira] [Resolved] (HUDI-5124) Fix HoodieInternalRowFileWriter#canWrite error return tag

2022-11-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-5124. -- > Fix HoodieInternalRowFileWriter#canWrite error return tag > --

[GitHub] [hudi] hudi-bot commented on pull request #6986: [HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
hudi-bot commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299477234 ## CI report: * 1d34a26c34ced9fead6af9725e68d8bb86cc934e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1258

[GitHub] [hudi] nsivabalan commented on pull request #7067: [HUDI-5054] Ensure archival also considers actual clean timeline files before archiving

2022-11-01 Thread GitBox
nsivabalan commented on PR #7067: URL: https://github.com/apache/hudi/pull/7067#issuecomment-1299476654 Can you please update PR description templte. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli

2022-11-01 Thread GitBox
nsivabalan commented on code in PR #7076: URL: https://github.com/apache/hudi/pull/7076#discussion_r1011089909 ## hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java: ## @@ -206,4 +250,22 @@ private Comparable[] readCommit(GenericRecord record, boole

[hudi] branch master updated (d5ef8a39b8 -> 8ce9f9c574)

2022-11-01 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from d5ef8a39b8 [HUDI-5074] Warn if table for metastore sync has capitals in it (#7077) add 8ce9f9c574 [HUDI-5124] F

[GitHub] [hudi] danny0405 merged pull request #7107: [HUDI-5124]. Fix HoodieInternalRowFileWriter#canWrite error return tag.

2022-11-01 Thread GitBox
danny0405 merged PR #7107: URL: https://github.com/apache/hudi/pull/7107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] nsivabalan merged pull request #7077: [HUDI-5074] Warn if table for metastore sync has capitals in it

2022-11-01 Thread GitBox
nsivabalan merged PR #7077: URL: https://github.com/apache/hudi/pull/7077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

[hudi] branch master updated: [HUDI-5074] Warn if table for metastore sync has capitals in it (#7077)

2022-11-01 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d5ef8a39b8 [HUDI-5074] Warn if table for metast

[GitHub] [hudi] nsivabalan commented on pull request #7078: [HUDI-5074] Website Changes to help users debug when their table has capital letters

2022-11-01 Thread GitBox
nsivabalan commented on PR #7078: URL: https://github.com/apache/hudi/pull/7078#issuecomment-1299473558 @jonvex @bhasudha : can you check why build is failing. you can ask Sudha for help if need be. She forsees all website updates. -- This is an automated message from the Apache Git Serv

[GitHub] [hudi] nsivabalan commented on pull request #7086: [HUDI-4624] Implement Closable for S3EventsSource

2022-11-01 Thread GitBox
nsivabalan commented on PR #7086: URL: https://github.com/apache/hudi/pull/7086#issuecomment-1299471528 rebased w/ latest mster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [hudi] danny0405 commented on a diff in pull request #6733: [HUDI-4880] Fix corrupted parquet file issue left by a leaked thread in CompactFunction

2022-11-01 Thread GitBox
danny0405 commented on code in PR #6733: URL: https://github.com/apache/hudi/pull/6733#discussion_r1011085806 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionPlanOperator.java: ## @@ -129,9 +128,6 @@ private void scheduleCompaction(Hoodie

[GitHub] [hudi] xicm commented on pull request #6986: [HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
xicm commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299469454 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [hudi] xicm commented on pull request #6986: [WIP][HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
xicm commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299469078 Hi @alexeykudinkin, I create a new model ```HoodieLogFileWithPartition``` extends ```HoodieLogFile``` (is it a good way?), and init partitionValues when ```buildSplits``` in ```MergeOnReadS

[GitHub] [hudi] danny0405 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent iss

2022-11-01 Thread GitBox
danny0405 commented on code in PR #7105: URL: https://github.com/apache/hudi/pull/7105#discussion_r1011083024 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java: ## @@ -79,7 +79,7 @@ public static List>> getAllLeafFoldersW

[GitHub] [hudi] danny0405 commented on a diff in pull request #7103: [HUDI-5126] Delete duplicate configuration items PAYLOAD_CLASS_NAME

2022-11-01 Thread GitBox
danny0405 commented on code in PR #7103: URL: https://github.com/apache/hudi/pull/7103#discussion_r1011080724 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java: ## @@ -106,7 +106,7 @@ private FlinkOptions() { + "determin

[GitHub] [hudi] slfan1989 commented on pull request #7107: [HUDI-5124]. Fix HoodieInternalRowFileWriter#canWrite error return tag.

2022-11-01 Thread GitBox
slfan1989 commented on PR #7107: URL: https://github.com/apache/hudi/pull/7107#issuecomment-1299461904 @xushiyan Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] xicm commented on pull request #6986: [WIP][HUDI-5047] Add partition value in HoodieLogRecordReader when hoodie.datasource.write.drop.partition.columns=true

2022-11-01 Thread GitBox
xicm commented on PR #6986: URL: https://github.com/apache/hudi/pull/6986#issuecomment-1299448812 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [hudi] hudi-bot commented on pull request #7109: [MINOR] Adding tags to assist in filtering tests from maven command line

2022-11-01 Thread GitBox
hudi-bot commented on PR #7109: URL: https://github.com/apache/hudi/pull/7109#issuecomment-1299421210 ## CI report: * d30d7c50600fe776995460702f24ba62888d64e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1271

[GitHub] [hudi] hudi-bot commented on pull request #7109: [MINOR] Adding tags to assist in filtering tests from maven command line

2022-11-01 Thread GitBox
hudi-bot commented on PR #7109: URL: https://github.com/apache/hudi/pull/7109#issuecomment-1299417508 ## CI report: * d30d7c50600fe776995460702f24ba62888d64e1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] yanghua commented on a diff in pull request #7103: [HUDI-5126] Delete duplicate configuration items PAYLOAD_CLASS_NAME

2022-11-01 Thread GitBox
yanghua commented on code in PR #7103: URL: https://github.com/apache/hudi/pull/7103#discussion_r1011044313 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java: ## @@ -106,7 +106,7 @@ private FlinkOptions() { + "determined

[GitHub] [hudi] nsivabalan commented on pull request #6419: [HUDI-2057] CTAS Generate An External Table When Create Managed Table

2022-11-01 Thread GitBox
nsivabalan commented on PR #6419: URL: https://github.com/apache/hudi/pull/6419#issuecomment-1299403044 @xushiyan : can you assist the author and help take this home. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [hudi] nsivabalan commented on pull request #6681: [HUDI-4071] Remove default value for mandatory record key field

2022-11-01 Thread GitBox
nsivabalan commented on PR #6681: URL: https://github.com/apache/hudi/pull/6681#issuecomment-1299401275 @codope : once the patch is ready to be reviewed again, remove WIP label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] nsivabalan commented on pull request #6736: [HUDI-4894] Fix ClassCastException when using fixed type defining dec…

2022-11-01 Thread GitBox
nsivabalan commented on PR #6736: URL: https://github.com/apache/hudi/pull/6736#issuecomment-1299399947 @xushiyan : PR is assigned to you. Will you be following up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] nsivabalan commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.

2022-11-01 Thread GitBox
nsivabalan commented on PR #6705: URL: https://github.com/apache/hudi/pull/6705#issuecomment-1299399185 @watermelon12138 : can you follow up here please whenyou get a chance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] nsivabalan opened a new pull request, #7109: [MINOR] Adding tags to assist in filtering tests from maven command line

2022-11-01 Thread GitBox
nsivabalan opened a new pull request, #7109: URL: https://github.com/apache/hudi/pull/7109 ### Change Logs Adding support to filter tests based on tags. We can now filter both java and scala tests from maven command line. Eg for java: Add ``` @Tag("FlakyTests") ```

[jira] [Updated] (HUDI-5065) HoodieCleaner does not exit after completion

2022-11-01 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5065: -- Status: Patch Available (was: In Progress) > HoodieCleaner does not exit after completion > ---

[jira] [Updated] (HUDI-5065) HoodieCleaner does not exit after completion

2022-11-01 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5065: -- Status: In Progress (was: Open) > HoodieCleaner does not exit after completion > --

[GitHub] [hudi] yihua commented on issue #6137: [SUPPORT] Hudi 0.10.1 throws NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileStatusCache.putLeafFiles(Lorg/apache/hadoop/fs/Path;[Lorg

2022-11-01 Thread GitBox
yihua commented on issue #6137: URL: https://github.com/apache/hudi/issues/6137#issuecomment-1299285953 Here is one fix that is in progress, by optionally falling back to using Spark's data source with`HoodieROTablePathFilter` (how data source read is implemented pre-0.9.0 release) instead

[GitHub] [hudi] hudi-bot commented on pull request #7101: [HUDI-5065] Call close on SparkRDDWriteClient in HoodieCleaner

2022-11-01 Thread GitBox
hudi-bot commented on PR #7101: URL: https://github.com/apache/hudi/pull/7101#issuecomment-1299185005 ## CI report: * e69e44de825f93e90c245e36311a576fd427b374 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1270

[jira] [Closed] (HUDI-5099) Update stock data so that new records are added in batch_2

2022-11-01 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler closed HUDI-5099. - Resolution: Fixed Too much stuff changes with docker demo so we're not going to do this > Update

[GitHub] [hudi] jonvex closed pull request #7070: [HUDI-5099] Update stock data to be more useful for testing

2022-11-01 Thread GitBox
jonvex closed pull request #7070: [HUDI-5099] Update stock data to be more useful for testing URL: https://github.com/apache/hudi/pull/7070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [hudi] HEPBO3AH commented on issue #7062: [SUPPORT] Appeding to files during UPSERT causes executors to die due to memory issues.

2022-11-01 Thread GitBox
HEPBO3AH commented on issue #7062: URL: https://github.com/apache/hudi/issues/7062#issuecomment-1299068825 Hi @xushiyan , thank you for picking this up. Have you had any luck so far? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [hudi] hudi-bot commented on pull request #7101: [HUDI-5065] Call close on SparkRDDWriteClient in HoodieCleaner

2022-11-01 Thread GitBox
hudi-bot commented on PR #7101: URL: https://github.com/apache/hudi/pull/7101#issuecomment-1298960206 ## CI report: * 15bd8c389842379083edfe11827f802ad175fcb5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1269

[GitHub] [hudi] hudi-bot commented on pull request #7101: [HUDI-5065] Call close on SparkRDDWriteClient in HoodieCleaner

2022-11-01 Thread GitBox
hudi-bot commented on PR #7101: URL: https://github.com/apache/hudi/pull/7101#issuecomment-1298953904 ## CI report: * 15bd8c389842379083edfe11827f802ad175fcb5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1269

[GitHub] [hudi] hudi-bot commented on pull request #6725: [HUDI-4881] Push down filters if possible when syncing partitions to Hive

2022-11-01 Thread GitBox
hudi-bot commented on PR #6725: URL: https://github.com/apache/hudi/pull/6725#issuecomment-1298952968 ## CI report: * 0997396d2cde3e3faba38ab15c5ae227de4f20d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1270

[GitHub] [hudi] nsivabalan commented on pull request #7035: [HUDI-5075] Adding support to rollback residual clustering after disabling clustering

2022-11-01 Thread GitBox
nsivabalan commented on PR #7035: URL: https://github.com/apache/hudi/pull/7035#issuecomment-1298929539 @SteNicholas : sure. you can put up a patch for flink side fo things. I can help review once you have it ready -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7035: [HUDI-5075] Adding support to rollback residual clustering after disabling clustering

2022-11-01 Thread GitBox
nsivabalan commented on code in PR #7035: URL: https://github.com/apache/hudi/pull/7035#discussion_r1010757391 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -588,6 +588,19 @@ protected void runTableServicesInline(HoodieT

[jira] [Commented] (HUDI-5126) remove hudi duplicate code

2022-11-01 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627256#comment-17627256 ] Jonathan Vexler commented on HUDI-5126: --- This issue is related to [https://github.co

[GitHub] [hudi] jonvex commented on pull request #7103: [HUDI-5126] Delete duplicate configuration items PAYLOAD_CLASS_NAME

2022-11-01 Thread GitBox
jonvex commented on PR #7103: URL: https://github.com/apache/hudi/pull/7103#issuecomment-1298837016 To make it pass the validate pr check you have to write something under impact and also write a risk level in the risk level section. I'll ask around and see if someone can review the code.

[GitHub] [hudi] hudi-bot commented on pull request #6725: [HUDI-4881] Push down filters if possible when syncing partitions to Hive

2022-11-01 Thread GitBox
hudi-bot commented on PR #6725: URL: https://github.com/apache/hudi/pull/6725#issuecomment-1298771428 ## CI report: * 6fb34d880cc2eaf3d70e309dbd4a0556749739cb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=118

  1   2   3   4   >