Re: [PR] [MINOR][DO NOT MERGE] Turn on 1.0.0 features by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11621: URL: https://github.com/apache/hudi/pull/11621#issuecomment-2224792280 ## CI report: * aee780891cfe22ae23e41928b313c2cb7f9f Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on 1.0.0 features by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11621: URL: https://github.com/apache/hudi/pull/11621#issuecomment-2224772039 ## CI report: * aee780891cfe22ae23e41928b313c2cb7f9f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11620: URL: https://github.com/apache/hudi/pull/11620#issuecomment-2224771905 ## CI report: * 661b18187f772d0b21d0fbebd3cdb7209ef2f1f2 Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on all indexingand data skipping by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11619: URL: https://github.com/apache/hudi/pull/11619#issuecomment-2224771822 ## CI report: * 706d366ca54db41dd40348f09b513ad3da51f94e Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on all indexingand data skipping by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11619: URL: https://github.com/apache/hudi/pull/11619#issuecomment-2224751243 ## CI report: * 706d366ca54db41dd40348f09b513ad3da51f94e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11620: URL: https://github.com/apache/hudi/pull/11620#issuecomment-2224751375 ## CI report: * 661b18187f772d0b21d0fbebd3cdb7209ef2f1f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11615: URL: https://github.com/apache/hudi/pull/11615#issuecomment-2224751113 ## CI report: * 95e8a5740c50d337f3c2033d1e5e380865fd6fed Azure:

(hudi) branch master updated: [HUDI-7979] Adjusting defaults with spillable map memory (#11611)

2024-07-11 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9ee96655094 [HUDI-7979] Adjusting defaults

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
nsivabalan merged PR #11611: URL: https://github.com/apache/hudi/pull/11611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
nsivabalan commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2224664688 I have not tested it as of now. but will be testing it in a week or so. bcoz, spillable is not enabled out of the box. -- This is an automated message from the Apache Git Service.

[PR] [MINOR][DO NOT MERGE] Turn on 1.0.0 features by default [hudi]

2024-07-11 Thread via GitHub
codope opened a new pull request, #11621: URL: https://github.com/apache/hudi/pull/11621 ### Change Logs **Do not merge** The purpose of this PR is to enable 1.0.0 features, including indexes, native hfile reader, instant state from timeline, by default and identify failures.

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11615: URL: https://github.com/apache/hudi/pull/11615#issuecomment-2224621535 ## CI report: * 95e8a5740c50d337f3c2033d1e5e380865fd6fed Azure:

[PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-11 Thread via GitHub
codope opened a new pull request, #11620: URL: https://github.com/apache/hudi/pull/11620 ### Change Logs **Do not merge** The purpose of this PR is to enable data skipping and native HFile reader by default and identify failures. ### Impact Stability for 1.0.0

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675313052 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,167 @@ +/* + * Licensed to

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675310720 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,140 @@ +/* + * Licensed to

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675313052 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,167 @@ +/* + * Licensed to

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675313052 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,167 @@ +/* + * Licensed to

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675312608 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -99,6 +99,33 @@ class

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675312608 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -99,6 +99,33 @@ class

[PR] [MINOR][DO NOT MERGE] Turn on all indexingand data skipping by default [hudi]

2024-07-11 Thread via GitHub
codope opened a new pull request, #11619: URL: https://github.com/apache/hudi/pull/11619 ### Change Logs **Do not merge** The purpose of this PR is to enable indexes and data skipping by default and identify failures. ### Impact Stability for 1.0.0 ###

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
geserdugarov commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1675310720 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,140 @@ +/* + * Licensed to

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675307482 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675305938 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675303077 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675301492 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675296125 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7882][WIP] Adding RFC 78 for bridge release to assist users to migrate to 1.x from 0.x [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11514: URL: https://github.com/apache/hudi/pull/11514#discussion_r1675266281 ## rfc/rfc-78/rfc-78.md: ## @@ -0,0 +1,339 @@ + +# RFC-76: [Bridge release for 1.x] + +## Proposers + +- @nsivabalan +- @vbalaji + +## Approvers + - @yihua + -

Re: [PR] [HUDI-7980] Optimize the configuration content when performing clustering with row writer [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11614: URL: https://github.com/apache/hudi/pull/11614#issuecomment-2224406619 ## CI report: * 9167ed3b7e6f0a1be32a91aa4c034df1b14355d9 Azure:

Re: [PR] [HUDI-7976] Fix BUG introduced in HUDI-7955 due to usage of wrong class [hudi]

2024-07-11 Thread via GitHub
voonhous commented on code in PR #11612: URL: https://github.com/apache/hudi/pull/11612#discussion_r1675052225 ## hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive3Shim.java: ## @@ -40,7 +40,8 @@ public class Hive3Shim implements HiveShim { public static

Re: [PR] [HUDI-7980] Optimize the configuration content when performing clustering with row writer [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11614: URL: https://github.com/apache/hudi/pull/11614#issuecomment-2224329106 ## CI report: * 305cfba4c163a2d70bcbeff8029c9f2a2d205a3c Azure:

Re: [PR] [HUDI-7980] Optimize the configuration content when performing clustering with row writer [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11614: URL: https://github.com/apache/hudi/pull/11614#issuecomment-2224324193 ## CI report: * 305cfba4c163a2d70bcbeff8029c9f2a2d205a3c Azure:

Re: [I] Tracking issue for release `0.1.0` [hudi-rs]

2024-07-11 Thread via GitHub
xushiyan commented on issue #62: URL: https://github.com/apache/hudi-rs/issues/62#issuecomment-2224322166 # Changelog ## [release-0.1.0-rc.2] - 2024-07-12 ### Build - Bump version to 0.1.0-rc.2 ## [release-0.1.0-rc.1] - 2024-07-12 ###  Features -

svn commit: r70245 - in /dev/hudi/hudi-rs-0.1.0-rc.2: ./ hudi-rs-0.1.0-rc.2.src.tgz hudi-rs-0.1.0-rc.2.src.tgz.asc hudi-rs-0.1.0-rc.2.src.tgz.sha512

2024-07-11 Thread xushiyan
Author: xushiyan Date: Fri Jul 12 02:06:32 2024 New Revision: 70245 Log: add hudi-rs-0.1.0-rc.2 Added: dev/hudi/hudi-rs-0.1.0-rc.2/ dev/hudi/hudi-rs-0.1.0-rc.2/hudi-rs-0.1.0-rc.2.src.tgz (with props) dev/hudi/hudi-rs-0.1.0-rc.2/hudi-rs-0.1.0-rc.2.src.tgz.asc

(hudi-rs) tag release-0.1.0-rc.2 created (now a5321c9)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to tag release-0.1.0-rc.2 in repository https://gitbox.apache.org/repos/asf/hudi-rs.git at a5321c9 (commit) No new revisions were added by this update.

(hudi-rs) branch release/0.1.x updated: build: bump version to 0.1.0-rc.2

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch release/0.1.x in repository https://gitbox.apache.org/repos/asf/hudi-rs.git The following commit(s) were added to refs/heads/release/0.1.x by this push: new a5321c9 build: bump version

(hudi-rs) branch main updated: build: bump version to 0.2.0 (#61)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hudi-rs.git The following commit(s) were added to refs/heads/main by this push: new 7b2d199 build: bump version to 0.2.0 (#61)

Re: [PR] build: bump version to 0.2.0 [hudi-rs]

2024-07-11 Thread via GitHub
xushiyan merged PR #61: URL: https://github.com/apache/hudi-rs/pull/61 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi-rs) tag release-0.1.0-rc.1 created (now ebf8fdb)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to tag release-0.1.0-rc.1 in repository https://gitbox.apache.org/repos/asf/hudi-rs.git at ebf8fdb (commit) No new revisions were added by this update.

(hudi-rs) tag release-0.1.0-rc.1 deleted (was 8250682)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to tag release-0.1.0-rc.1 in repository https://gitbox.apache.org/repos/asf/hudi-rs.git *** WARNING: tag release-0.1.0-rc.1 was deleted! *** was 8250682 build: bump version to 0.1.0-rc.1 This

(hudi-rs) branch release-0.1.x deleted (was 8250682)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch release-0.1.x in repository https://gitbox.apache.org/repos/asf/hudi-rs.git was 8250682 build: bump version to 0.1.0-rc.1 The revisions that were on this branch are still contained in

(hudi-rs) 01/01: build: bump version to 0.1.0-rc.1

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch release/0.1.x in repository https://gitbox.apache.org/repos/asf/hudi-rs.git commit ebf8fdbb2151bbfe1c2a99ee03ef468af8c36921 Author: Shiyan Xu <2701446+xushi...@users.noreply.github.com>

(hudi-rs) branch release/0.1.x updated (8250682 -> ebf8fdb)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch release/0.1.x in repository https://gitbox.apache.org/repos/asf/hudi-rs.git omit 8250682 build: bump version to 0.1.0-rc.1 add 4918c19 chore: add scripts to streamline source

(hudi-rs) branch main updated: chore: add scripts to streamline source release (#64)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hudi-rs.git The following commit(s) were added to refs/heads/main by this push: new 4918c19 chore: add scripts to streamline source

Re: [PR] chore: add scripts to streamline source release [hudi-rs]

2024-07-11 Thread via GitHub
xushiyan merged PR #64: URL: https://github.com/apache/hudi-rs/pull/64 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Assigned] (HUDI-7972) Add fallback for deletion vector in 0.16.x reader while reading 1.x tables

2024-07-11 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-7972: - Assignee: sivabalan narayanan > Add fallback for deletion vector in 0.16.x

(hudi-rs) tag release-0.1.0-rc.1 deleted (was 8250682)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to tag release-0.1.0-rc.1 in repository https://gitbox.apache.org/repos/asf/hudi-rs.git *** WARNING: tag release-0.1.0-rc.1 was deleted! *** was 8250682 build: bump version to 0.1.0-rc.1 The

(hudi-rs) tag release-0.1.0-rc.1 created (now 8250682)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to tag release-0.1.0-rc.1 in repository https://gitbox.apache.org/repos/asf/hudi-rs.git at 8250682 (commit) No new revisions were added by this update.

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674899742 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java: ## @@ -437,6 +437,15 @@ static boolean compareTimestamps(String commit1,

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674898261 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java: ## @@ -553,13 +552,62 @@ public Pair> getDeletePaths(String

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674893075 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java: ## @@ -264,20 +268,46 @@ private List

Re: [PR] [HUDI-7976] Fix BUG introduced in HUDI-7955 due to usage of wrong class [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on code in PR #11612: URL: https://github.com/apache/hudi/pull/11612#discussion_r1674887232 ## hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive3Shim.java: ## @@ -40,7 +40,8 @@ public class Hive3Shim implements HiveShim { public static

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2224205748 > if some user is retaining 100 file versions, then the amount of memory required to store file group is 100x of that. So, just taking some educated guess to allot 5% to bootstrap, 2%

(hudi) branch master updated: [MINOR] Codestyle of license and package statement (#11617)

2024-07-11 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new c51f4675e2d [MINOR] Codestyle of license and

Re: [PR] [MINOR] Codestyle of license and package statement ordering [hudi]

2024-07-11 Thread via GitHub
danny0405 merged PR #11617: URL: https://github.com/apache/hudi/pull/11617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] hive sql查询hudi分区表,如果分区字段不是表最后一列,解析parquet文件后返回的数据,没有查询分区字段单在分区字段列位置自动增加了分区字段的值,导致后续列错误发生类型转换问题 [hudi]

2024-07-11 Thread via GitHub
danny0405 commented on issue #11609: URL: https://github.com/apache/hudi/issues/11609#issuecomment-2224194005 > We needs to reorder the fields when writing with flink Maybe we should create table with schema including the partition fields as the last. -- This is an automated

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2224087398 ## CI report: * 4b71cd0fc10d71c1a43cd242f721e3057a4b3552 UNKNOWN * eaf91aa6e9a69c17572734c573de424c1e2b4da1 Azure:

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2224080682 ## CI report: * dac29c7e89201f0ced6d394bf6fd4a5c0622167b UNKNOWN * dca6e353979bdd1b9d5f1a3247737e847658f905 Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2224074218 ## CI report: * 4b71cd0fc10d71c1a43cd242f721e3057a4b3552 UNKNOWN * a52f48c62a8afd8f5b5ec50140c5199ea5f450af Azure:

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2224073943 ## CI report: * dac29c7e89201f0ced6d394bf6fd4a5c0622167b UNKNOWN * e8ad55251f5c98b061208ceb2e52637b345e9db0 Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2224010972 ## CI report: * 4de9be3830a22f5dc11538353b469c2083f1a35e Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-222389 ## CI report: * 4de9be3830a22f5dc11538353b469c2083f1a35e Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2223986612 ## CI report: * 4de9be3830a22f5dc11538353b469c2083f1a35e Azure:

Re: [PR] [HUDI-7970] Add support to read partition fields when partition type is also stored in table config [hudi]

2024-07-11 Thread via GitHub
CTTY commented on code in PR #11601: URL: https://github.com/apache/hudi/pull/11601#discussion_r1674723507 ## hudi-common/src/main/java/org/apache/hudi/common/util/TableConfigUtils.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2223925757 ## CI report: * 4de9be3830a22f5dc11538353b469c2083f1a35e Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2223914136 ## CI report: * 4de9be3830a22f5dc11538353b469c2083f1a35e Azure:

Re: [PR] [HUDI-7979] Adjusting defaults with spillable map memory [hudi]

2024-07-11 Thread via GitHub
nsivabalan commented on PR #11611: URL: https://github.com/apache/hudi/pull/11611#issuecomment-2223895790 @danny0405 : generally number of pending compactions is relatively proportional to number of file groups. but to store actual file groups, if some user is retaining 100 file versions,

svn commit: r70243 - /dev/hudi/KEYS

2024-07-11 Thread xushiyan
Author: xushiyan Date: Thu Jul 11 20:27:08 2024 New Revision: 70243 Log: add signing key for Shiyan Xu Modified: dev/hudi/KEYS Modified: dev/hudi/KEYS == --- dev/hudi/KEYS (original) +++ dev/hudi/KEYS Thu Jul 11

svn commit: r70242 - in /dev/hudi/hudi-rs-0.1.0-rc.1: ./ hudi-rs-0.1.0-rc.1.src.tar.gz hudi-rs-0.1.0-rc.1.src.tar.gz.asc hudi-rs-0.1.0-rc.1.src.tar.gz.sha512

2024-07-11 Thread xushiyan
Author: xushiyan Date: Thu Jul 11 18:50:01 2024 New Revision: 70242 Log: add hudi-rs-0.1.0-rc.1 Added: dev/hudi/hudi-rs-0.1.0-rc.1/ dev/hudi/hudi-rs-0.1.0-rc.1/hudi-rs-0.1.0-rc.1.src.tar.gz (with props) dev/hudi/hudi-rs-0.1.0-rc.1/hudi-rs-0.1.0-rc.1.src.tar.gz.asc

[jira] [Updated] (HUDI-6227) Improve logging in ReflectionUtils.hasConstructor

2024-07-11 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6227: Fix Version/s: 0.14.0 (was: 1.1.0) > Improve logging in

[jira] [Closed] (HUDI-6227) Improve logging in ReflectionUtils.hasConstructor

2024-07-11 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-6227. --- Resolution: Fixed > Improve logging in ReflectionUtils.hasConstructor >

Re: [PR] [HUDI-7976] Fix BUG introduced in HUDI-7955 due to usage of wrong class [hudi]

2024-07-11 Thread via GitHub
voonhous commented on code in PR #11612: URL: https://github.com/apache/hudi/pull/11612#discussion_r1674475818 ## hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive3Shim.java: ## @@ -40,7 +40,8 @@ public class Hive3Shim implements HiveShim { public static

Re: [PR] chore: add script to streamline source release [hudi-rs]

2024-07-11 Thread via GitHub
codecov[bot] commented on PR #64: URL: https://github.com/apache/hudi-rs/pull/64#issuecomment-2223598349 ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/64?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All

(hudi-rs) branch release/0.1.x created (now 8250682)

2024-07-11 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch release/0.1.x in repository https://gitbox.apache.org/repos/asf/hudi-rs.git at 8250682 build: bump version to 0.1.0-rc.1 No new revisions were added by this update.

[PR] chore: add script to streamline source release [hudi-rs]

2024-07-11 Thread via GitHub
xushiyan opened a new pull request, #64: URL: https://github.com/apache/hudi-rs/pull/64 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Exception while using HoodieStreamer protobuf data from Kafka [hudi]

2024-07-11 Thread via GitHub
gauravg1977 commented on issue #11598: URL: https://github.com/apache/hudi/issues/11598#issuecomment-2223445290 Thanks @the-other-tim-brown ! Looking forward to it!! Is my overall usage as described (arguments and properties used) of the HoodieStreamer for consuming proto from Kafka

Re: [PR] [HUDI-7774] Add Avro Logical type support for Merciful Java convertor [hudi]

2024-07-11 Thread via GitHub
Davis-Zhang-Onehouse commented on PR #11265: URL: https://github.com/apache/hudi/pull/11265#issuecomment-2223310964 @codope is onboarded with the change. @yihua your review is required. Please take a look when you get a chance (after you got bandwidth from your current priority) -- This

[PR] [DOCS] Release notes 1.0.0-beta2 [hudi]

2024-07-11 Thread via GitHub
codope opened a new pull request, #11618: URL: https://github.com/apache/hudi/pull/11618 ### Change Logs Release notes 1.0.0-beta2 ### Impact release notes ### Risk level (write none, low medium or high below) low ### Documentation Update

Re: [I] [SUPPORT] Mandatory registration of `HoodieSparkKryoRegistrar` using PySpark for 1.0.0-beta2-rc2 [hudi]

2024-07-11 Thread via GitHub
codope commented on issue #11616: URL: https://github.com/apache/hudi/issues/11616#issuecomment-2223235971 @geserdugarov I think we should keep the same behaviour as 0.15.0 release. So, I would treat it as a bug. Do you mind firing a fix for this? -- This is an automated message from the

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-11 Thread via GitHub
KnightChess commented on PR #11568: URL: https://github.com/apache/hudi/pull/11568#issuecomment-2223061487 @codope if specified, look like work well. https://github.com/apache/hudi/assets/20125927/9ab0f7b8-76d1-4281-b693-bc1de1e412cc;> -- This is an automated message from the

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-11 Thread via GitHub
KnightChess commented on PR #11568: URL: https://github.com/apache/hudi/pull/11568#issuecomment-2223023363 @codope got it o( ̄▽ ̄)o -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-11 Thread via GitHub
codope commented on PR #11568: URL: https://github.com/apache/hudi/pull/11568#issuecomment-998105 @KnightChess Yes it is not due to this PR. I just tested by creating a parquet table and it's still the same behavior. So, issue is something unrelated to Hudi. You could try as well:

[jira] [Created] (HUDI-7981) page not found error on documentation page

2024-07-11 Thread prabodh agarwal (Jira)
prabodh agarwal created HUDI-7981: - Summary: page not found error on documentation page Key: HUDI-7981 URL: https://issues.apache.org/jira/browse/HUDI-7981 Project: Apache Hudi Issue Type:

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-11 Thread via GitHub
KnightChess commented on PR #11568: URL: https://github.com/apache/hudi/pull/11568#issuecomment-856656 @codope hi, need specified like `insert into aaa (id, day, price, name, hour) values (2, '01', 12.2, 'bbb', '02')` will trigger this logical, but you case not specified columns, and I

Re: [PR] [MINOR] Codestyle of license and package statement ordering [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11617: URL: https://github.com/apache/hudi/pull/11617#issuecomment-833970 ## CI report: * 1bb7e5f2471aef65ef48d0d3860d612d035c486e Azure:

Re: [PR] [HUDI-7976] Fix BUG introduced in HUDI-7955 due to usage of wrong class [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11612: URL: https://github.com/apache/hudi/pull/11612#issuecomment-833753 ## CI report: * 6ceca16530ca218d73a2624c18b09bd07b28b116 Azure:

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
codope commented on code in PR #11615: URL: https://github.com/apache/hudi/pull/11615#discussion_r1673923538 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithTimestampKeyGenerator.scala: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the

[jira] [Commented] (HUDI-7709) ClassCastException while reading the data using TimestampBasedKeyGenerator

2024-07-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17865037#comment-17865037 ] Sagar Sumit commented on HUDI-7709: --- I was hitting NPE for non-partitioned table. Let me take a look at

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-11 Thread via GitHub
codope commented on PR #11568: URL: https://github.com/apache/hudi/pull/11568#issuecomment-779094 @KnightChess Looks like this PR does not handle partition spec? Can you please check HUDI-7964? -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Updated] (HUDI-7964) Partitions not created correctly with SQL when multiple partitions specified out of order

2024-07-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7964: -- Attachment: Screenshot 2024-07-11 at 5.43.41 PM.png > Partitions not created correctly with SQL when

[jira] [Updated] (HUDI-7964) Partitions not created correctly with SQL when multiple partitions specified out of order

2024-07-11 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7964: -- Description: When multiple partitions are specified out of order (as compared to the order of fields

Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-73 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * 39affbca1f586891c4567ab9aa99713011d2dee4 Azure:

Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11553: URL: https://github.com/apache/hudi/pull/11553#issuecomment-721280 ## CI report: * c5bde7f662a930b9a10b79fa38f9567300c0674a UNKNOWN * dd1dde0eedca311f2fd0803a4b86f0689a5f528a Azure:

Re: [PR] [MINOR] Codestyle of license and package statement ordering [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11617: URL: https://github.com/apache/hudi/pull/11617#issuecomment-705452 ## CI report: * 1bb7e5f2471aef65ef48d0d3860d612d035c486e Azure:

Re: [PR] [HUDI-7709] Pass partition paths as partition column values if `TimestampBasedKeyGenerator` is used [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11615: URL: https://github.com/apache/hudi/pull/11615#issuecomment-705387 ## CI report: * 95e8a5740c50d337f3c2033d1e5e380865fd6fed Azure:

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-704418 ## CI report: * 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure:

Re: [I] hive sql查询hudi分区表,如果分区字段不是表最后一列,解析parquet文件后返回的数据,没有查询分区字段单在分区字段列位置自动增加了分区字段的值,导致后续列错误发生类型转换问题 [hudi]

2024-07-11 Thread via GitHub
xicm commented on issue #11609: URL: https://github.com/apache/hudi/issues/11609#issuecomment-660526 You can put the partition field last to work around. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [MINOR] Codestyle of license and package statement ordering [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11617: URL: https://github.com/apache/hudi/pull/11617#issuecomment-625137 ## CI report: * 1bb7e5f2471aef65ef48d0d3860d612d035c486e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7976] Fix BUG introduced in HUDI-7955 due to usage of wrong class [hudi]

2024-07-11 Thread via GitHub
hudi-bot commented on PR #11612: URL: https://github.com/apache/hudi/pull/11612#issuecomment-624984 ## CI report: * 8669c1c9afa99b08f866c97ed18eac0446cb1b36 Azure:

  1   2   >