Re: [PR] feat(spec): Add primitive data types [paimon-rust]

2024-07-24 Thread via GitHub
Aitozi commented on PR #18: URL: https://github.com/apache/paimon-rust/pull/18#issuecomment-2249574299 I have one more question, Why the DataType is not defined as trait ? If not, how the DataType used in other class. eg: `org.apache.paimon.types.DataField#asSQLString` the type string can n

Re: [PR] [cdc] upgrade to flink cdc 3.1.1 [paimon]

2024-07-24 Thread via GitHub
hadoopkandy commented on PR #3764: URL: https://github.com/apache/paimon/pull/3764#issuecomment-2249578604 similar PR: https://github.com/apache/paimon/pull/3600 https://github.com/apache/paimon/pull/2604 -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] [cdc] upgrade flink cdc version to 3.1.1 (#3599) [paimon]

2024-07-24 Thread via GitHub
hadoopkandy commented on PR #3600: URL: https://github.com/apache/paimon/pull/3600#issuecomment-2249575078 https://github.com/apache/paimon/pull/3764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [core] Support branch function for system tables. [paimon]

2024-07-24 Thread via GitHub
discivigour opened a new pull request, #3812: URL: https://github.com/apache/paimon/pull/3812 ### Purpose Support branch function for system tables. ### Tests - BranchSqlITCase - OptionsTableTest - AggregationFieldsTableTest ### API and Format

Re: [PR] feat(spec): Add primitive data types [paimon-rust]

2024-07-24 Thread via GitHub
Xuanwo commented on PR #18: URL: https://github.com/apache/paimon-rust/pull/18#issuecomment-2249557504 cc @SteNicholas, @Aitozi and @JingsongLi for another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat(spec): Add primitive data types [paimon-rust]

2024-07-24 Thread via GitHub
Xuanwo commented on code in PR #18: URL: https://github.com/apache/paimon-rust/pull/18#discussion_r1690759304 ## crates/paimon/src/spec/types.rs: ## @@ -0,0 +1,1035 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See

[I] [Feature] Are there plans to support ClickHouse? [paimon]

2024-07-24 Thread via GitHub
xleoken opened a new issue, #3810: URL: https://github.com/apache/paimon/issues/3810 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Apache Paimon is a streaming data lake pl

[PR] [core]: support rename primary key columns and bucket key columns [paimon]

2024-07-24 Thread via GitHub
zhongyujiang opened a new pull request, #3809: URL: https://github.com/apache/paimon/pull/3809 ### Purpose Support renaming primary keys and bucket keys that are not partition keys. ### Tests Added schema evolution tests and Spark e2e tests

Re: [PR] [core]: support rename primary key columns and bucket key columns [paimon]

2024-07-24 Thread via GitHub
zhongyujiang commented on PR #3809: URL: https://github.com/apache/paimon/pull/3809#issuecomment-2248271313 cc @JingsongLi @FangYongs Could you please review this? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] [core] Clarify the purpose of method findOverlappedSnapshots and move… [paimon]

2024-07-24 Thread via GitHub
LinMingQiang opened a new pull request, #3808: URL: https://github.com/apache/paimon/pull/3808 … to snapshotManager. ### Purpose Clarify the purpose of method `findOverlappedSnapshots` is to find the overlapping snapshots between the snapshot that still referenced by t

Re: [PR] feat(spec): Add Manifest List [paimon-rust]

2024-07-24 Thread via GitHub
Asura7969 commented on PR #17: URL: https://github.com/apache/paimon-rust/pull/17#issuecomment-2247755247 > Hi @Asura7969 , can you rebase and resolve conflicts. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [core] Remove duplicate ConfigOption.withDeprecatedKeys [paimon]

2024-07-24 Thread via GitHub
JingsongLi merged PR #3785: URL: https://github.com/apache/paimon/pull/3785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@paimon.ap

Re: [PR] [core] Support commit isolation level [paimon]

2024-07-24 Thread via GitHub
BsoBird commented on code in PR #3805: URL: https://github.com/apache/paimon/pull/3805#discussion_r1689057953 ## paimon-core/src/main/java/org/apache/paimon/utils/BranchManager.java: ## @@ -284,7 +284,7 @@ public void fastForward(String branchName) {

Re: [PR] feat(spec): Add Manifest List [paimon-rust]

2024-07-24 Thread via GitHub
JingsongLi commented on PR #17: URL: https://github.com/apache/paimon-rust/pull/17#issuecomment-2247357277 Hi @Asura7969 , can you rebase and resolve conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [spark] refactor delete with deletion vector [paimon]

2024-07-24 Thread via GitHub
YannByron commented on PR #3744: URL: https://github.com/apache/paimon/pull/3744#issuecomment-2247284365 The old logic is to write the unchanged deletion vector in driver side, which has a bad performance. Now, we can do this in parallel by the granularity of `partition` and `bucket`. --

Re: [PR] [core] Expire snapshot should skip branches create snapshot. [paimon]

2024-07-24 Thread via GitHub
JingsongLi commented on code in PR #3787: URL: https://github.com/apache/paimon/pull/3787#discussion_r1689392727 ## paimon-core/src/main/java/org/apache/paimon/operation/FileDeletionBase.java: ## @@ -329,7 +329,7 @@ protected void cleanUnusedManifests( public Predicate da

Re: [PR] [flink] Improve the parallelism of Flink database compression commit operator [paimon]

2024-07-24 Thread via GitHub
JingsongLi merged PR #3789: URL: https://github.com/apache/paimon/pull/3789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@paimon.ap

Re: [PR] [core] Refactor exception handling of PartitionTimeExtractor [paimon]

2024-07-24 Thread via GitHub
JingsongLi commented on code in PR #3806: URL: https://github.com/apache/paimon/pull/3806#discussion_r1689379527 ## paimon-core/src/main/java/org/apache/paimon/partition/PartitionTimeExtractor.java: ## @@ -112,22 +107,31 @@ public LocalDateTime extract(List partitionKeys, List

Re: [PR] Compaction Operator parallelism in paimon MUST be one [paimon]

2024-07-24 Thread via GitHub
tsreaper merged PR #3758: URL: https://github.com/apache/paimon/pull/3758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@paimon.apac

Re: [PR] [flink] Support options filter in FlinkGenericCatalogFactory [paimon]

2024-07-24 Thread via GitHub
xiangyuf commented on PR #3740: URL: https://github.com/apache/paimon/pull/3740#issuecomment-2247219610 > I haven't understood the specific problem yet, can you show me exact example? I think Flink `HiveCatalog` does not have problem? @JingsongLi HiveCatalogFactory will throw validate

Re: [PR] [core] Support commit isolation level [paimon]

2024-07-24 Thread via GitHub
BsoBird commented on code in PR #3805: URL: https://github.com/apache/paimon/pull/3805#discussion_r1689265509 ## paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java: ## @@ -1274,4 +1308,101 @@ static ConflictCheck noConflictCheck() { public static

Re: [PR] [core] Fix HashBucketAssigner load index too large error with refactor exception [paimon]

2024-07-24 Thread via GitHub
xuzifu666 commented on PR #3796: URL: https://github.com/apache/paimon/pull/3796#issuecomment-2247054329 > So is this not a fix but an exception msg improvement? Yes, currently we can give more advise to user about it. @izhangzhihao -- This is an automated message from the Apache G