Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1403014630 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/CatalogTestBase.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1403012486 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/CatalogTestBase.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1403012159 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1403010787 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java: ## @@ -0,0 +1,287 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1403010020 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestHelperBase.java: ## @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Spark: Fix Fast forward procedure output for non-main branches [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #8854: URL: https://github.com/apache/iceberg/pull/8854#discussion_r1403010409 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/FastForwardBranchProcedure.java: ## @@ -77,9 +77,9 @@ public InternalRow[] call(InternalR

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1403006302 ## core/src/main/java/org/apache/iceberg/UpdateSnapshotReferencesOperation.java: ## @@ -106,40 +106,42 @@ public UpdateSnapshotReferencesOperation replaceBranch(String

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1403004798 ## core/src/test/java/org/apache/iceberg/TestSnapshotManager.java: ## @@ -427,6 +427,26 @@ public void testReplaceBranchNonExistingSourceFails() { .hasMessage("

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-22 Thread via GitHub
cgpoh commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1402840153 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/MiniFlinkClusterResource.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
jia-zhengwei commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823787054 > Got it. It seems that the document https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--set-identifier-fields will give users the misconception that Spark can use` I

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
zhangbutao commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823767653 Got it. It seems that the document https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--set-identifier-fields will give users the misconception that Spark can use` IDENT

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
ConeyLiu commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823763813 Yes, `IDENTIFIER FIELDS` is mostly used in equality delete files. Right now only Flink has implemented the MOR with equality delete files. -- This is an automated message from the

Re: [PR] Hive: Refactor HiveTableOperations with common code for View. [iceberg]

2023-11-22 Thread via GitHub
nk1506 commented on code in PR #9011: URL: https://github.com/apache/iceberg/pull/9011#discussion_r1402866684 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -207,11 +195,15 @@ protected void doCommit(TableMetadata base, TableMetadata met

Re: [PR] Hive: Refactor HiveTableOperations with common code for View. [iceberg]

2023-11-22 Thread via GitHub
nk1506 commented on code in PR #9011: URL: https://github.com/apache/iceberg/pull/9011#discussion_r1402866684 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -207,11 +195,15 @@ protected void doCommit(TableMetadata base, TableMetadata met

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-11-22 Thread via GitHub
liurenjie1024 commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1402858796 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
baiyangtx commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823738673 `IDENTIFIER FIELDS` only work for Flink Streaming upsert. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
zhangbutao commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823737591 > You should use `MERGE INTO` to upsert in Spark. https://iceberg.apache.org/docs/latest/spark-writes/#merge-into Hi @ConeyLiu , IMHO, `MERGE INTO` has nothing to do with `I

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402854044 ## api/src/main/java/org/apache/iceberg/ManageSnapshots.java: ## @@ -163,26 +163,26 @@ default ManageSnapshots createBranch(String name) { ManageSnapshots r

Re: [I] Can sparksql ddl define primary key now? [iceberg]

2023-11-22 Thread via GitHub
ConeyLiu commented on issue #8508: URL: https://github.com/apache/iceberg/issues/8508#issuecomment-1823735875 You should use `MERGE INTO` to upsert in Spark. https://iceberg.apache.org/docs/latest/spark-writes/#merge-into -- This is an automated message from the Apache Git Service. To res

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402845199 ## api/src/main/java/org/apache/iceberg/ManageSnapshots.java: ## @@ -163,26 +163,26 @@ default ManageSnapshots createBranch(String name) { ManageSnapshots r

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402845199 ## api/src/main/java/org/apache/iceberg/ManageSnapshots.java: ## @@ -163,26 +163,26 @@ default ManageSnapshots createBranch(String name) { ManageSnapshots r

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-22 Thread via GitHub
cgpoh commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1402840153 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/MiniFlinkClusterResource.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
rakesh-das08 commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402838230 ## api/src/main/java/org/apache/iceberg/ManageSnapshots.java: ## @@ -163,26 +163,26 @@ default ManageSnapshots createBranch(String name) { ManageSnapshots repla

Re: [I] Prohibit rewrites of equality deletes across sequence numbers [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] closed issue #7452: Prohibit rewrites of equality deletes across sequence numbers URL: https://github.com/apache/iceberg/issues/7452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Prohibit rewrites of equality deletes across sequence numbers [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] commented on issue #7452: URL: https://github.com/apache/iceberg/issues/7452#issuecomment-1823681788 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] commented on issue #7554: URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1823681773 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] closed issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table URL: https://github.com/apache/iceberg/issues/7554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] planFiles with ParallelIterator OOM(Out of memory) [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] commented on issue #7594: URL: https://github.com/apache/iceberg/issues/7594#issuecomment-1823681742 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] planFiles with ParallelIterator OOM(Out of memory) [iceberg]

2023-11-22 Thread via GitHub
github-actions[bot] closed issue #7594: planFiles with ParallelIterator OOM(Out of memory) URL: https://github.com/apache/iceberg/issues/7594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Build: Bump mkdocs-material-extensions from 1.3 to 1.3.1 [iceberg-python]

2023-11-22 Thread via GitHub
Fokko merged PR #160: URL: https://github.com/apache/iceberg-python/pull/160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.10 [iceberg-python]

2023-11-22 Thread via GitHub
Fokko merged PR #155: URL: https://github.com/apache/iceberg-python/pull/155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mkdocstrings-python from 1.7.4 to 1.7.5 [iceberg-python]

2023-11-22 Thread via GitHub
Fokko merged PR #157: URL: https://github.com/apache/iceberg-python/pull/157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Build: Bump mkdocs-material-extensions from 1.3 to 1.3.1 [iceberg-python]

2023-11-22 Thread via GitHub
dependabot[bot] opened a new pull request, #160: URL: https://github.com/apache/iceberg-python/pull/160 Bumps [mkdocs-material-extensions](https://github.com/facelessuser/mkdocs-material-extensions) from 1.3 to 1.3.1. Release notes Sourced from https://github.com/facelessuser/mkdo

Re: [PR] Build: Bump pydantic from 2.4.2 to 2.5.1 [iceberg-python]

2023-11-22 Thread via GitHub
dependabot[bot] closed pull request #152: Build: Bump pydantic from 2.4.2 to 2.5.1 URL: https://github.com/apache/iceberg-python/pull/152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Build: Bump pydantic from 2.4.2 to 2.5.1 [iceberg-python]

2023-11-22 Thread via GitHub
dependabot[bot] commented on PR #152: URL: https://github.com/apache/iceberg-python/pull/152#issuecomment-1823611322 Superseded by #159. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Build: Bump pydantic from 2.4.2 to 2.5.2 [iceberg-python]

2023-11-22 Thread via GitHub
dependabot[bot] opened a new pull request, #159: URL: https://github.com/apache/iceberg-python/pull/159 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.4.2 to 2.5.2. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-22 Thread via GitHub
cccs-jc commented on PR #8980: URL: https://github.com/apache/iceberg/pull/8980#issuecomment-1823493615 > @cccs-jc i would recommend to make the changes 1 spark version at a time and then create back-port pr, i am not sure what is the preferred though but checking it 1 version at a time hel

Re: [I] java.lang.IllegalArgumentException: requirement failed while read migrated parquet table [iceberg]

2023-11-22 Thread via GitHub
valerio-leap commented on issue #8863: URL: https://github.com/apache/iceberg/issues/8863#issuecomment-1823493018 Any idea how to overcome this issue if upgrading is not an option? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-22 Thread via GitHub
cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1402704727 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -392,8 +392,15 @@ public Offset latestOffset(Offset startOffset, Rea

[I] With spark and hive thrift server, inserts to iceberg table in one connection are not seen in another [iceberg]

2023-11-22 Thread via GitHub
AntonOvsyannikov opened a new issue, #9135: URL: https://github.com/apache/iceberg/issues/9135 ### Query engine spark 3.5.0 with odbc hive thrift server ### Question Hello, I am not sure it's real bug or it's thrift server issue, or we are doing something complet

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-22 Thread via GitHub
cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1402703648 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -406,6 +413,30 @@ public Offset latestOffset(Offset startOffset, Rea

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402641790 ## api/src/main/java/org/apache/iceberg/ManageSnapshots.java: ## @@ -163,26 +163,26 @@ default ManageSnapshots createBranch(String name) { ManageSnapshots r

Re: [PR] Spark: Fix Fast forward procedure output for non-main branches [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #8854: URL: https://github.com/apache/iceberg/pull/8854#discussion_r1402640731 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/FastForwardBranchProcedure.java: ## @@ -77,9 +77,9 @@ public InternalRow[] call(InternalR

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on code in PR #9134: URL: https://github.com/apache/iceberg/pull/9134#discussion_r1402637674 ## core/src/test/java/org/apache/iceberg/TestSnapshotManager.java: ## @@ -427,6 +427,26 @@ public void testReplaceBranchNonExistingSourceFails() { .ha

Re: [PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar commented on PR #9134: URL: https://github.com/apache/iceberg/pull/9134#issuecomment-1823395448 cc @rakesh-das08 This stems from our discussion on https://github.com/apache/iceberg/pull/8854#discussion_r1361985873 -- This is an automated message from the Apache Git Servi

[PR] API, Core: Fix naming in fastForwardBranch/replaceBranch APIs [iceberg]

2023-11-22 Thread via GitHub
amogh-jahagirdar opened a new pull request, #9134: URL: https://github.com/apache/iceberg/pull/9134 This change fixes some very confusing naming from my original implementation of the fastForward/replaceBranch APIs. Prior to this change, the APIs had the following naming: ```

Re: [PR] Spec: Clarify partition equality [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1402521562 ## format/spec.md: ## @@ -305,6 +305,10 @@ The source column, selected by id, must be a primitive type and cannot be contai Partition specs capture the transform fro

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
tomtongue commented on PR #9129: URL: https://github.com/apache/iceberg/pull/9129#issuecomment-1823273318 Add the following classes that basically cover JUnit5 and AssertJ styles. Each class has a lot of inheritants so that new classes are created for now. @nastra -- This is an automate

Re: [PR] Spec: Clarify partition equality [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1402519684 ## format/spec.md: ## @@ -607,6 +611,8 @@ Notes: 1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in t

Re: [PR] Spec: Clarify partition equality [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1402518409 ## format/spec.md: ## @@ -607,6 +611,8 @@ Notes: 1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in t

Re: [PR] Core: Schema for a branch should return table schema [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on code in PR #9131: URL: https://github.com/apache/iceberg/pull/9131#discussion_r1402479551 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java: ## @@ -425,16 +426,35 @@ public void testSnapshotSelectionByBranchWithSche

Re: [PR] Core: Schema for a branch should return table schema [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on code in PR #9131: URL: https://github.com/apache/iceberg/pull/9131#discussion_r1402477101 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java: ## @@ -425,16 +426,35 @@ public void testSnapshotSelectionByBranchWithSche

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1402429495 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -429,6 +444,30 @@ public Builder setAll(Map properties) { return t

Re: [I] Failed to create namespace using spark sql based on iceberg hadoop catalog (rest catalog) [iceberg]

2023-11-22 Thread via GitHub
rdblue commented on issue #9072: URL: https://github.com/apache/iceberg/issues/9072#issuecomment-1823151897 Yeah, it looks like the issue is that the REST catalog will create properties that can't be handled when the underlying catalog is a Hadoop catalog. -- This is an automated message

Re: [I] Location Ownership [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on issue #9133: URL: https://github.com/apache/iceberg/issues/9133#issuecomment-1823148668 Related issue: https://github.com/apache/iceberg/issues/4159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Multi-Column Transforms [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on issue #9132: URL: https://github.com/apache/iceberg/issues/9132#issuecomment-1823145815 Related issue: https://github.com/apache/iceberg/issues/8258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Spark SQL DESCRIBE not showing proper schema on a branch [iceberg]

2023-11-22 Thread via GitHub
nastra commented on issue #9026: URL: https://github.com/apache/iceberg/issues/9026#issuecomment-1823100622 @cccs-eric I took another look at this. When describing the schema of a branch, then that output should always show the schema of the current table itself and not the schema of the cu

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1402326168 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/MiniFlinkClusterResource.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-11-22 Thread via GitHub
adutra commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1823031640 I took the time to investigate what it would take to use the client-side classes consistently. My final opinion is that it's not a good solution, for the below reasons: * Client-sid

Re: [PR] Replace i64 with DateTime [iceberg-rust]

2023-11-22 Thread via GitHub
Fokko commented on PR #94: URL: https://github.com/apache/iceberg-rust/pull/94#issuecomment-1822991443 Thanks @fqaiser94 for the PR, and @liurenjie1024 & @Xuanwo for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Replace i64 with DateTime [iceberg-rust]

2023-11-22 Thread via GitHub
Fokko merged PR #94: URL: https://github.com/apache/iceberg-rust/pull/94 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Substitue in memory data struct's timestamp type for DataTime rather i64 to simplify usage. [iceberg-rust]

2023-11-22 Thread via GitHub
Fokko closed issue #90: Substitue in memory data struct's timestamp type for DataTime rather i64 to simplify usage. URL: https://github.com/apache/iceberg-rust/issues/90 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Core: Schema for a branch should return table schema [iceberg]

2023-11-22 Thread via GitHub
nastra opened a new pull request, #9131: URL: https://github.com/apache/iceberg/pull/9131 When retrieving the schema for branch we should always return the table schema instead of the snapshot schema. This is because the table schema is the schema that will be used when the branch will be c

Re: [PR] Add `iceberg-bom` artifact [iceberg]

2023-11-22 Thread via GitHub
snazy commented on PR #8065: URL: https://github.com/apache/iceberg/pull/8065#issuecomment-1822721081 Ping for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
tomtongue commented on PR #9129: URL: https://github.com/apache/iceberg/pull/9129#issuecomment-1822698391 Thank you! I'm working on changes. Let me do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] The query result of `col > x` may be incorrect when there are NaN values in the column `col` [iceberg]

2023-11-22 Thread via GitHub
zhongyujiang commented on issue #9130: URL: https://github.com/apache/iceberg/issues/9130#issuecomment-1822694319 Since NaNs are not permitted as lower or upper bounds in Iceberg, should we convert Spark's `col > x` filter to Iceberg's `col > x OR col is NaN` if `col` is `float` or `double`

[I] The query result of `col > x` may be incorrect when there are NaN values in the column `col` [iceberg]

2023-11-22 Thread via GitHub
zhongyujiang opened a new issue, #9130: URL: https://github.com/apache/iceberg/issues/9130 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 I found that when there are NaN values in the columns, the resul

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-11-22 Thread via GitHub
mobley-trent commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1401933787 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Docs: Remove UNIQUE keyword as it is not supported in Flink [iceberg]

2023-11-22 Thread via GitHub
PrabhuJoseph commented on code in PR #9046: URL: https://github.com/apache/iceberg/pull/9046#discussion_r1401936374 ## docs/flink-writes.md: ## @@ -69,7 +69,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ```sql CREATE TABLE `hive_catal

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-11-22 Thread via GitHub
mobley-trent commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1401933787 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2023-11-22 Thread via GitHub
lyohar commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-1822629812 Got similar issue in 1.4.2, spark 3.5 My iceberg catalogue in Spark is configured via org.apache.iceberg.aws.s3.S3FileIO filesystem. I store files using s3 prefix; Hovewer, when

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on PR #8909: URL: https://github.com/apache/iceberg/pull/8909#issuecomment-1822627656 @snazy: I have addressed most of the comments and replied the reason for unhandled comments. Please take a look again. Thanks. -- This is an automated message from the Apache

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401930877 ## nessie/src/main/java/org/apache/iceberg/nessie/UpdateableReference.java: ## @@ -62,7 +62,7 @@ public Reference getReference() { public void checkMutable()

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401930448 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +617,72 @@ public void close() { api.close(); } } + + publ

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401929783 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -378,27 +400,72 @@ public void renameTable(TableIdentifier from, TableIdentifier

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401929044 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -170,11 +184,12 @@ private TableIdentifier toIdentifier(EntriesResponse.Entry en

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401929044 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -170,11 +184,12 @@ private TableIdentifier toIdentifier(EntriesResponse.Entry en

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-22 Thread via GitHub
cgpoh commented on PR #9120: URL: https://github.com/apache/iceberg/pull/9120#issuecomment-1822599964 @pvary will do. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on PR #9129: URL: https://github.com/apache/iceberg/pull/9129#issuecomment-1822596157 @tomtongue I think we first need to have a JUnit5 equivalent of `SparkTestBase` and `SparkTestBaseWithCatalog` before we can migrate `SparkCatalogTestBase` -- This is an automated messa

Re: [PR] Create SparkCatalogTestBase class for the migration to JUnit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9129: URL: https://github.com/apache/iceberg/pull/9129#discussion_r1401906293 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/SparkCatalogTestBaseForJU5.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401896252 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + publ

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on PR #9122: URL: https://github.com/apache/iceberg/pull/9122#issuecomment-1822571206 > Btw, should I squash the commits together or you will do that when merging the pr? commits will be squashed when the PR is getting merged, so it's up to you if you want to squash

Re: [PR] Support tencent COS fileIO [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9048: URL: https://github.com/apache/iceberg/pull/9048#discussion_r1401887508 ## build.gradle: ## @@ -948,6 +948,48 @@ project(':iceberg-snowflake') { } } +project(':iceberg-tencentcloud') { + test { +useJUnitPlatform() + } + + depend

Re: [PR] Support tencent COS fileIO [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9048: URL: https://github.com/apache/iceberg/pull/9048#discussion_r1401886198 ## build.gradle: ## @@ -948,6 +948,48 @@ project(':iceberg-snowflake') { } } +project(':iceberg-tencentcloud') { + test { +useJUnitPlatform() + } + + depend

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-22 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1401883956 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Support tencent COS fileIO [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9048: URL: https://github.com/apache/iceberg/pull/9048#discussion_r1401883707 ## build.gradle: ## @@ -948,6 +948,48 @@ project(':iceberg-snowflake') { } } +project(':iceberg-tencentcloud') { + test { +useJUnitPlatform() + } + + depend

Re: [PR] Support tencent COS fileIO [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9048: URL: https://github.com/apache/iceberg/pull/9048#discussion_r1401825158 ## tencent/src/test/java/org/apache/iceberg/tencentcloud/cos/TestCosClientFactories.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Hive: Refactor HiveTableOperations with common code for View. [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #9011: URL: https://github.com/apache/iceberg/pull/9011#discussion_r1401876287 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -207,11 +195,15 @@ protected void doCommit(TableMetadata base, TableMetadata meta

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-22 Thread via GitHub
pvary commented on PR #9120: URL: https://github.com/apache/iceberg/pull/9120#issuecomment-1822550315 @cgpoh: Please run `./gradlew spotlessApply` before pushing the changes to fix splotless errors and minimally `./gradlew build -x test -x javadoc -x integrationTest` to catch checkstyle err

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401866044 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,451 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401857507 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-22 Thread via GitHub
lisirrx commented on PR #9122: URL: https://github.com/apache/iceberg/pull/9122#issuecomment-1822488153 > just a few small things to fix, but overall this LGTM. > > I also checked how the env variables are being used and they are not set by our CI. They were introduced by #3687, so pi

Re: [PR] Support tencent COS fileIO [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9048: URL: https://github.com/apache/iceberg/pull/9048#discussion_r1401806041 ## tencent/src/test/java/org/apache/iceberg/tencentcloud/cos/TestCosFileIO.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401803993 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401800615 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-22 Thread via GitHub
nastra commented on code in PR #9122: URL: https://github.com/apache/iceberg/pull/9122#discussion_r1401773295 ## aliyun/src/test/java/org/apache/iceberg/aliyun/TestAliyunClientFactories.java: ## @@ -22,43 +22,45 @@ import java.util.Map; import org.apache.iceberg.relocated.com.

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401797073 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401790452 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401785416 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailoverWithWatermarkExtractor.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apac

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-22 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401784310 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software

  1   2   >