Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568292423 ## data/src/test/java/org/apache/iceberg/data/TestReadProjection.java: ## @@ -76,20 +78,21 @@ public void testSpecialCharacterProjection() throws Exception { Rec

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568271659 ## data/src/test/java/org/apache/iceberg/data/TestReadProjection.java: ## @@ -31,16 +34,14 @@ import org.apache.iceberg.types.Comparators; import org.apache.iceberg.ty

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568270818 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -281,36 +272,36 @@ public void testFullScan() { expected.addAll(file3FirstSnapshotRecords);

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568270613 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -281,36 +272,36 @@ public void testFullScan() { expected.addAll(file3FirstSnapshotRecords);

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568270029 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -266,12 +254,15 @@ public void testRandomData() throws IOException { append.commit();

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568268561 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -32,21 +32,14 @@ import java.io.IOException; import java.nio.ByteBuffer; import java.nio.ByteO

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568268169 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -60,15 +62,12 @@ import org.apache.iceberg.types.Types; import org.apache.iceberg.util.DateTime

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568268169 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -60,15 +62,12 @@ import org.apache.iceberg.types.Types; import org.apache.iceberg.util.DateTime

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568267658 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -189,10 +188,11 @@ public void testEqualityDeletePlanningStats() throws IOExcept

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568267490 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -163,10 +163,10 @@ public void testPositionDeletePlanningPathFilter() throws IOE

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568266969 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -137,9 +135,11 @@ public void testPositionDeletePlanningPath() throws IOExceptio

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568265618 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -94,24 +93,23 @@ public void createTableAndData() throws IOException {

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568264984 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -68,9 +67,9 @@ public class TestDataFileIndexStatsFilters { private DataFile d

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568264243 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -58,7 +57,7 @@ public class TestDataFileIndexStatsFilters { Types.Nest

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568263803 ## data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java: ## @@ -45,10 +38,12 @@ import org.apache.iceberg.util.Pair; import org.apache.iceberg.util.StructLi

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568262204 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -84,16 +86,15 @@ private static void assertEquals(Type type, Object expected, Object actual) {

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568261644 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -43,7 +42,9 @@ public static void assertEquals(Types.StructType struct, Record expected, Recor

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568263144 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -84,16 +86,15 @@ private static void assertEquals(Type type, Object expected, Object actual) {

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568260729 ## data/src/test/java/org/apache/iceberg/data/TestGenericRecord.java: ## @@ -35,7 +34,7 @@ public void testGetNullValue() { GenericRecord record = GenericRecord.cre

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568261957 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -56,7 +57,8 @@ public static void assertEquals(Types.ListType list, List expected, List a p

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568261117 ## data/src/test/java/org/apache/iceberg/data/DataTest.java: ## @@ -21,6 +21,7 @@ import static org.apache.iceberg.types.Types.NestedField.optional; import static org.

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1568260531 ## data/src/test/java/org/apache/iceberg/data/TestGenericReaderDeletes.java: ## @@ -18,28 +18,23 @@ */ package org.apache.iceberg.data; -import static org.assertj.c

Re: [I] rewriting manifest can rewrite based on filter? [iceberg]

2024-04-16 Thread via GitHub
chenwyi2 closed issue #10129: rewriting manifest can rewrite based on filter? URL: https://github.com/apache/iceberg/issues/10129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] rewriting manifest can rewrite based on filter? [iceberg]

2024-04-16 Thread via GitHub
chenwyi2 commented on issue #10129: URL: https://github.com/apache/iceberg/issues/10129#issuecomment-2060385444 https://github.com/apache/iceberg/pull/9447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
wForget commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1568234803 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java: ## @@ -275,18 +275,20 @@ public Table alterTable(Identifier ident, TableChange...

Re: [PR] Sql catalog [iceberg-rust]

2024-04-16 Thread via GitHub
JanKaul commented on PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#issuecomment-2060377330 I have to add a couple of more changes. I'll notify you when I'm finished. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
wForget commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1568227470 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkSessionCatalog.java: ## @@ -98,4 +103,20 @@ public void testLoadFunction() { // TODO: fix l

Re: [PR] Updates flink versioning information in our docs [iceberg]

2024-04-16 Thread via GitHub
pvary commented on PR #10155: URL: https://github.com/apache/iceberg/pull/10155#issuecomment-2060366264 Thanks for the PR @rodmeneses and @Fokko and @manuzhang for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Docs: Remove link to Flink unit test [iceberg]

2024-04-16 Thread via GitHub
pvary commented on PR #10160: URL: https://github.com/apache/iceberg/pull/10160#issuecomment-2060367028 @manuzhang: Could you please rebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Updates flink versioning information in our docs [iceberg]

2024-04-16 Thread via GitHub
pvary merged PR #10155: URL: https://github.com/apache/iceberg/pull/10155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
wForget commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1568220924 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkSessionCatalog.java: ## @@ -98,4 +103,20 @@ public void testLoadFunction() { // TODO: fix l

Re: [PR] Updates flink versioning information in our docs [iceberg]

2024-04-16 Thread via GitHub
manuzhang commented on code in PR #10155: URL: https://github.com/apache/iceberg/pull/10155#discussion_r1568215330 ## docs/docs/flink-writes.md: ## @@ -103,7 +103,7 @@ FlinkSink.forRowData(input) env.execute("Test Iceberg DataStream"); ``` -The iceberg API also allows users

Re: [I] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-04-16 Thread via GitHub
wForget commented on issue #10157: URL: https://github.com/apache/iceberg/issues/10157#issuecomment-2060344320 FYI #10128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Docs: Remove link to Flink unit test [iceberg]

2024-04-16 Thread via GitHub
manuzhang opened a new pull request, #10160: URL: https://github.com/apache/iceberg/pull/10160 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Add metadata tables [iceberg-python]

2024-04-16 Thread via GitHub
rahil-c commented on issue #511: URL: https://github.com/apache/iceberg-python/issues/511#issuecomment-2060301685 Hey @Fokko I can try working on `Manifests` table if no one is assigned it? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Test: Add test to partition on field with a dot [iceberg-python]

2024-04-16 Thread via GitHub
HonahX commented on code in PR #610: URL: https://github.com/apache/iceberg-python/pull/610#discussion_r1568179853 ## tests/integration/test_reads.py: ## @@ -473,6 +474,31 @@ def test_sanitize_character(catalog: Catalog) -> None: assert arrow_table.schema.names[0] == table

Re: [I] Iceberg Spark Extensions conflict with Paimon [iceberg]

2024-04-16 Thread via GitHub
wForget commented on issue #10143: URL: https://github.com/apache/iceberg/issues/10143#issuecomment-2060201859 > Never mind. I think what you mean it can be fixed either by Iceberg or by paimon? Yes, I'm not sure if my idea is acceptable, so I submitted issues on both sides hoping to

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
wForget commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1568109837 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkSessionCatalog.java: ## @@ -98,4 +103,20 @@ public void testLoadFunction() { // TODO: fix l

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1568086253 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [I] Flink cdc events with update or delete doesn't work in 0.11.0 branch [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2409: Flink cdc events with update or delete doesn't work in 0.11.0 branch URL: https://github.com/apache/iceberg/issues/2409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Flink cdc events with update or delete doesn't work in 0.11.0 branch [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2409: URL: https://github.com/apache/iceberg/issues/2409#issuecomment-2060100555 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Integrate Parquet bloomfilter feature [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2391: URL: https://github.com/apache/iceberg/issues/2391#issuecomment-2060100541 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Integrate Parquet bloomfilter feature [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2391: Integrate Parquet bloomfilter feature URL: https://github.com/apache/iceberg/issues/2391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Commit failed due to Internal error processing lock in metastore DB [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2387: URL: https://github.com/apache/iceberg/issues/2387#issuecomment-2060100523 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] CodePointCharStream not found while using the latest snapshot version [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2383: CodePointCharStream not found while using the latest snapshot version URL: https://github.com/apache/iceberg/issues/2383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] Commit failed due to Internal error processing lock in metastore DB [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2387: Commit failed due to Internal error processing lock in metastore DB URL: https://github.com/apache/iceberg/issues/2387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] CodePointCharStream not found while using the latest snapshot version [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2383: URL: https://github.com/apache/iceberg/issues/2383#issuecomment-2060100511 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Is Apache Iceberg table support spatial data? How to store spatial data in iceberg tables? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2378: URL: https://github.com/apache/iceberg/issues/2378#issuecomment-2060100495 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to use credential in Apache iceberg? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2377: How to use credential in Apache iceberg? URL: https://github.com/apache/iceberg/issues/2377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] How to use credential in Apache iceberg? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2377: URL: https://github.com/apache/iceberg/issues/2377#issuecomment-2060100470 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Is Apache Iceberg table support spatial data? How to store spatial data in iceberg tables? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2378: Is Apache Iceberg table support spatial data? How to store spatial data in iceberg tables? URL: https://github.com/apache/iceberg/issues/2378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] How to update number of buckets in partition spec ? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2375: URL: https://github.com/apache/iceberg/issues/2375#issuecomment-2060100456 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to update number of buckets in partition spec ? [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2375: How to update number of buckets in partition spec ? URL: https://github.com/apache/iceberg/issues/2375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] SparkSessionCatalog Drop Issues [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2374: SparkSessionCatalog Drop Issues URL: https://github.com/apache/iceberg/issues/2374 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] SparkSessionCatalog Drop Issues [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2374: URL: https://github.com/apache/iceberg/issues/2374#issuecomment-2060100437 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Rotation of encryption keys [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] commented on issue #2373: URL: https://github.com/apache/iceberg/issues/2373#issuecomment-2060100420 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Rotation of encryption keys [iceberg]

2024-04-16 Thread via GitHub
github-actions[bot] closed issue #2373: Rotation of encryption keys URL: https://github.com/apache/iceberg/issues/2373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
stevenzwu commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1568040503 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
stevenzwu commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1568039670 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1568016911 ## data/src/main/java/org/apache/iceberg/data/MigrationService.java: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1568016911 ## data/src/main/java/org/apache/iceberg/data/MigrationService.java: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1568016911 ## data/src/main/java/org/apache/iceberg/data/MigrationService.java: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1568005309 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/comet/CometIcebergColumnarBatchReader.java: ## @@ -0,0 +1,303 @@ +/* + * Licensed to the

Re: [PR] Add Refs metadata table [iceberg-python]

2024-04-16 Thread via GitHub
Fokko commented on PR #602: URL: https://github.com/apache/iceberg-python/pull/602#issuecomment-2059988471 @geruh Thanks for creating this PR! Can you resolve the conflicts so we can get this in? Thanks! -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add Refs metadata table [iceberg-python]

2024-04-16 Thread via GitHub
Fokko commented on code in PR #602: URL: https://github.com/apache/iceberg-python/pull/602#discussion_r1567985291 ## pyiceberg/table/__init__.py: ## @@ -3410,6 +3410,32 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: schema=entries_sche

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1567983615 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkSessionCatalog.java: ## @@ -98,4 +103,20 @@ public void testLoadFunction() { // TODO: f

[PR] Test: Add test to partition on field with a dot [iceberg-python]

2024-04-16 Thread via GitHub
Fokko opened a new pull request, #610: URL: https://github.com/apache/iceberg-python/pull/610 This is not allowed in Avro, just checking if it works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567967380 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567967116 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -1796,6 +1799,48 @@ public void testCatalogExpiredBearerTokenIsRefreshedWithCredential(String

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1567966520 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkSessionCatalog.java: ## @@ -98,4 +103,20 @@ public void testLoadFunction() { // TODO: f

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567730841 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List listTa

Re: [PR] Spark 3.5: Check table existence to determine which catalog for drop table [iceberg]

2024-04-16 Thread via GitHub
aokolnychyi commented on code in PR #10128: URL: https://github.com/apache/iceberg/pull/10128#discussion_r1567963497 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java: ## @@ -275,18 +275,20 @@ public Table alterTable(Identifier ident, TableChang

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

2024-04-16 Thread via GitHub
stargrey102 commented on PR #10148: URL: https://github.com/apache/iceberg/pull/10148#issuecomment-2059891565 Hi @deniskuzZ: for more info: we identified the autogather executed (controlled by hive.stats.autogather) to collect stats from Hive when committing a new Iceberg table to Hive (Hi

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-16 Thread via GitHub
igoradulian commented on PR #9793: URL: https://github.com/apache/iceberg/pull/9793#issuecomment-2059879886 @nastra please review the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567888070 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567888070 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567888070 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567892975 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567891507 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567888070 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-16 Thread via GitHub
javrasya commented on code in PR #9464: URL: https://github.com/apache/iceberg/pull/9464#discussion_r1567888070 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/split/SerializerHelper.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation

Re: [I] Metadata table queries fail if a partition column was reused in V2 [iceberg]

2024-04-16 Thread via GitHub
d4r3topk commented on issue #4661: URL: https://github.com/apache/iceberg/issues/4661#issuecomment-2059838101 This issue has not been resolved yet for V2 API. I have a table where one of the columns is "region" and "region" is also part of the partition spec. I run a query

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567855351 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,36 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567854386 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -105,9 +105,32 @@ public void testBuilderDoesNotCreateInvalidObjects() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567853170 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -83,9 +83,32 @@ public void testBuilderDoesNotCreateInvalidObjects() {

Re: [I] [feature request] Allow Java Iceberg library to write parquet files with special character column names [iceberg]

2024-04-16 Thread via GitHub
Fokko commented on issue #10120: URL: https://github.com/apache/iceberg/issues/10120#issuecomment-2059777982 I can confirm this, when creating a table: ```sql CREATE TABLE default.abc(`a.b+c` string); %%sql INSERT INTO default.abc VALUES ('a'), ('b') ``` The field name is s

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

2024-04-16 Thread via GitHub
Fokko commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2059779295 Related: https://github.com/apache/iceberg/issues/10120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

2024-04-16 Thread via GitHub
szehon-ho commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2059772658 I see, is it because of the dot character? I havent looked deeply at the problem but both solutions make sense, I think the first one may be preferable because then there is les

Re: [I] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-04-16 Thread via GitHub
chinnaraolalam commented on issue #10157: URL: https://github.com/apache/iceberg/issues/10157#issuecomment-2059697372 @RussellSpitzer @manuzhang ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-04-16 Thread via GitHub
chinnaraolalam opened a new pull request, #10158: URL: https://github.com/apache/iceberg/pull/10158 Drop table purge issue for parquet tables with SparkSessionCatalog. This was identified in Iceberg 1.4.3 + Spark 3.4.1 + SPARK-43203(On patch) in our environment https://github.com/

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-16 Thread via GitHub
sdd commented on PR #335: URL: https://github.com/apache/iceberg-rust/pull/335#issuecomment-2059682467 @liurenjie1024 @Fokko @marvinlanhenke @ZENOTME PTAL, I've refactored this on top of the `BoundPredicateVisitor` that was merged a few hours ago and it is ready for review -- This is an

[I] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-04-16 Thread via GitHub
chinnaraolalam opened a new issue, #10157: URL: https://github.com/apache/iceberg/issues/10157 ### Apache Iceberg version 1.4.3 ### Query engine Spark ### Please describe the bug 🐞 Drop table purge issue for parquet tables with **SparkSessionCatalog**. T

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567755548 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTh

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567748849 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2374,57 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertTh

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567745943 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -36,7 +36,7 @@ public class TestListTablesResponse extends RequestResponseTe

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-16 Thread via GitHub
sdd commented on PR #321: URL: https://github.com/apache/iceberg-rust/pull/321#issuecomment-2059636629 Closing this one in favour of the alternate PR based on the BoundPredicateVisitor design that got merged (https://github.com/apache/iceberg-rust/pull/335) -- This is an automated messag

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-16 Thread via GitHub
sdd closed pull request #321: add `InclusiveProjection` Visitor URL: https://github.com/apache/iceberg-rust/pull/321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567734951 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -34,7 +34,7 @@ public class TestListNamespacesResponse extends RequestRe

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-16 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1567730841 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List listTa

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-16 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1567725081 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

  1   2   >