Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435521777 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -118,45 +117,45 @@ public TestRewritePositionDeleteFi

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435521682 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -94,9 +96,7 @@ public class TestRewritePositionDelete

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435519924 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction.java: ## @@ -938,8 +946,8 @@ public void testRemoveOrphanFilesWithStatisti

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435519148 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction.java: ## @@ -139,36 +138,43 @@ public void testDryRun() throws IOException

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on PR #9367: URL: https://github.com/apache/iceberg/pull/9367#issuecomment-1868232570 @chinmay-bhat can you also please include `TestCreateActions` as that's in the same directory -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1434835739 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestMapRangePartitioner.java: ## @@ -0,0 +1,511 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

chinmay-bhat commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435511387 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestDeleteReachableFilesAction.java: ## @@ -345,11 +345,10 @@ public void testIgnoreMetadataFile

Re: [PR] Spark 3.5: Migrate tests to JUnit5 in actions directory [iceberg]

nastra commented on code in PR #9367: URL: https://github.com/apache/iceberg/pull/9367#discussion_r1435510145 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestDeleteReachableFilesAction.java: ## @@ -345,11 +345,10 @@ public void testIgnoreMetadataFilesNotFo

Re: [PR] Build: Bump pydantic from 2.5.2 to 2.5.3 [iceberg-python]

Fokko merged PR #237: URL: https://github.com/apache/iceberg-python/pull/237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mypy-boto3-glue from 1.34.0 to 1.34.7 [iceberg-python]

Fokko merged PR #238: URL: https://github.com/apache/iceberg-python/pull/238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat: Introduce basic file scan planning. [iceberg-rust]

ZENOTME commented on code in PR #129: URL: https://github.com/apache/iceberg-rust/pull/129#discussion_r1435478844 ## crates/iceberg/src/table.rs: ## @@ -42,8 +50,624 @@ impl Table { &self.metadata } +/// Returns current metadata ref. +pub fn metadata_ref(

Re: [I] field-id in avro schema is miss now [iceberg-rust]

ZENOTME commented on issue #131: URL: https://github.com/apache/iceberg-rust/issues/131#issuecomment-1868202853 Track in https://issues.apache.org/jira/browse/AVRO-3920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] feat: Add roadmap and features status in README.md [iceberg-rust]

ZENOTME commented on code in PR #134: URL: https://github.com/apache/iceberg-rust/pull/134#discussion_r1435471104 ## README.md: ## @@ -21,6 +21,37 @@ Native Rust implementation of [Apache Iceberg](https://iceberg.apache.org/). +## RoadMap + +### Catalog + +| Catalog Type |

Re: [I] When using the Flink upsert mode, the speed of reading data from the iceberg table is very slow. [iceberg]

coolderli commented on issue #9363: URL: https://github.com/apache/iceberg/issues/9363#issuecomment-1868194076 Did you try to execute a compaction -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Table Corruption iceberg [iceberg]

github-actions[bot] commented on issue #6989: URL: https://github.com/apache/iceberg/issues/6989#issuecomment-1868144249 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Avro: Add Avro-assisted name mapping [iceberg]

rdblue commented on PR #7392: URL: https://github.com/apache/iceberg/pull/7392#issuecomment-1868128440 Thanks, @wmoustafa! Good to have the ability to generate these mappings. Can you also update `ApplyNameMapping` to use them? It was added in https://github.com/apache/iceberg/pull/9347.

Re: [PR] Avro: Add Avro-assisted name mapping [iceberg]

rdblue merged PR #7392: URL: https://github.com/apache/iceberg/pull/7392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Data: Support reading default values from generic Avro readers [iceberg]

rdblue commented on code in PR #6004: URL: https://github.com/apache/iceberg/pull/6004#discussion_r1435386937 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -566,8 +569,10 @@ public Map read(Decoder decoder, Object reuse) throws IOException { public

Re: [PR] Data: Support reading default values from generic Avro readers [iceberg]

rdblue commented on PR #6004: URL: https://github.com/apache/iceberg/pull/6004#issuecomment-1868125419 @wmoustafa, I just opened a refactor that should make this a lot easier to get in: https://github.com/apache/iceberg/pull/9366 When I went to thoroughly review this, I had to do a lo

[PR] Build: Bump mypy-boto3-glue from 1.34.0 to 1.34.7 [iceberg-python]

dependabot[bot] opened a new pull request, #238: URL: https://github.com/apache/iceberg-python/pull/238 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.0 to 1.34.7. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commits";

[PR] Build: Bump pydantic from 2.5.2 to 2.5.3 [iceberg-python]

dependabot[bot] opened a new pull request, #237: URL: https://github.com/apache/iceberg-python/pull/237 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.5.2 to 2.5.3. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370739 ## core/src/test/java/org/apache/iceberg/avro/TestAvroNameMapping.java: ## @@ -80,10 +80,7 @@ public void testMapProjections() throws IOException { Record project

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370465 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -739,17 +956,18 @@ protected void set(R struct, int pos, Object value) { } } - static c

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370012 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -564,6 +678,115 @@ public Map read(Decoder decoder, Object reuse) throws IOException { }

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435369249 ## core/src/main/java/org/apache/iceberg/avro/ValueReader.java: ## @@ -23,4 +23,8 @@ public interface ValueReader { T read(Decoder decoder, Object reuse) throws IO

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435368996 ## core/src/main/java/org/apache/iceberg/avro/SupportsCustomRecords.java: ## @@ -0,0 +1,33 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one +

[PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue opened a new pull request, #9366: URL: https://github.com/apache/iceberg/pull/9366 This refactors the Avro generic reader so that it resolves schemas directly (like PyIceberg) rather than creating an Avro schema to trick Avro's `ResolvingDecoder` into projecting columns correctly.

Re: [PR] Core: Add ApplyNameMapping for Avro [iceberg]

rdblue merged PR #9347: URL: https://github.com/apache/iceberg/pull/9347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Add ApplyNameMapping for Avro [iceberg]

rdblue commented on PR #9347: URL: https://github.com/apache/iceberg/pull/9347#issuecomment-1868096519 Thanks for reviewing, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

Fokko commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1435350938 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFiles)

[PR] Core: Use CharSequenceMap for writing unordered deletes [iceberg]

aokolnychyi opened a new pull request, #9365: URL: https://github.com/apache/iceberg/pull/9365 This PR migrates `SortingPositionOnlyDeleteWriter` to use `CharSequenceMap` to simplify the code. The new solution is essentially the same in terms of functionality and performance. **Befor

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1435311105 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFiles

Re: [PR] Core: Add ApplyNameMapping for Avro [iceberg]

rdblue commented on code in PR #9347: URL: https://github.com/apache/iceberg/pull/9347#discussion_r1435310822 ## core/src/main/java/org/apache/iceberg/avro/ApplyNameMapping.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add ApplyNameMapping for Avro [iceberg]

rdblue commented on code in PR #9347: URL: https://github.com/apache/iceberg/pull/9347#discussion_r1435310612 ## core/src/main/java/org/apache/iceberg/avro/ApplyNameMapping.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867984456 Addressed Following review comments - Using static imports - Removing redundant newline - Using Assumptions from Assertj library -- This is an automated message from the

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435274336 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -121,166 +126,194 @@ public void testDropNonEmptyNamespace() {

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435274208 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -29,88 +29,93 @@ import org.apache.iceberg.catalog.TableIdentifier;

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435274057 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -29,88 +29,93 @@ import org.apache.iceberg.catalog.TableIdentifier;

Re: [I] Add View Support to Spark [iceberg]

jzhuge commented on issue #7938: URL: https://github.com/apache/iceberg/issues/7938#issuecomment-1867948485 Rebased https://github.com/apache/spark/pull/44197 with fixes for review comments and some unit tests -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] feat: Add website layout [iceberg-rust]

bitsondatadev commented on PR #130: URL: https://github.com/apache/iceberg-rust/pull/130#issuecomment-1867938398 Thank you all for being so considerate and willing to move outside of Rust. When I first replied yesterday I only saw not using our unified framework as a hindrence to update doc

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435220090 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435218641 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -810,4 +823,123 @@ private Catalog.TableBuilder newBuilder(Identifier ident, Schema

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435206411 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -810,4 +823,123 @@ private Catalog.TableBuilder newBuilder(Identifier ident, Schema

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435206166 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -810,4 +823,123 @@ private Catalog.TableBuilder newBuilder(Identifier ident, Schema

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435205674 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -810,4 +823,123 @@ private Catalog.TableBuilder newBuilder(Identifier ident, Schema

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435204168 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435202719 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435201928 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435200838 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435198050 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkCatalog.java: ## @@ -31,8 +31,10 @@ import org.apache.spark.sql.connector.catalog.SupportsNam

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435196601 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/logicalOperators.scala: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435195918 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/V2ViewDescription.scala: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435193321 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -121,166 +126,194 @@ public void testDropNonEmptyNamespace() { sql(

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435194286 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435193759 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435192720 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435192434 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -121,166 +126,194 @@ public void testDropNonEmptyNamespace() { sql(

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435192186 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -121,166 +126,194 @@ public void testDropNonEmptyNamespace() { sql(

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435191708 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -29,88 +29,93 @@ import org.apache.iceberg.catalog.TableIdentifier; im

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

rdblue commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1435191468 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435191020 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -29,88 +29,93 @@ import org.apache.iceberg.catalog.TableIdentifier; im

Re: [PR] Spark 3:5 Migrate tests to JUnit5 in source directory [iceberg]

nastra merged PR #9342: URL: https://github.com/apache/iceberg/pull/9342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435190729 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogDatabase.java: ## @@ -29,88 +29,93 @@ import org.apache.iceberg.catalog.TableIdentifier; im

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867821695 Thanks for the review, @nastra I have added following changes - Rename FlinkCatalogTestBaseJU5 to CatalogTestBase - Added Junit5 testcase and AssertJ style for TestFlinkCatal

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

Fokko commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1435150752 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFiles)

Re: [PR] feat: Add roadmap and features status in README.md [iceberg-rust]

Fokko commented on code in PR #134: URL: https://github.com/apache/iceberg-rust/pull/134#discussion_r1435135883 ## README.md: ## @@ -21,6 +21,37 @@ Native Rust implementation of [Apache Iceberg](https://iceberg.apache.org/). +## RoadMap + +### Catalog + +| Catalog Type | St

Re: [PR] feat: Add roadmap and features status in README.md [iceberg-rust]

Xuanwo commented on code in PR #134: URL: https://github.com/apache/iceberg-rust/pull/134#discussion_r1435111602 ## README.md: ## @@ -21,6 +21,37 @@ Native Rust implementation of [Apache Iceberg](https://iceberg.apache.org/). +## RoadMap + +### Catalog + +| Catalog Type | S

Re: [PR] Build: Bump fastavro from 1.9.1 to 1.9.2 [iceberg-python]

Fokko merged PR #236: URL: https://github.com/apache/iceberg-python/pull/236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867717673 > I tested this implementation with couple of subclasses and it seems to be working fine. The difference that I observed here as compared to junit4 in terms of test execution. First test i

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867701145 > can you also please convert `TestMetadataTableReadableMetrics` to JUnit5 and use `CatalogTestBase` as its new base class? ack -- This is an automated message from the Apache

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435062930 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/FlinkCatalogTestBaseJU5.java: ## @@ -0,0 +1,143 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867698318 I tested this implementation with couple of subclasses and it seems to be working fine. The difference that I observed here as compared to junit4 in terms of test execution. First test

Re: [PR] Write support [iceberg-python]

HonahX commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1435031878 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2004,158 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Fix spark AddFilesProcedure log tip [iceberg]

nastra merged PR #9357: URL: https://github.com/apache/iceberg/pull/9357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on PR #9364: URL: https://github.com/apache/iceberg/pull/9364#issuecomment-1867693190 can you also please convert `TestMetadataTableReadableMetrics` to JUnit5 and use `CatalogTestBase` as its new base class? -- This is an automated message from the Apache Git Service. To

Re: [PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

nastra commented on code in PR #9364: URL: https://github.com/apache/iceberg/pull/9364#discussion_r1435056808 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/FlinkCatalogTestBaseJU5.java: ## @@ -0,0 +1,143 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[PR] Create FlinkCatalogTestBaseJU5 for migration to JUnit5 [iceberg]

vinitpatni opened a new pull request, #9364: URL: https://github.com/apache/iceberg/pull/9364 Base class creation of FlinkCatalogTestBase for the migration to JUnit5 in regards to https://github.com/apache/iceberg/issues/9079 -- This is an automated message from the Apache Git Service. To

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

nastra commented on PR #9302: URL: https://github.com/apache/iceberg/pull/9302#issuecomment-1867599839 @pvary sorry I was not able to reproduce this. I checked out the commit prior to this PR, ran `./gradlew clean build -x test -x integrationTest`, then cherry-picked this commit and ran `./

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on PR #9185: URL: https://github.com/apache/iceberg/pull/9185#issuecomment-1867581096 @cgpoh I took the liberty to address the remaining items so that we can get this in. @pvary could you do a final review on this one please? I thin we would also want to apply this to Fli

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434967495 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedSql.java: ## @@ -26,26 +26,27 @@ import org.apache.flink.table.api.TableEnvir

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434965909 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkScan.java: ## @@ -49,41 +53,36 @@ import org.apache.iceberg.types.Types; import org.apache.

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434965644 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkScan.java: ## @@ -49,41 +53,36 @@ import org.apache.iceberg.types.Types; import org.apache.

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434965278 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkScan.java: ## @@ -49,41 +53,36 @@ import org.apache.iceberg.types.Types; import org.apache.

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434965009 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormat.java: ## @@ -33,21 +33,20 @@ import org.apache.iceberg.Schema; import org.apach

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434962899 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormat.java: ## @@ -140,9 +140,9 @@ public void testBasicProjection() throws IOException

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434958256 ## data/src/test/java/org/apache/iceberg/data/GenAppenderHelper.java: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434950138 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -337,82 +334,72 @@ private static void assertAvroEquals( if (expected == null &&

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1434948333 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -193,109 +192,106 @@ private static void assertEquals( return; } -A

Re: [PR] Spark 3:5 Migrate tests to JUnit5 in source directory [iceberg]

chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1434929879 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java: ## @@ -271,28 +271,30 @@ public void testCustomMetricCollectionForNe

Re: [I] How to improve write speed for data in the same partition? [iceberg]

atifiu commented on issue #9330: URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1867510715 I am also facing the same issue when writing data to single partition of iceberg table using dataframe when I write using spark.sql with insert into select * then performance is prett

Re: [PR] feat: Add roadmap and features status in README.md [iceberg-rust]

liurenjie1024 commented on PR #134: URL: https://github.com/apache/iceberg-rust/pull/134#issuecomment-1867507025 cc @Xuanwo @ZENOTME @Fokko PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[PR] feat: Add roadmap and features status in README.md [iceberg-rust]

liurenjie1024 opened a new pull request, #134: URL: https://github.com/apache/iceberg-rust/pull/134 Close #133 . Adding roadmap and feature status in main page. This part is heavily inspired by [apache/iceberg-go](https://github.com/apache/iceberg-go). -- This is an automate

Re: [PR] Spark 3:5 Migrate tests to JUnit5 in source directory [iceberg]

nastra commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1434921722 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java: ## @@ -271,28 +271,30 @@ public void testCustomMetricCollectionForNestedPa

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1434909916 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Spark 3:5 Migrate tests to JUnit5 in source directory [iceberg]

chinmay-bhat commented on PR #9342: URL: https://github.com/apache/iceberg/pull/9342#issuecomment-1867476496 done! Also, I added the parameterized annotations to `CatalogTestBase`, since `SparkCatalogTestBase` was parameterized but `CatalogTestBase` was surprisingly not. -- This i

Re: [PR] Spark 3:5 Migrate tests to JUnit5 in source directory [iceberg]

nastra commented on PR #9342: URL: https://github.com/apache/iceberg/pull/9342#issuecomment-1867460904 yes you need a rebase + force push -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: Expression system. [iceberg-rust]

Fokko commented on code in PR #132: URL: https://github.com/apache/iceberg-rust/pull/132#discussion_r1434871605 ## crates/iceberg/src/expr/mod.rs: ## @@ -0,0 +1,49 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See

Re: [I] Deleting a column from an iceberg table breaks schema in AWS Glue catalog [iceberg]

qoqajr commented on issue #6340: URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1867417317 I did the same but having to write custom code for both deleting a column with spark.sql and then in the glue catalog (I did it with awswrangler lib) is a bit messy to me. -- This

Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]

BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867408622 Does that cause any problems? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]

Zhangg7723 commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867393450 step 3:At the time of the OOM, the dwd.b_std_category table was executing this command. -- This is an automated message from the Apache Git Service. To respond to the message, p

  1   2   >