[PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.11 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9801: URL: https://github.com/apache/iceberg/pull/9801 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.9 to 9.5.11. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-m

[PR] Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.6 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9800: URL: https://github.com/apache/iceberg/pull/9800 Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.19.5 to 1.19.6. Release notes Sourced from https://github.com/testcontainers/t

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.39.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] commented on PR #9745: URL: https://github.com/apache/iceberg/pull/9745#issuecomment-1962808204 Superseded by #9799. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.39.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] closed pull request #9745: Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.39.0 URL: https://github.com/apache/iceberg/pull/9745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.41.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9799: URL: https://github.com/apache/iceberg/pull/9799 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 4.42.0 to 5.41.0. Release notes Sourced from https://github.com/palantir/gradle-b

Re: [PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.4.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] commented on PR #9750: URL: https://github.com/apache/iceberg/pull/9750#issuecomment-1962808181 Superseded by #9798. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.4.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] closed pull request #9750: Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.4.0 URL: https://github.com/apache/iceberg/pull/9750 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.1 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9798: URL: https://github.com/apache/iceberg/pull/9798 Bumps com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.1. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.5 to 3.15.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9797: URL: https://github.com/apache/iceberg/pull/9797 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.14.5 to 3.15.0. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/re

[PR] Build: Bump spring-boot from 2.5.4 to 3.2.3 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9796: URL: https://github.com/apache/iceberg/pull/9796 Bumps `spring-boot` from 2.5.4 to 3.2.3. Updates `org.springframework.boot:spring-boot-starter-jetty` from 2.5.4 to 3.2.3 Release notes Sourced from https://github.com/spring-proje

[PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.24.10 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9795: URL: https://github.com/apache/iceberg/pull/9795 Bumps software.amazon.awssdk:bom from 2.24.5 to 2.24.10. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwar

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.32.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] commented on PR #9747: URL: https://github.com/apache/iceberg/pull/9747#issuecomment-1962808119 Superseded by #9794. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.32.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] closed pull request #9747: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.32.0 URL: https://github.com/apache/iceberg/pull/9747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.33.0 [iceberg]

2024-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #9794: URL: https://github.com/apache/iceberg/pull/9794 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.33.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bom/

Re: [I] How to use VectorizedRowBatchIterator [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1486: URL: https://github.com/apache/iceberg/issues/1486#issuecomment-1962764811 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Reconsider handling of spaces in PartitionSpec$partitionToPath [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1479: URL: https://github.com/apache/iceberg/issues/1479#issuecomment-1962764801 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Make HiveCatalog inheritable for custom IMetaStoreClient [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1470: URL: https://github.com/apache/iceberg/issues/1470#issuecomment-1962764791 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] any plan for Iceberg Table on S3? [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1468: URL: https://github.com/apache/iceberg/issues/1468#issuecomment-1962764779 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] make Flink iceberg sink work without checkpoint enabled. [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1442: URL: https://github.com/apache/iceberg/issues/1442#issuecomment-1962764763 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Uppercased schemas are not readable in Iceberg-mr/ hive [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1445: URL: https://github.com/apache/iceberg/issues/1445#issuecomment-1962764772 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Why do we need two avro record readers & writers ? [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1152: URL: https://github.com/apache/iceberg/issues/1152#issuecomment-1962764717 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Why do we need two avro record readers & writers ? [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1152: Why do we need two avro record readers & writers ? URL: https://github.com/apache/iceberg/issues/1152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Add docker demo for iceberg starters [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1081: Add docker demo for iceberg starters URL: https://github.com/apache/iceberg/issues/1081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Add docker demo for iceberg starters [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1081: URL: https://github.com/apache/iceberg/issues/1081#issuecomment-1962764708 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Storing Lot of Sparse Columns [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1078: URL: https://github.com/apache/iceberg/issues/1078#issuecomment-1962764700 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Storing Lot of Sparse Columns [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1078: Storing Lot of Sparse Columns URL: https://github.com/apache/iceberg/issues/1078 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Support container reuse in SparkOrcValueReader [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1058: URL: https://github.com/apache/iceberg/issues/1058#issuecomment-1962764690 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Support container reuse in SparkOrcValueReader [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1058: Support container reuse in SparkOrcValueReader URL: https://github.com/apache/iceberg/issues/1058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Adding an attribute in ORC TypeDescription causes failures. [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1057: Adding an attribute in ORC TypeDescription causes failures. URL: https://github.com/apache/iceberg/issues/1057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Adding an attribute in ORC TypeDescription causes failures. [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1057: URL: https://github.com/apache/iceberg/issues/1057#issuecomment-1962764678 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [Question]How iceberg read data from metadata, has any paper introduce it? [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] commented on issue #1050: URL: https://github.com/apache/iceberg/issues/1050#issuecomment-1962764670 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [Question]How iceberg read data from metadata, has any paper introduce it? [iceberg]

2024-02-24 Thread via GitHub
github-actions[bot] closed issue #1050: [Question]How iceberg read data from metadata, has any paper introduce it? URL: https://github.com/apache/iceberg/issues/1050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[I] Inconsistency in deleting manifest and data files [iceberg]

2024-02-24 Thread via GitHub
namrathamyske opened a new issue, #9792: URL: https://github.com/apache/iceberg/issues/9792 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 When there are any exceptions of type `CleanableFailure`, we go ahead and [delete

[I] Glue Streaming to Iceberg not reflecting in changelog [iceberg]

2024-02-24 Thread via GitHub
94Sip opened a new issue, #9791: URL: https://github.com/apache/iceberg/issues/9791 ### Apache Iceberg version 1.4.3 (latest release) ### Query engine Other ### Please describe the bug 🐞 I realize this might be a bug with AWS Glue, but thought I would post h

Re: [PR] Allow non-string typed values in table properties [iceberg-python]

2024-02-24 Thread via GitHub
kevinjqliu commented on code in PR #469: URL: https://github.com/apache/iceberg-python/pull/469#discussion_r1501690489 ## tests/integration/test_writes.py: ## @@ -676,3 +676,37 @@ def test_write_and_evolve(session_catalog: Catalog, format_version: int) -> None with txn

[PR] Kevinjqliu/property value coerce to string [iceberg-python]

2024-02-24 Thread via GitHub
kevinjqliu opened a new pull request, #469: URL: https://github.com/apache/iceberg-python/pull/469 Addresses #376 We want to be able to accept `int` type in the `properties` field of Table and TableMetadata. For example, > create_table(..., properties={"write.parquet.compressio

[PR] Cleanup conftest, remove LocalOutputFile [iceberg-python]

2024-02-24 Thread via GitHub
kevinjqliu opened a new pull request, #468: URL: https://github.com/apache/iceberg-python/pull/468 While browsing `conftest.py`, I noticed `LocalOutputFile` which is not used anywhere. It's first introduced in [this commit](https://github.com/apache/iceberg/commit/98dc1e63d240c32ec7d48e

Re: [I] Parallel Table.append [iceberg-python]

2024-02-24 Thread via GitHub
kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1962460623 hm. Looks like something weird is going on if the resulting parquet file is 1.6 GB. Each parquet file size should be at most 512 MB, if not less. See the [bin packing logic]

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501620445 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +474,309 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501623001 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +474,309 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501622838 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501622467 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501621548 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501620980 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +47,38 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501620445 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +474,309 @@ public void testRewriteLargeManifestsParti

Re: [PR] Docs: Sync contributing page / refer to website for contributing [iceberg]

2024-02-24 Thread via GitHub
Fokko commented on code in PR #9776: URL: https://github.com/apache/iceberg/pull/9776#discussion_r1501571410 ## site/docs/contribute.md: ## @@ -145,6 +148,84 @@ Example: void sequenceNumber(long sequenceNumber); ``` +## Adding new functionality without breaking APIs +Ideal

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-24 Thread via GitHub
tomtongue commented on PR #9760: URL: https://github.com/apache/iceberg/pull/9760#issuecomment-1962385601 @nastra Thanks for your reviews! There are still some spark-extensions sub-classes that are necessary to be migrated (I believe they are 5 classes). I will submit the remaining classes.

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-24 Thread via GitHub
bitsondatadev commented on PR #9779: URL: https://github.com/apache/iceberg/pull/9779#issuecomment-1962358115 @manuzhang I'm on vacation, away from my computer and can't verify the link changes. I'll look at this when I return. -- This is an automated message from the Apache Git Service.

Re: [PR] Spark: Add support for Iceberg views [iceberg]

2024-02-24 Thread via GitHub
nastra closed pull request #9332: Spark: Add support for Iceberg views URL: https://github.com/apache/iceberg/pull/9332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Spark: Add support for Iceberg views [iceberg]

2024-02-24 Thread via GitHub
nastra commented on PR #9332: URL: https://github.com/apache/iceberg/pull/9332#issuecomment-1962328022 Closing this as all of the functionality has been merged individually -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501400728 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +474,309 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501400378 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1501399936 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +47,38 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Predi

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-24 Thread via GitHub
nastra merged PR #9760: URL: https://github.com/apache/iceberg/pull/9760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501378409 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -471,6 +471,39 @@ public void readFromViewReferencingGlobalTempVie

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501376695 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501376695 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [I] Parallel Table.append [iceberg-python]

2024-02-24 Thread via GitHub
bigluck commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1962308872 Ciao @kevinjqliu, thanks! I've tested it on the same `c5ad.16xlarge` machine, but the results are pretty similar, 27s vs 28s for this table: ``` $ pip install git+h

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501371107 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark 3.4,3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-24 Thread via GitHub
nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501368418 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,35 @@ public void showViews() throws NoSuchTableExce