Re: [PR] Add doc for rewriting manifest with spec id [iceberg]

2023-12-09 Thread via GitHub
aokolnychyi commented on code in PR #9253: URL: https://github.com/apache/iceberg/pull/9253#discussion_r1421696977 ## docs/spark-procedures.md: ## @@ -446,6 +446,7 @@ This procedure invalidates all cached Spark plans that reference the affected ta |---|---|

[PR] Data: Add GenericFileWriterFactory [iceberg]

2023-12-09 Thread via GitHub
aokolnychyi opened a new pull request, #9267: URL: https://github.com/apache/iceberg/pull/9267 This PR adds `GenericFileWriterFactory`, similar to `FlinkFileWriterFactory` and `SparkFileWriterFactory`. This is a new API that should be used in favor of methods in `FileAppenderFactory` for cr

[PR] Build: Bump actions/labeler from 4 to 5 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9264: URL: https://github.com/apache/iceberg/pull/9264 Bumps [actions/labeler](https://github.com/actions/labeler) from 4 to 5. Release notes Sourced from https://github.com/actions/labeler/releases";>actions/labeler's releases. v5

[PR] Build: Bump actions/setup-python from 4 to 5 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9266: URL: https://github.com/apache/iceberg/pull/9266 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. Release notes Sourced from https://github.com/actions/setup-python/releases";>actions/setup-python's

[PR] Build: Bump actions/stale from 8.0.0 to 9.0.0 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9265: URL: https://github.com/apache/iceberg/pull/9265 Bumps [actions/stale](https://github.com/actions/stale) from 8.0.0 to 9.0.0. Release notes Sourced from https://github.com/actions/stale/releases";>actions/stale's releases. v9

Re: [PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.17.2 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] closed pull request #8986: Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.17.2 URL: https://github.com/apache/iceberg/pull/8986 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.17.2 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] commented on PR #8986: URL: https://github.com/apache/iceberg/pull/8986#issuecomment-1848855650 Superseded by #9263. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.23.2 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] closed pull request #9201: Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.23.2 URL: https://github.com/apache/iceberg/pull/9201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.23.2 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] commented on PR #9201: URL: https://github.com/apache/iceberg/pull/9201#issuecomment-1848855638 Superseded by #9262. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.17.3 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9263: URL: https://github.com/apache/iceberg/pull/9263 Bumps com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.17.3. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.a

[PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.23.3 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9262: URL: https://github.com/apache/iceberg/pull/9262 Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 6.13.0 to 6.23.3. Commits https://github.com/diffplug/spotless/commit/29bb75cdb026

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.18 to 1.2.19 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9261: URL: https://github.com/apache/iceberg/pull/9261 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.18 to 1.2.19. Commits https://github.com/Azure/azure-sdk-for-java/commit/473a7627260dae487e7f

[PR] Build: Bump com.google.cloud:libraries-bom from 26.27.0 to 26.28.0 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9258: URL: https://github.com/apache/iceberg/pull/9258 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.27.0 to 26.28.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bom/

[PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.2.3 to 5.3 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9260: URL: https://github.com/apache/iceberg/pull/9260 Bumps [org.apache.httpcomponents.client5:httpclient5](https://github.com/apache/httpcomponents-client) from 5.2.3 to 5.3. Changelog Sourced from https://github.com/apache/httpcompone

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.21.29 to 2.21.38 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] commented on PR #9214: URL: https://github.com/apache/iceberg/pull/9214#issuecomment-1848855492 Superseded by #9259. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.21.29 to 2.21.38 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] closed pull request #9214: Build: Bump software.amazon.awssdk:bom from 2.21.29 to 2.21.38 URL: https://github.com/apache/iceberg/pull/9214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[PR] Build: Bump software.amazon.awssdk:bom from 2.21.29 to 2.21.42 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9259: URL: https://github.com/apache/iceberg/pull/9259 Bumps software.amazon.awssdk:bom from 2.21.29 to 2.21.42. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwa

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.3 to 3.14.4 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9257: URL: https://github.com/apache/iceberg/pull/9257 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.14.3 to 3.14.4. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/re

Re: [PR] Build: Bump mkdocs-material from 9.4.12 to 9.4.14 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] closed pull request #9198: Build: Bump mkdocs-material from 9.4.12 to 9.4.14 URL: https://github.com/apache/iceberg/pull/9198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Build: Bump mkdocs-material from 9.4.12 to 9.4.14 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] commented on PR #9198: URL: https://github.com/apache/iceberg/pull/9198#issuecomment-1848853419 Superseded by #9256. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump mkdocs-material from 9.4.12 to 9.5.1 [iceberg]

2023-12-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9256: URL: https://github.com/apache/iceberg/pull/9256 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.4.12 to 9.5.1. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-m

Re: [I] Uploading Data to Iceberg [iceberg-python]

2023-12-09 Thread via GitHub
dbuades commented on issue #181: URL: https://github.com/apache/iceberg-python/issues/181#issuecomment-1848827430 > @dbuades We're aiming for this year :) That's great to hear, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Support parameterized tests at class-level with JUnit5 [iceberg]

2023-12-09 Thread via GitHub
GianlucaPrincipini commented on PR #9161: URL: https://github.com/apache/iceberg/pull/9161#issuecomment-1848794660 Hi @nastra, I migrated `TestIcebergInputFormats` in MR module, which is a slightly more complex test class than `TestDictionaryRowGroupFilter`. I found some other class that co

Re: [I] Docs: Add Spec back to main menu [iceberg]

2023-12-09 Thread via GitHub
github-actions[bot] commented on issue #7794: URL: https://github.com/apache/iceberg/issues/7794#issuecomment-1848790182 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Multiple Catalog Connection [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on issue #156: URL: https://github.com/apache/iceberg-python/issues/156#issuecomment-1848673647 Thanks @jayceslesar for updating the docs here! 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Multiple Catalog Connection [iceberg-python]

2023-12-09 Thread via GitHub
Fokko closed issue #156: Multiple Catalog Connection URL: https://github.com/apache/iceberg-python/issues/156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-09 Thread via GitHub
Fokko merged PR #194: URL: https://github.com/apache/iceberg-python/pull/194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on PR #194: URL: https://github.com/apache/iceberg-python/pull/194#issuecomment-1848672326 > my org has a manual job that can be run as part of a PR/MR that can run the lint and commit the changes? Is it worth looking at adding something similar? Ha, I'm aware of thos

Re: [PR] Arrow: Allow missing field-ids from Schema [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on PR #183: URL: https://github.com/apache/iceberg-python/pull/183#issuecomment-1848669009 Thanks for the Java context here! Appreciate it! > The table_schema must be processed with assign_field_id (or its equivalent function in Java) before being written to the table.

Re: [PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-09 Thread via GitHub
jayceslesar commented on PR #194: URL: https://github.com/apache/iceberg-python/pull/194#issuecomment-1848667046 @Fokko my org has a manual job that can be run as part of a PR/MR that can run the lint and commit the changes? Is it worth looking at adding something similar? Or stick with the

Re: [PR] Arrow: Allow missing field-ids from Schema [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on code in PR #183: URL: https://github.com/apache/iceberg-python/pull/183#discussion_r1421564859 ## pyiceberg/io/pyarrow.py: ## @@ -713,28 +721,49 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: """Visit a primitive type.""" -def _ge

Re: [I] Uploading Data to Iceberg [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on issue #181: URL: https://github.com/apache/iceberg-python/issues/181#issuecomment-1848666228 @dbuades We're aiming for this year :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Create namespace through the CLI [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on issue #185: URL: https://github.com/apache/iceberg-python/issues/185#issuecomment-1848663919 Done! Looking forward to it! 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Replace black by Ruff Formatter [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on PR #127: URL: https://github.com/apache/iceberg-python/pull/127#issuecomment-1848662110 @hussein-awala the PR has been merged :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-09 Thread via GitHub
Fokko commented on PR #194: URL: https://github.com/apache/iceberg-python/pull/194#issuecomment-1848655988 @jayceslesar The Markdown linter has an improvement, can you run `make lint`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Build: Bump actions/stale from 8.0.0 to 9.0.0 [iceberg-python]

2023-12-09 Thread via GitHub
Fokko merged PR #195: URL: https://github.com/apache/iceberg-python/pull/195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mkdocs-material from 9.4.14 to 9.5.1 [iceberg-python]

2023-12-09 Thread via GitHub
Fokko merged PR #197: URL: https://github.com/apache/iceberg-python/pull/197 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Flink: Backport PR #9212 - switching to SortKey for data statistics [iceberg]

2023-12-09 Thread via GitHub
stevenzwu commented on PR #9254: URL: https://github.com/apache/iceberg/pull/9254#issuecomment-1848619220 thanks @pvary and @yegangy0718 for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Flink: Backport PR #9212 - switching to SortKey for data statistics [iceberg]

2023-12-09 Thread via GitHub
stevenzwu merged PR #9254: URL: https://github.com/apache/iceberg/pull/9254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-09 Thread via GitHub
advancedxy commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-1848427549 > Spark 3.4 (same as my app) with the condition on the partitions will actually prune the unaffected partitions, while 3.5 will not. I did some quick debug. The reason why spark

Re: [PR] feat(docs): example of multiple catalogs defined in .pyiceberg.yaml [iceberg-python]

2023-12-09 Thread via GitHub
jayceslesar commented on code in PR #194: URL: https://github.com/apache/iceberg-python/pull/194#discussion_r1421424492 ## mkdocs/docs/api.md: ## @@ -33,6 +33,20 @@ catalog: credential: t-1234:secret ``` +Note that multiple catalogs can be defined in the same `.pyiceberg

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2023-12-09 Thread via GitHub
fqaiser94 commented on code in PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#discussion_r1421245396 ## crates/iceberg/src/spec/sort.rs: ## @@ -88,15 +91,106 @@ impl SortOrder { pub fn is_unsorted(&self) -> bool { self.fields.is_empty() } + +

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2023-12-09 Thread via GitHub
fqaiser94 commented on code in PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#discussion_r1421245396 ## crates/iceberg/src/spec/sort.rs: ## @@ -88,15 +91,106 @@ impl SortOrder { pub fn is_unsorted(&self) -> bool { self.fields.is_empty() } + +

Re: [PR] Core: Optimize manifest evaluation for super wide tables [iceberg]

2023-12-09 Thread via GitHub
irshadcc commented on PR #9147: URL: https://github.com/apache/iceberg/pull/9147#issuecomment-1848403130 @Fokko Can we merge this PR ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

2023-12-09 Thread via GitHub
nastra commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1421376428 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -193,109 +192,106 @@ private static void assertEquals( return; } -A

Re: [PR] Switch to junit5 for mr [iceberg]

2023-12-09 Thread via GitHub
nastra commented on PR #9241: URL: https://github.com/apache/iceberg/pull/9241#issuecomment-1848342540 @lschetanrao this just needs one minor update around the assumption. Can you also please open an issue to address parameterized tests in this module? -- This is an automated message from

Re: [PR] Switch to junit5 for mr [iceberg]

2023-12-09 Thread via GitHub
nastra commented on code in PR #9241: URL: https://github.com/apache/iceberg/pull/9241#discussion_r1421374570 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestDeserializer.java: ## @@ -155,13 +155,13 @@ public void testListDeserialize() { Object[] data = new Object[] {new

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

2023-12-09 Thread via GitHub
pvary commented on PR #9173: URL: https://github.com/apache/iceberg/pull/9173#issuecomment-1848319352 @mas-chen: Please do not forget to port the changes to the other Flink versions. Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

2023-12-09 Thread via GitHub
pvary commented on PR #9173: URL: https://github.com/apache/iceberg/pull/9173#issuecomment-1848319141 Merged to main. Thanks for the PR @mas-chen and @stevenzwu for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

2023-12-09 Thread via GitHub
pvary merged PR #9173: URL: https://github.com/apache/iceberg/pull/9173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach