Re: [PR] API, Core: Add schema_id to ContentFile [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on PR #4898: URL: https://github.com/apache/iceberg/pull/4898#issuecomment-227895 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Core: Recover the schema by reading previous metadata files [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on PR #4809: URL: https://github.com/apache/iceberg/pull/4809#issuecomment-2278899955 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] [CORE] Support file filtering based on schema [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on PR #4842: URL: https://github.com/apache/iceberg/pull/4842#issuecomment-2278899968 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] [WIP] AWS: Add partition info to Glue [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on PR #4775: URL: https://github.com/apache/iceberg/pull/4775#issuecomment-2278899946 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Support ORDERED BY in CTAS statement [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] closed issue #4889: Support ORDERED BY in CTAS statement URL: https://github.com/apache/iceberg/issues/4889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Support ORDERED BY in CTAS statement [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on issue #4889: URL: https://github.com/apache/iceberg/issues/4889#issuecomment-227887 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Infra: add AWS CI workflow configuration [iceberg]

2024-08-09 Thread via GitHub
github-actions[bot] commented on PR #4855: URL: https://github.com/apache/iceberg/pull/4855#issuecomment-2278899989 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] fix: don't silently drop errors encountered in table scan file planning [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on PR #535: URL: https://github.com/apache/iceberg-rust/pull/535#issuecomment-2278895706 Fixes the error suppression issue reported in https://github.com/apache/iceberg-rust/issues/495 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Scan does not work as expected [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2278895420 I have fixed the issue where errors were not returned to the user, in https://github.com/apache/iceberg-rust/pull/535 -- This is an automated message from the Apache Git Service. To

[PR] fix: don't silently drop errors encountered in table scan file planning [iceberg-rust]

2024-08-09 Thread via GitHub
sdd opened a new pull request, #535: URL: https://github.com/apache/iceberg-rust/pull/535 Ensure that errors get propagated back to the caller when encountered during generation of the file plan. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712265470 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -78,6 +78,11 @@ impl ManifestList { pub fn entries(&self) -> &[ManifestFile] { &self.entries } + +

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#issuecomment-2278812113 Thanks for the review @Xuanwo, much appreciated! Back to you, I've addressed your comments. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712263915 ## crates/iceberg/src/scan.rs: ## @@ -199,298 +256,588 @@ impl<'a> TableScanBuilder<'a> { field_ids.push(field_id); } -Ok(TableScan { +

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712259304 ## crates/iceberg/src/io/object_cache.rs: ## @@ -0,0 +1,161 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712257994 ## crates/iceberg/src/io/object_cache.rs: ## @@ -0,0 +1,161 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712256232 ## crates/iceberg/src/io/object_cache.rs: ## @@ -0,0 +1,161 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712255347 ## crates/iceberg/src/table.rs: ## @@ -16,28 +16,156 @@ // under the License. //! Table API for Apache Iceberg -use typed_builder::TypedBuilder; + +use std::sync::Ar

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1712254637 ## crates/iceberg/src/table.rs: ## @@ -16,28 +16,156 @@ // under the License. //! Table API for Apache Iceberg -use typed_builder::TypedBuilder; + +use std::sync::Ar

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712169542 ## mkdocs/docs/configuration.md: ## @@ -77,13 +77,14 @@ For the FileIO there are several configuration options available: | Key | Examp

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712144018 ## mkdocs/docs/configuration.md: ## @@ -77,13 +77,14 @@ For the FileIO there are several configuration options available: | Key | Examp

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712053032 ## mkdocs/docs/configuration.md: ## @@ -77,13 +77,14 @@ For the FileIO there are several configuration options available: | Key | Examp

[PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-09 Thread via GitHub
callum-ryan opened a new pull request, #534: URL: https://github.com/apache/iceberg-rust/pull/534 Carrying on from [PR #524 ](https://github.com/apache/iceberg-rust/pull/524), this is the implementation of the namespace functionality -- This is an automated message from the Apache Git Se

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712047933 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712047613 ## pyiceberg/io/__init__.py: ## @@ -58,6 +58,8 @@ S3_PROXY_URI = "s3.proxy-uri" S3_CONNECT_TIMEOUT = "s3.connect-timeout" S3_SIGNER_URI = "s3.signer.uri" +

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712044507 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1712042160 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2278578863 okay, this doesn't look like an issue with reading many metadata files. I wonder if the `limit` is respected for table scans. Things I want to compare * readin

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-09 Thread via GitHub
jkleinkauff commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2278564297 Hey, thank you for taking a time to answer me! 1. My files are in S3. 2. Sure! It's something I could do on my end? Do you have any recommendation on that? (I

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2278556792 There's a nontrivial cost in reading metadata files in Iceberg. Can you run this command, ``` table.inspect.manifests().to_pandas() ``` This will show the nu

[PR] feat: support timestamp columns in row filters [iceberg-rust]

2024-08-09 Thread via GitHub
sdd opened a new pull request, #533: URL: https://github.com/apache/iceberg-rust/pull/533 Fixes: https://github.com/apache/iceberg-rust/issues/532. Timezone needs to be explicitly set to `UTC` to match values written by Iceberg to underlying Parquet files. -- This is an automated m

[I] Filtering rows based on timestamp columns is not yet supported [iceberg-rust]

2024-08-09 Thread via GitHub
sdd opened a new issue, #532: URL: https://github.com/apache/iceberg-rust/issues/532 If attempting a table scan where one of the columns in the table is a `Timestamptz` or a `Timestamp`, you will encounter an error like this: ``` called `Result::unwrap()` on an `Err` value: Feature

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1711978272 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr

Re: [PR] Rename `gcs.endpoint` to `gcs.service.host` [iceberg-python]

2024-08-09 Thread via GitHub
ndrluis commented on code in PR #1007: URL: https://github.com/apache/iceberg-python/pull/1007#discussion_r1711967438 ## pyiceberg/io/pyarrow.py: ## @@ -388,7 +389,14 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste gcs_kwargs

Re: [PR] Rename `gcs.endpoint` to `gcs.service.host` [iceberg-python]

2024-08-09 Thread via GitHub
ndrluis commented on code in PR #1007: URL: https://github.com/apache/iceberg-python/pull/1007#discussion_r1711967825 ## pyiceberg/io/fsspec.py: ## @@ -76,6 +77,7 @@ OutputStream, ) from pyiceberg.typedef import Properties +from pyiceberg.utils.deprecated import deprecate

Re: [PR] Rename `gcs.endpoint` to `gcs.service.host` [iceberg-python]

2024-08-09 Thread via GitHub
ndrluis commented on code in PR #1007: URL: https://github.com/apache/iceberg-python/pull/1007#discussion_r1711966599 ## pyiceberg/io/fsspec.py: ## @@ -158,6 +160,14 @@ def _gs(properties: Properties) -> AbstractFileSystem: # https://gcsfs.readthedocs.io/en/latest/api.html#

Re: [PR] Rename `gcs.endpoint` to `gcs.service.host` [iceberg-python]

2024-08-09 Thread via GitHub
ndrluis commented on code in PR #1007: URL: https://github.com/apache/iceberg-python/pull/1007#discussion_r1711963854 ## pyiceberg/io/fsspec.py: ## @@ -158,6 +160,14 @@ def _gs(properties: Properties) -> AbstractFileSystem: # https://gcsfs.readthedocs.io/en/latest/api.html#

Re: [PR] Deprecate ADLFS prefix in favor of ADLS [iceberg-python]

2024-08-09 Thread via GitHub
ndrluis commented on code in PR #961: URL: https://github.com/apache/iceberg-python/pull/961#discussion_r1711960193 ## pyiceberg/io/fsspec.py: ## @@ -175,14 +184,50 @@ def _gs(properties: Properties) -> AbstractFileSystem: def _adlfs(properties: Properties) -> AbstractFileSyste

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
laksh-krishna-sharma commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2278491228 Thank you, @kevinjqliu I am grateful for the opportunity to contribute and for your patient guidance. I would like to work more under your guidance in this repository

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-09 Thread via GitHub
sungwy commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2278456915 Hi @jkleinkauff , that's indeed an interesting observation. I have some follow up questions to help us understand it better. 1. Where are your files stored? 2. Is t

Re: [PR] Bump version to 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy merged PR #1034: URL: https://github.com/apache/iceberg-python/pull/1034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]

2024-08-09 Thread via GitHub
pvary commented on issue #10892: URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2278393063 > I would say no. multiple jobs writing to the same table is a good example. Theoretically it is possible to revert only the changes done by the given job (for append only tabl

[PR] Bump aiohttp from 3.10.1 to 3.10.2 [iceberg-python]

2024-08-09 Thread via GitHub
dependabot[bot] opened a new pull request, #1035: URL: https://github.com/apache/iceberg-python/pull/1035 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.10.1 to 3.10.2. Release notes Sourced from https://github.com/aio-libs/aiohttp/releases";>aiohttp's releases.

Re: [I] DOCS: Improve Documentation on Write Support [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on issue #1008: URL: https://github.com/apache/iceberg-python/issues/1008#issuecomment-2278370759 someone asked about merge/upserts use cases in the slack channel as well, similar to `overwrite_filter` -- This is an automated message from the Apache Git Service. To r

[PR] Bump version to 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy opened a new pull request, #1034: URL: https://github.com/apache/iceberg-python/pull/1034 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] Table maintenace procedure(expire_snapshots) not work as expceted [iceberg]

2024-08-09 Thread via GitHub
RussellSpitzer commented on issue #10907: URL: https://github.com/apache/iceberg/issues/10907#issuecomment-2278344995 Expire snapshots only removes data files which are no longer needed by any remaining snapshots. The output of your command shows no files were needed to be removed. Based on

Re: [I] Support for Nessie Rest s3 signer [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on issue #1028: URL: https://github.com/apache/iceberg-python/issues/1028#issuecomment-2278328228 Create the PR #1029 and #1033 to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[PR] Access delegation [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro opened a new pull request, #1033: URL: https://github.com/apache/iceberg-python/pull/1033 This PR fix the hardcoded `X-Iceberg-Access-Delegation` header, the second point of https://github.com/apache/iceberg-python/issues/1028. It's based on the #1029 , and must be merged

Re: [PR] HA HMS support [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #752: URL: https://github.com/apache/iceberg-python/pull/752#discussion_r1711764892 ## tests/catalog/test_hive.py: ## @@ -1195,3 +1195,40 @@ def test_hive_wait_for_lock() -> None: with pytest.raises(WaitingForLockException): catal

[I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-09 Thread via GitHub
jkleinkauff opened a new issue, #1032: URL: https://github.com/apache/iceberg-python/issues/1032 ### Question Hey, thanks for this very convenient library. This is not a bug, just want to better understand something. I have a question regarding the performance - ie time t

Re: [I] Inconsistency in catalog.list_tables Behavior Across Python and Java: Returns Non-Iceberg Tables in Python Only [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on issue #314: URL: https://github.com/apache/iceberg-python/issues/314#issuecomment-2278310300 > Why do we choose to include non-iceberg tables in the result in python? I don't think we should. Using HMS for both hive and iceberg tables is pretty common, we shoul

Re: [I] Set table properties through Hive [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on issue #206: URL: https://github.com/apache/iceberg-python/issues/206#issuecomment-2278308121 This is already done using [`session_catalog_hive` ](https://github.com/apache/iceberg-python/blob/678facc09b99394acf4c1c12b5a8a67bf2a20f5d/tests/conftest.py#L2202-L2213) in

Re: [I] Set table properties through Hive [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu closed issue #206: Set table properties through Hive URL: https://github.com/apache/iceberg-python/issues/206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Implement Kerberos authentication support for Hive Catalog [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on PR #766: URL: https://github.com/apache/iceberg-python/pull/766#issuecomment-2278296348 Hi @yothinix do you mind rebasing with main? I think this is very close to being merged, I can work with you to get it through. -- This is an automated message from the Apa

Re: [PR] Deprecate ADLFS prefix in favor of ADLS [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on code in PR #961: URL: https://github.com/apache/iceberg-python/pull/961#discussion_r1711732413 ## pyiceberg/io/fsspec.py: ## @@ -175,14 +184,50 @@ def _gs(properties: Properties) -> AbstractFileSystem: def _adlfs(properties: Properties) -> AbstractFileSy

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2278268207 Thanks for the contribution, @laksh-krishna-sharma -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu merged PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] [documentation] library version upgrade fails `test_version_format` [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu closed issue #949: [documentation] library version upgrade fails `test_version_format` URL: https://github.com/apache/iceberg-python/issues/949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2278229829 I can run `make lint` for this branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
kevinjqliu commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2278229440 That's odd, it might be due to the specific Python version on Windows. I found a similar issue https://github.com/python-poetry/poetry/issues/3894#issuecomment-1130034154 --

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711679815 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,32 @@ public static String encodeNamespace(Namespace ns) { * @return a namespace

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711677062 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,32 @@ public static String encodeNamespace(Namespace ns) { * @return a namespace

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711532749 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encoded s

Re: [PR] Core, Flink: Fix build warnings [iceberg]

2024-08-09 Thread via GitHub
nastra merged PR #10899: URL: https://github.com/apache/iceberg/pull/10899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1711612526 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr

Re: [PR] Patch Release 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy commented on PR #1031: URL: https://github.com/apache/iceberg-python/pull/1031#issuecomment-2278054652 Done - I'll start the release process on https://github.com/apache/iceberg-python/tree/pyiceberg-0.7.x -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711532749 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encoded s

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711532266 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encoded s

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711530996 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,32 @@ public static String encodeNamespace(Namespace ns) { * @return a namespace

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711529328 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,32 @@ public static String encodeNamespace(Namespace ns) { * @return a namespace

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
nastra commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711528454 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encoded s

Re: [PR] Patch Release 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy commented on PR #1031: URL: https://github.com/apache/iceberg-python/pull/1031#issuecomment-2278010259 I'll get that done. Thank you for the input @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Patch Release 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy closed pull request #1031: Patch Release 0.7.1 URL: https://github.com/apache/iceberg-python/pull/1031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Patch Release 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
Fokko commented on PR #1031: URL: https://github.com/apache/iceberg-python/pull/1031#issuecomment-2278008651 @sungwy Looks good! I'm also okay with just cherry-picking these directly on the branch. All the commits are good because they have already been merged. I think a PR makes sense when

[PR] Patch Release 0.7.1 [iceberg-python]

2024-08-09 Thread via GitHub
sungwy opened a new pull request, #1031: URL: https://github.com/apache/iceberg-python/pull/1031 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] DOCS: Remove older row for `write.parquet.row-group-limit` [iceberg-python]

2024-08-09 Thread via GitHub
sungwy merged PR #1030: URL: https://github.com/apache/iceberg-python/pull/1030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
Fokko commented on code in PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#discussion_r1711510285 ## mkdocs/docs/configuration.md: ## @@ -84,6 +84,7 @@ For the FileIO there are several configuration options available: | s3.session-token | AQoDYXdzEJr...

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-09 Thread via GitHub
guitcastro commented on PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#issuecomment-2277988150 @Fokko @ndrluis I've applied the code suggestions. Can you please review again? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]

2024-08-09 Thread via GitHub
stevenzwu commented on issue #10892: URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277963876 > whether we think that the Iceberg table state is part of the Flink state, or not. I would say no. multiple jobs writing to the same table is a good example. > I ha

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1711376787 ## .github/workflows/ci.yml: ## @@ -113,7 +113,7 @@ jobs: run: cargo test --no-fail-fast --all-targets --all-features --workspace - name: Async-s

[PR] DOCS: Remove older row for `write.parquet.row-group-limit` [iceberg-python]

2024-08-09 Thread via GitHub
sungwy opened a new pull request, #1030: URL: https://github.com/apache/iceberg-python/pull/1030 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

2024-08-09 Thread via GitHub
sungwy commented on code in PR #1016: URL: https://github.com/apache/iceberg-python/pull/1016#discussion_r1711386006 ## mkdocs/docs/configuration.md: ## @@ -32,6 +32,7 @@ Iceberg tables support table properties to configure table behavior. | ---

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
Xuanwo commented on PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#issuecomment-2277849379 The opendal release is about to start, I expect to finish in next monday. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
Xuanwo commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1711383831 ## .github/workflows/ci.yml: ## @@ -113,7 +113,7 @@ jobs: run: cargo test --no-fail-fast --all-targets --all-features --workspace - name: Async-std

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1711376787 ## .github/workflows/ci.yml: ## @@ -113,7 +113,7 @@ jobs: run: cargo test --no-fail-fast --all-targets --all-features --workspace - name: Async-s

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1711376787 ## .github/workflows/ci.yml: ## @@ -113,7 +113,7 @@ jobs: run: cargo test --no-fail-fast --all-targets --all-features --workspace - name: Async-s

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-09 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1711376787 ## .github/workflows/ci.yml: ## @@ -113,7 +113,7 @@ jobs: run: cargo test --no-fail-fast --all-targets --all-features --workspace - name: Async-s

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
Xuanwo commented on code in PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#discussion_r1711342057 ## crates/iceberg/src/io/object_cache.rs: ## @@ -0,0 +1,161 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreemen

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-09 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711350427 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-09 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711350427 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-09 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1711348696 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

[I] Table maintenace procedure(expire_snapshots) not work as expceted [iceberg]

2024-08-09 Thread via GitHub
toien opened a new issue, #10907: URL: https://github.com/apache/iceberg/issues/10907 ### Query engine Spark SQL on AWS EMR(7.1.0) Versions: - Spark: 3.5.0 - Iceberg: 1.4.3 - Flink: 1.18 (Managed Apache Flink of AWS) ### Question First i create an iceberg

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
laksh-krishna-sharma commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2277796260 ![image](https://github.com/user-attachments/assets/be106ac7-9ced-4bb1-8a88-7ce59b23e1b5) While running `make install`, I encountered the error shown in the above s

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-09 Thread via GitHub
laksh-krishna-sharma commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-225803 Yes, I followed all the steps mentioned in the contribution guide and ran `make install` and `make lint`. I also resolved the issues that came up. Please review the up

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
sdd commented on PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#issuecomment-222962 Sure, will do so in a few hours time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-09 Thread via GitHub
snazy commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1711255861 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encoded st

Re: [PR] REST: AuthManager API [iceberg]

2024-08-09 Thread via GitHub
adutra commented on code in PR #10753: URL: https://github.com/apache/iceberg/pull/10753#discussion_r1711222487 ## aws/src/main/java/org/apache/iceberg/aws/RESTSigV4AuthManager.java: ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] REST: AuthManager API [iceberg]

2024-08-09 Thread via GitHub
adutra commented on code in PR #10753: URL: https://github.com/apache/iceberg/pull/10753#discussion_r1711196190 ## core/src/main/java/org/apache/iceberg/rest/auth/HttpRequestFacade.java: ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] REST: AuthManager API [iceberg]

2024-08-09 Thread via GitHub
adutra commented on code in PR #10753: URL: https://github.com/apache/iceberg/pull/10753#discussion_r1711190677 ## core/src/main/java/org/apache/iceberg/rest/auth/HttpRequestFacade.java: ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Object Cache: caches parsed Manifests and ManifestLists for performance [iceberg-rust]

2024-08-09 Thread via GitHub
liurenjie1024 commented on PR #512: URL: https://github.com/apache/iceberg-rust/pull/512#issuecomment-2277578710 cc @sdd Would you mind to update pr to resolve conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Remove deprecated APIs up to 1.7.0 [iceberg]

2024-08-09 Thread via GitHub
nastra merged PR #10818: URL: https://github.com/apache/iceberg/pull/10818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

  1   2   >