Re: [PR] Build: Use official revapi Gradle plugin [iceberg]

2024-07-04 Thread via GitHub
nastra merged PR #10631: URL: https://github.com/apache/iceberg/pull/10631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Bump pydantic from 2.8.0 to 2.8.2 [iceberg-python]

2024-07-04 Thread via GitHub
Fokko merged PR #894: URL: https://github.com/apache/iceberg-python/pull/894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-07-04 Thread via GitHub
Fokko merged PR #10276: URL: https://github.com/apache/iceberg/pull/10276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-07-04 Thread via GitHub
Fokko commented on PR #10276: URL: https://github.com/apache/iceberg/pull/10276#issuecomment-2210287505 Let's get this in, thanks @lawofcycles for working on this, and @amogh-jahagirdar and @rahil-c for the review! -- This is an automated message from the Apache Git Service. To respond to

Re: [I] AWS: Updating Glue catalog table removes column descriptions [iceberg]

2024-07-04 Thread via GitHub
Fokko closed issue #10220: AWS: Updating Glue catalog table removes column descriptions URL: https://github.com/apache/iceberg/issues/10220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[I] Unable to get field from serde: org.apache.iceberg.mr.hive.HiveIcebergSerDe [iceberg]

2024-07-04 Thread via GitHub
Sourabh-Dilraj opened a new issue, #10633: URL: https://github.com/apache/iceberg/issues/10633 ### Query engine HIve ### Question I am using HIve branch-4.0.0-alpha-1 , I have placed iceberg-hive-runtime-1.6.0-SNAPSHOT.jar jar in hive/lib/ dir. Also made all the requi

Re: [PR] Migrate source package in Flink [iceberg]

2024-07-04 Thread via GitHub
tomtongue commented on code in PR #10632: URL: https://github.com/apache/iceberg/pull/10632#discussion_r1666304183 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestArrayPoolDataIteratorBatcherRowData.java: ## @@ -107,7 +114,11 @@ public void testSing

Re: [PR] Migrate source package in Flink [iceberg]

2024-07-04 Thread via GitHub
tomtongue commented on PR #10632: URL: https://github.com/apache/iceberg/pull/10632#issuecomment-2210176248 @nastra when you have a chance, could you review this PR? Currently this PR only includes Flink 1.10 because of the a lot of file changes. -- This is an automated message from the A

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666297390 ## Cargo.toml: ## @@ -82,6 +82,7 @@ serde_repr = "0.1.16" serde_with = "3.4.0" tempfile = "3.8" tokio = { version = "1", features = ["macros"] } +tokio-shared-rt

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-04 Thread via GitHub
fengjiajie commented on PR #10565: URL: https://github.com/apache/iceberg/pull/10565#issuecomment-2210172557 > +1 from me. Let's wait a bit, to see if someone else is interested in reviewing. If there're no more comments, I will merge next Tuesday. (Ping me if I forgot πŸ˜„) Thanks --

Re: [PR] Migrate source package in Flink [iceberg]

2024-07-04 Thread via GitHub
tomtongue commented on PR #10632: URL: https://github.com/apache/iceberg/pull/10632#issuecomment-2210148539 This PR currently only has Flink 1.19 because the number of class changes are a bit many. Will create the backport PR for other Flink versions 1.17 and 1.18 -- This is an automated

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
Xuanwo commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666227308 ## Cargo.toml: ## @@ -82,6 +82,7 @@ serde_repr = "0.1.16" serde_with = "3.4.0" tempfile = "3.8" tokio = { version = "1", features = ["macros"] } +tokio-shared-rt =

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666223166 ## Cargo.toml: ## @@ -82,6 +82,7 @@ serde_repr = "0.1.16" serde_with = "3.4.0" tempfile = "3.8" tokio = { version = "1", features = ["macros"] } +tokio-shared-rt

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-07-04 Thread via GitHub
rahil-c commented on code in PR #10276: URL: https://github.com/apache/iceberg/pull/10276#discussion_r1666192983 ## aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogTable.java: ## @@ -221,6 +221,69 @@ public void testUpdateTable() { assertThat(response.ta

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
Xuanwo commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666177869 ## Cargo.toml: ## @@ -82,6 +82,7 @@ serde_repr = "0.1.16" serde_with = "3.4.0" tempfile = "3.8" tokio = { version = "1", features = ["macros"] } +tokio-shared-rt =

Re: [PR] feat: runtime module [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 merged PR #233: URL: https://github.com/apache/iceberg-rust/pull/233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] fix: Fix namespace identifier in url [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #435: URL: https://github.com/apache/iceberg-rust/pull/435#discussion_r1666126704 ## crates/iceberg/src/catalog/mod.rs: ## @@ -123,9 +122,9 @@ impl NamespaceIdent { Self::from_vec(iter.into_iter().map(|s| s.to_string()).collect()) }

Re: [PR] Support create multiple element ns together for nessie [iceberg]

2024-07-04 Thread via GitHub
zymap commented on code in PR #10630: URL: https://github.com/apache/iceberg/pull/10630#discussion_r1666167004 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -290,6 +291,14 @@ public void renameTable(TableIdentifier from, TableIdentifier to) {

Re: [I] Tracking issues of aligning storage support with iceberg-java [iceberg-rust]

2024-07-04 Thread via GitHub
Xuanwo commented on issue #408: URL: https://github.com/apache/iceberg-rust/issues/408#issuecomment-2209879237 > @Xuanwo I can take the Azure datalake FileIO Implementation + the corresponding infrastructure set up, sound ok? Welcome, have fun! -- This is an automated message from

Re: [PR] fix: Fix namespace identifier in url [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 commented on code in PR #435: URL: https://github.com/apache/iceberg-rust/pull/435#discussion_r1666155211 ## crates/iceberg/src/catalog/mod.rs: ## @@ -123,9 +122,9 @@ impl NamespaceIdent { Self::from_vec(iter.into_iter().map(|s| s.to_string()).collect())

[PR] Build: Use official revapi Gradle plugin [iceberg]

2024-07-04 Thread via GitHub
ajantha-bhat opened a new pull request, #10631: URL: https://github.com/apache/iceberg/pull/10631 https://github.com/revapi/gradle-revapi/releases/tag/1.8.0 Special thanks to @nastra, @jbonofre -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] fix: Fix namespace identifier in url [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on PR #435: URL: https://github.com/apache/iceberg-rust/pull/435#issuecomment-2209672375 > cc @thexiay Would you take a look if this helps? yes, it really helps! -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666119516 ## crates/catalog/glue/tests/glue_catalog_test.rs: ## @@ -92,6 +96,24 @@ async fn set_test_fixture(func: &str) -> TestFixture { } } +async fn lazy_reuse_dc()

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666119013 ## crates/catalog/glue/tests/glue_catalog_test.rs: ## @@ -92,6 +96,24 @@ async fn set_test_fixture(func: &str) -> TestFixture { } } +async fn lazy_reuse_dc()

Re: [PR] Implement Sorted Writes [iceberg-python]

2024-07-04 Thread via GitHub
vinjai commented on PR #871: URL: https://github.com/apache/iceberg-python/pull/871#issuecomment-2209659177 This PR solves for: 1. Writing sorted datasets to a partitioned or non-partitioned iceberg table. 2. Generating manifests with correct sort-order-id. 3. Integration tests to ma

Re: [PR] Implement Sorted Writes [iceberg-python]

2024-07-04 Thread via GitHub
vinjai commented on PR #871: URL: https://github.com/apache/iceberg-python/pull/871#issuecomment-2209659324 @Fokko This PR is ready for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on code in PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#discussion_r1666118879 ## crates/catalog/glue/tests/glue_catalog_test.rs: ## @@ -92,6 +96,24 @@ async fn set_test_fixture(func: &str) -> TestFixture { } } +async fn lazy_reuse_dc()

Re: [I] Add more description to the Write-audit-publish feature [iceberg]

2024-07-04 Thread via GitHub
github-actions[bot] commented on issue #2802: URL: https://github.com/apache/iceberg/issues/2802#issuecomment-2209658051 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Alter table to generic types [iceberg]

2024-07-04 Thread via GitHub
github-actions[bot] commented on issue #2791: URL: https://github.com/apache/iceberg/issues/2791#issuecomment-2209658040 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Forward Compatible large_* type support: read as large, write as small [iceberg-python]

2024-07-04 Thread via GitHub
syun64 commented on code in PR #890: URL: https://github.com/apache/iceberg-python/pull/890#discussion_r1666094679 ## pyiceberg/table/__init__.py: ## @@ -1866,7 +1866,7 @@ def plan_files(self) -> Iterable[FileScanTask]: for data_entry in data_entries ] -

[PR] Bump pydantic from 2.8.0 to 2.8.2 [iceberg-python]

2024-07-04 Thread via GitHub
dependabot[bot] opened a new pull request, #894: URL: https://github.com/apache/iceberg-python/pull/894 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.8.0 to 2.8.2. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [I] Rename `data_sequence_number` to `sequence_number` [iceberg-python]

2024-07-04 Thread via GitHub
kevinjqliu commented on issue #893: URL: https://github.com/apache/iceberg-python/issues/893#issuecomment-2209610949 Is there a way on the Java/spark side to turn metadata information into JSON? With #535, perhaps we can compare the two JSON results and check for mismatches like this one.

Re: [PR] Bump Iceberg in CI to 1.5.2 [iceberg-python]

2024-07-04 Thread via GitHub
Fokko merged PR #878: URL: https://github.com/apache/iceberg-python/pull/878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-07-04 Thread via GitHub
lawofcycles commented on code in PR #10276: URL: https://github.com/apache/iceberg/pull/10276#discussion_r1666063693 ## aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogTable.java: ## @@ -221,6 +221,69 @@ public void testUpdateTable() { assertThat(respons

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-04 Thread via GitHub
Fokko merged PR #886: URL: https://github.com/apache/iceberg-python/pull/886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1666060990 ## mkdocs/docs/configuration.md: ## @@ -222,6 +222,12 @@ catalog: uri: sqlite:tmp/pyiceberg.db ``` +| Key | Example

Re: [I] Check nullability of `508: field_summary` [iceberg-python]

2024-07-04 Thread via GitHub
Fokko closed issue #891: Check nullability of `508: field_summary` URL: https://github.com/apache/iceberg-python/issues/891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Check nullability of `508: field_summary` [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on issue #891: URL: https://github.com/apache/iceberg-python/issues/891#issuecomment-2209531903 I checked with Spark, and this seems to be correct: ```json { "name" : "partitions", "type" : [ "null", { "type" : "array", "items" : {

Re: [PR] Forward Compatible large_* type support: read as large, write as small [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #890: URL: https://github.com/apache/iceberg-python/pull/890#discussion_r1666027762 ## pyiceberg/table/__init__.py: ## @@ -1866,7 +1866,7 @@ def plan_files(self) -> Iterable[FileScanTask]: for data_entry in data_entries ] -

Re: [I] Tracking issues of aligning storage support with iceberg-java [iceberg-rust]

2024-07-04 Thread via GitHub
jsimbadev commented on issue #408: URL: https://github.com/apache/iceberg-rust/issues/408#issuecomment-2209489934 @Xuanwo I can take the Azure datalake FileIO Implementation + the corresponding infrastructure set up, sound ok? -- This is an automated message from the Apache Git Service. T

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on PR #363: URL: https://github.com/apache/iceberg-python/pull/363#issuecomment-2209468950 Another test with `commit.manifest.min-count-to-merge` set to 100, and doing 500 append operations: ``` avro-tools tojson /tmp/some.db/w/metadata/snap-395291108779496

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1666006402 ## mkdocs/docs/api.md: ## @@ -273,6 +273,10 @@ tbl.append(df) # or +tbl.merge_append(df) Review Comment: btw, I like how you split it out in classes, it is

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on PR #363: URL: https://github.com/apache/iceberg-python/pull/363#issuecomment-2209460864 Doing some testing with `avro-tools`, asserting the state after 5 append operations with `"commit.manifest.min-count-to-merge": "2"` # V1 Table ## Manifest-list ###

[I] Rename `data_sequence_number` to `sequence_number` [iceberg-python]

2024-07-04 Thread via GitHub
Fokko opened a new issue, #893: URL: https://github.com/apache/iceberg-python/issues/893 ### Feature Request / Improvement It looks like a misnamed field slipped in: ```json { "status": 1, "snapshot_id": { "long": 898025966831056900 }, "

Re: [PR] Core: Fix create v1 table on REST Catalog [iceberg]

2024-07-04 Thread via GitHub
hantangwangd commented on code in PR #10369: URL: https://github.com/apache/iceberg/pull/10369#discussion_r1665997350 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -375,7 +376,16 @@ private static TableMetadata create(TableOperations ops, UpdateTable

Re: [I] Unable to load an iceberg table from aws glue catalog [iceberg-python]

2024-07-04 Thread via GitHub
kevinjqliu commented on issue #515: URL: https://github.com/apache/iceberg-python/issues/515#issuecomment-2209446665 Opened #892 to track the issue with AWS credential formats -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[I] [Bug] Load the proper AWS credential for glue/dynamodb catalog [iceberg-python]

2024-07-04 Thread via GitHub
kevinjqliu opened a new issue, #892: URL: https://github.com/apache/iceberg-python/issues/892 ### Apache Iceberg version None ### Please describe the bug 🐞 `s3.access-key-id` and `s3.secret-access-key` are the pyiceberg way to pass AWS credentials as properties to the ca

Re: [I] Unable to load an iceberg table from aws glue catalog [iceberg-python]

2024-07-04 Thread via GitHub
kevinjqliu commented on issue #515: URL: https://github.com/apache/iceberg-python/issues/515#issuecomment-2209443484 @impproductions Thanks for the detailed explanation. Great catch! Looking through the code, there's indeed an expectation for both AWS credential formats. `s3.acces

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-04 Thread via GitHub
kevinjqliu commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1665991459 ## mkdocs/docs/configuration.md: ## @@ -222,6 +222,12 @@ catalog: uri: sqlite:tmp/pyiceberg.db ``` +| Key | Example

[I] Check nullability of `508: field_summary` [iceberg-python]

2024-07-04 Thread via GitHub
Fokko opened a new issue, #891: URL: https://github.com/apache/iceberg-python/issues/891 ### Apache Iceberg version 0.6.1 (latest release) ### Please describe the bug 🐞 Field-id 508 is the struct in the list of partition summaries. It marked as optional, but I think it s

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1665964764 ## mkdocs/docs/api.md: ## @@ -273,6 +273,10 @@ tbl.append(df) # or +tbl.merge_append(df) Review Comment: Thanks! I think the use-case of the Java library is

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1665964062 ## pyiceberg/table/__init__.py: ## @@ -2998,21 +3074,22 @@ def _deleted_entries(self) -> List[ManifestEntry]: ... @abstractmethod def _existing_manifests(s

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-04 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1665281650 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthManager.java: ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Handles possible heap data corruption of `OAuth2Util.AuthSession#headers` [iceberg]

2024-07-04 Thread via GitHub
amogh-jahagirdar merged PR #10615: URL: https://github.com/apache/iceberg/pull/10615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Possible heap data corruption of `org.apache.iceberg.rest.auth.OAuth2Util.AuthSession#headers` [iceberg]

2024-07-04 Thread via GitHub
amogh-jahagirdar closed issue #10591: Possible heap data corruption of `org.apache.iceberg.rest.auth.OAuth2Util.AuthSession#headers` URL: https://github.com/apache/iceberg/issues/10591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Core: Fix create v1 table on REST Catalog [iceberg]

2024-07-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #10369: URL: https://github.com/apache/iceberg/pull/10369#discussion_r1665910677 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -991,7 +991,7 @@ public Builder assignUUID(String newUuid) { // it is only safe to

Re: [PR] Core: Fix create v1 table on REST Catalog [iceberg]

2024-07-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #10369: URL: https://github.com/apache/iceberg/pull/10369#discussion_r1665909446 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -375,7 +376,16 @@ private static TableMetadata create(TableOperations ops, UpdateT

Re: [I] Support server side data commit in REST [iceberg]

2024-07-04 Thread via GitHub
snazy commented on issue #10618: URL: https://github.com/apache/iceberg/issues/10618#issuecomment-2209310322 @jackye1995 looks like you linked the wrong document? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Flink: Fix `long` casting issues [iceberg]

2024-07-04 Thread via GitHub
pvary commented on PR #10629: URL: https://github.com/apache/iceberg/pull/10629#issuecomment-2209285181 Thanks for the fix @snazy! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Flink: Fix `long` casting issues [iceberg]

2024-07-04 Thread via GitHub
pvary merged PR #10629: URL: https://github.com/apache/iceberg/pull/10629 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] fix: Fix namespace identifier in url [iceberg-rust]

2024-07-04 Thread via GitHub
Xuanwo commented on code in PR #435: URL: https://github.com/apache/iceberg-rust/pull/435#discussion_r1665841971 ## crates/iceberg/src/catalog/mod.rs: ## @@ -123,9 +122,9 @@ impl NamespaceIdent { Self::from_vec(iter.into_iter().map(|s| s.to_string()).collect()) }

Re: [PR] HA HMS support [iceberg-python]

2024-07-04 Thread via GitHub
awdavidson commented on PR #752: URL: https://github.com/apache/iceberg-python/pull/752#issuecomment-2209205175 > @awdavidson This looks like a nice addition. Could you add a test as well? Unit tests have been added -- This is an automated message from the Apache Git Service. To res

Re: [PR] HA HMS support [iceberg-python]

2024-07-04 Thread via GitHub
awdavidson commented on code in PR #752: URL: https://github.com/apache/iceberg-python/pull/752#discussion_r1665835337 ## pyiceberg/catalog/hive.py: ## @@ -271,6 +271,18 @@ def __init__(self, name: str, **properties: str): DEFAULT_LOCK_CHECK_RETRIES, ) +

Re: [PR] fix: Fix namespace identifier in url [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 commented on PR #435: URL: https://github.com/apache/iceberg-rust/pull/435#issuecomment-2209197404 cc @thexiay Would you take a look if this helps? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] bug: rest catalog don't support multiple level namespace ident [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 commented on issue #433: URL: https://github.com/apache/iceberg-rust/issues/433#issuecomment-2209197794 Fixed in #435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Disallow writing empty Manifest files [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on PR #876: URL: https://github.com/apache/iceberg-python/pull/876#issuecomment-2209093993 Yes, when you create a new table and you write an empty dataframe, it will create a manfist-list that's pointing to zero manifests. I don't think there are any cases where you would wr

Re: [PR] Disallow writing empty Manifest files [iceberg-python]

2024-07-04 Thread via GitHub
syun64 commented on PR #876: URL: https://github.com/apache/iceberg-python/pull/876#issuecomment-2208991366 Hi @Fokko this is an interesting find. Is my understanding of the issue correct? When we are writing an empty commit, we are already creating an empty avro file (manifest list)

Re: [I] Deprecate `adlfs.*` configuration properties in favor of `adls.*` [iceberg-python]

2024-07-04 Thread via GitHub
ndrluis commented on issue #866: URL: https://github.com/apache/iceberg-python/issues/866#issuecomment-2208987634 Hello @Fokko, I'd like to resolve this issue. I just want to confirm the expected behavior. Since we are replacing ADLFS with ADLS, I believe the expected behavior should

Re: [PR] Deprecate `oauth/tokens` endpoint [iceberg]

2024-07-04 Thread via GitHub
snazy commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1665623076 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -176,19 +176,34 @@ public void initialize(String name, Map unresolved) { long startTi

Re: [PR] Core: Assume issued_token_type is access_token to fully comply with RFC 6749 [iceberg]

2024-07-04 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2208819566 Thank you to all involved! I'm very glad that we could reach a consensus here πŸ™ -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Core: Assume issued_token_type is access_token to fully comply with RFC 6749 [iceberg]

2024-07-04 Thread via GitHub
nastra merged PR #10314: URL: https://github.com/apache/iceberg/pull/10314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Support create multiple element ns together for nessie [iceberg]

2024-07-04 Thread via GitHub
snazy commented on code in PR #10630: URL: https://github.com/apache/iceberg/pull/10630#discussion_r1665606754 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java: ## @@ -290,6 +291,14 @@ public void renameTable(TableIdentifier from, TableIdentifier to) {

Re: [I] bug: rest catalog don't support multiple level namespace ident [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 commented on issue #433: URL: https://github.com/apache/iceberg-rust/issues/433#issuecomment-2208778213 Cool, I will take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] bug: use s3 endpoint fail to read manifest list with s3a location [iceberg-rust]

2024-07-04 Thread via GitHub
chenzl25 opened a new issue, #434: URL: https://github.com/apache/iceberg-rust/issues/434 If an iceberg table is created and inserted with data by spark, we will get a manifest list with prefix `s3a`. ``` spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3

Re: [I] Unable to load an iceberg table from aws glue catalog [iceberg-python]

2024-07-04 Thread via GitHub
impproductions commented on issue #515: URL: https://github.com/apache/iceberg-python/issues/515#issuecomment-2208697240 We had the same problem within our Airflow deployment. The easy fix for us would have been to set the default aws credentials through environment variables: ```bash

Re: [PR] Support create multiple element ns together for nessie [iceberg]

2024-07-04 Thread via GitHub
zymap commented on PR #10630: URL: https://github.com/apache/iceberg/pull/10630#issuecomment-2208681983 @snazy Please take a look when you have time. Thanks! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[PR] Support create multiple element ns together for nessie [iceberg]

2024-07-04 Thread via GitHub
zymap opened a new pull request, #10630: URL: https://github.com/apache/iceberg/pull/10630 --- ### Motivation Create multiple level namespace together when creating namespace. For more time, the namespace level is one. But sometimes when we have multiple levels to construct the na

Re: [I] bug: rest catalog don't support multiple level namespace ident [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay commented on issue #433: URL: https://github.com/apache/iceberg-rust/issues/433#issuecomment-2208658504 > Interesting, could you elaborate the rest catalog server you use? In fact our integration tests have things dealing with multi level namespace ident: > > https://github.c

Re: [I] Running MERGE INTO with more than one WHEN condition fails if the number of columns in the target table is > 321 [iceberg]

2024-07-04 Thread via GitHub
krishan711 commented on issue #10294: URL: https://github.com/apache/iceberg/issues/10294#issuecomment-2208654859 I have the same issue also. i was hoping delete and then insert would work in separate statements (just to test it) but even this fails with too many columns: ``` MERGE IN

Re: [PR] Disallow writing empty Manifest files [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on PR #876: URL: https://github.com/apache/iceberg-python/pull/876#issuecomment-2208625219 I didn't think the fact that we allowed writing empty commits, so I was a bit strong in my first approach. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-04 Thread via GitHub
Fokko commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1665466731 ## mkdocs/docs/configuration.md: ## @@ -222,6 +222,12 @@ catalog: uri: sqlite:tmp/pyiceberg.db ``` +| Key | Example

[PR] Flink: Fix `long` casting issues [iceberg]

2024-07-04 Thread via GitHub
snazy opened a new pull request, #10629: URL: https://github.com/apache/iceberg/pull/10629 Backport of #10580 and resolve IDE warning/error wrt `floorMod`/`floorDiv`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] bug: rest catalog don't support multiple level namespace ident [iceberg-rust]

2024-07-04 Thread via GitHub
liurenjie1024 commented on issue #433: URL: https://github.com/apache/iceberg-rust/issues/433#issuecomment-2208492052 Interesting, could you elaborate the rest catalog server you use? In fact our integration tests have things dealing with multi level namespace ident: https://github.com/apac

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-04 Thread via GitHub
fengjiajie commented on code in PR #10565: URL: https://github.com/apache/iceberg/pull/10565#discussion_r1665382438 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/data/RowDataUtil.java: ## @@ -79,7 +79,11 @@ public static Object convertConstant(Type type, Object va

Re: [PR] Makefile clean utility to remove cython cached objects [iceberg-python]

2024-07-04 Thread via GitHub
MehulBatra commented on PR #881: URL: https://github.com/apache/iceberg-python/pull/881#issuecomment-2208439531 @Fokko do see when you get a chance, waiting for your feedback too! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-04 Thread via GitHub
nastra commented on code in PR #10314: URL: https://github.com/apache/iceberg/pull/10314#discussion_r1665347177 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -738,13 +738,21 @@ private static AuthSession fromTokenResponse( long startTimeMilli

Re: [I] bug: rest catalog don't support multiple level namespace ident [iceberg-rust]

2024-07-04 Thread via GitHub
Xuanwo commented on issue #433: URL: https://github.com/apache/iceberg-rust/issues/433#issuecomment-2208389939 Thanks for report! Would you like to give it a fix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-04 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1665327420 ## open-api/rest-catalog-open-api.yaml: ## @@ -61,12 +65,26 @@ security: - OAuth2: [catalog] - BearerAuth: [] +# Capability tags describe optional functionality

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-04 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1665314780 ## open-api/rest-catalog-open-api.yaml: ## @@ -191,7 +220,8 @@ paths: get: tags: -- Catalog API +- tables +- views Review Comment:

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-04 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1665304639 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthSession.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-04 Thread via GitHub
twuebi commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2208353517 Thanks for checking this @nastra, I understand that we should be relaxing the constraints of NamespaceIdent then if an empty default-namespace is expected. -- This is an automated me

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-04 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1665284071 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthManager.java: ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-04 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1665281650 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthManager.java: ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Create rollback and set snapshot APIs [iceberg-python]

2024-07-04 Thread via GitHub
chinmay-bhat commented on code in PR #758: URL: https://github.com/apache/iceberg-python/pull/758#discussion_r1665246340 ## pyiceberg/table/__init__.py: ## @@ -1975,6 +1948,52 @@ def _commit(self) -> UpdatesAndRequirements: """Apply the pending changes and commit."""

[I] bug: invalid unicode transfer in Catalog namespace interface [iceberg-rust]

2024-07-04 Thread via GitHub
thexiay opened a new issue, #433: URL: https://github.com/apache/iceberg-rust/issues/433 replace test unit `rest_catalog_test::test_list_empty_namespace` to this: ``` #[tokio::test] async fn test_list_empty_namespace() { let fixture = get_test_fixture("test_list_empty_namespac

Re: [PR] Create rollback and set snapshot APIs [iceberg-python]

2024-07-04 Thread via GitHub
chinmay-bhat commented on code in PR #758: URL: https://github.com/apache/iceberg-python/pull/758#discussion_r1665246340 ## pyiceberg/table/__init__.py: ## @@ -1975,6 +1948,52 @@ def _commit(self) -> UpdatesAndRequirements: """Apply the pending changes and commit."""

Re: [PR] Fixed an incorrect example [iceberg]

2024-07-04 Thread via GitHub
ajantha-bhat commented on code in PR #10627: URL: https://github.com/apache/iceberg/pull/10627#discussion_r1665247303 ## docs/docs/spark-getting-started.md: ## @@ -48,8 +48,6 @@ This command creates a path-based catalog named `local` for tables under `$PWD/w ```sh spark-sql -

Re: [PR] Create rollback and set snapshot APIs [iceberg-python]

2024-07-04 Thread via GitHub
chinmay-bhat commented on code in PR #758: URL: https://github.com/apache/iceberg-python/pull/758#discussion_r1665246340 ## pyiceberg/table/__init__.py: ## @@ -1975,6 +1948,52 @@ def _commit(self) -> UpdatesAndRequirements: """Apply the pending changes and commit."""

Re: [PR] AWS: Add Option to don't write non current columns in glue schema closes #7584 [iceberg]

2024-07-04 Thread via GitHub
Raphael-Vignes commented on PR #9420: URL: https://github.com/apache/iceberg/pull/9420#issuecomment-2208273896 Sorry but I never finished this PR, I'm not sure it's fully functional. I'll try to finish it when I've time. -- This is an automated message from the Apache Git Service. To resp