Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579323906 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
nastra merged PR #10221: URL: https://github.com/apache/iceberg/pull/10221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

2024-04-25 Thread via GitHub
pvary commented on PR #10208: URL: https://github.com/apache/iceberg/pull/10208#issuecomment-2076940629 Are we good to go with this @stevenzwu ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579292821 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,58 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on PR #10200: URL: https://github.com/apache/iceberg/pull/10200#issuecomment-2076931744 > After taking a closer look at `BaseTaskWriter`, I think we may have a correctness issue when encoding changes if the table contains multiple specs. Our current implementation of

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579259488 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -174,20 +208,21 @@ private void flushDeletes() { } catch (IOException e) {

Re: [I] Test suite for manifest filtering [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #351: URL: https://github.com/apache/iceberg-rust/issues/351#issuecomment-2076879150 Assigned, thanks @s-akhtar-baig ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579255478 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -118,6 +118,10 @@ private DeleteWriteResult writeFileDeletes() throws

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579255109 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [I] Implement the equality delete writer [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #341: URL: https://github.com/apache/iceberg-rust/issues/341#issuecomment-2076878340 Assigned to you, thanks @Dysprosium0626 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Implement all functions of BoundPredicateVisitor for ManifestFilterVisitor [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on issue #350: URL: https://github.com/apache/iceberg-rust/issues/350#issuecomment-2076877306 @s-akhtar-baig Assigned, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

2024-04-25 Thread via GitHub
liurenjie1024 commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1579248989 ## crates/iceberg/src/scan.rs: ## @@ -158,8 +196,24 @@ impl TableScan { .await?; // Generate data file stream -let mut

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579215950 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579208675 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -59,20 +60,58 @@ class SortedPosDeleteWriter implements FileWriter, DeleteWr

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579204149 ## core/src/main/java/org/apache/iceberg/io/SortedPosDeleteWriter.java: ## @@ -174,20 +208,21 @@ private void flushDeletes() { } catch (IOException e) {

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579198316 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[I] MinIO + Spark + hive metadata + iceberg format [iceberg]

2024-04-25 Thread via GitHub
rychu151 opened a new issue, #10222: URL: https://github.com/apache/iceberg/issues/10222 ### Query engine Spark ### Question Im trying to setup local develop env for my testing purposes using docker **Target is to save dataframe in a Iceberg format and

Re: [PR] Core: Use 'delete' / 'append' if OverwriteFiles only deletes/appends data files [iceberg]

2024-04-25 Thread via GitHub
nastra merged PR #10150: URL: https://github.com/apache/iceberg/pull/10150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Sql catalog [iceberg-rust]

2024-04-25 Thread via GitHub
JanKaul commented on PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#issuecomment-2076689179 Thank you all for your helpful comments. I think the PR is ready for review again. @liurenjie1024 @sdd @odysa @ZENOTME @martin-g -- This is an automated message from the

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10221: URL: https://github.com/apache/iceberg/pull/10221#issuecomment-2076687909 Hi @nastra would you review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1579107619 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

[PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

2024-04-25 Thread via GitHub
aajisaka opened a new pull request, #10221: URL: https://github.com/apache/iceberg/pull/10221 ## Problem TestGlueCatalogTable#testCreateTable failed by the following assertion error: ``` org.opentest4j.AssertionFailedError: expected:

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

2024-04-25 Thread via GitHub
pvary commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1579101566 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc,

Re: [I] Is the "Emitting watermarks" new feature can't be used in flink sql? [iceberg]

2024-04-25 Thread via GitHub
pvary commented on issue #10219: URL: https://github.com/apache/iceberg/issues/10219#issuecomment-2076664228 @yeezychao: Do you happen to know, what is needed from the connector side to make this work? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1579083787 ## pyiceberg/catalog/__init__.py: ## @@ -710,6 +760,45 @@ def _get_updated_props_and_update_summary( return properties_update_summary, updated_properties

Re: [PR] Refactor GlueCatalog's _commit_table [iceberg-python]

2024-04-25 Thread via GitHub
Fokko merged PR #653: URL: https://github.com/apache/iceberg-python/pull/653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1579059939 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -295,5 +343,6 @@ public void testValidatedOverwriteWithAppendSuccess() {

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

2024-04-25 Thread via GitHub
nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1579060905 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -135,6 +135,49 @@ public void createTestTable() throws IOException { commit(table,

Re: [PR] Test: Add test to partition on field with a dot [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #610: URL: https://github.com/apache/iceberg-python/pull/610#discussion_r1579050799 ## tests/integration/test_reads.py: ## @@ -473,6 +474,31 @@ def test_sanitize_character(catalog: Catalog) -> None: assert arrow_table.schema.names[0] ==

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

2024-04-25 Thread via GitHub
ludlows commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1579048510 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc,

[I] AWS: Updating Glue catalog table removes column descriptions [iceberg]

2024-04-25 Thread via GitHub
aajisaka opened a new issue, #10220: URL: https://github.com/apache/iceberg/issues/10220 ### Apache Iceberg version main (development) ### Query engine Spark ### Please describe the bug  In AWS Glue Catalog, user can set arbitrary descriptions to the table

Re: [PR] Introduce hierarchical namespaces into SqlCatalog [iceberg-python]

2024-04-25 Thread via GitHub
Fokko commented on PR #591: URL: https://github.com/apache/iceberg-python/pull/591#issuecomment-2076573394 > Should we allow table identifiers to have no namespace at all, like we do in Java? (pending @Fokko ) No, I don't think we should allow this. We also disallow this for the REST

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2076564061 Also, I built the jars and manually tested on AWS Glue as a Spark runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579004695 ## crates/iceberg/src/io.rs: ## @@ -368,6 +368,9 @@ impl Storage { new_props.insert("root".to_string(), DEFAULT_ROOT_PATH.to_string()); match

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579004695 ## crates/iceberg/src/io.rs: ## @@ -368,6 +368,9 @@ impl Storage { new_props.insert("root".to_string(), DEFAULT_ROOT_PATH.to_string()); match

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1578992843 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [I] Tracking issues of iceberg-rust v0.3.0 [iceberg-rust]

2024-04-25 Thread via GitHub
Fokko commented on issue #348: URL: https://github.com/apache/iceberg-rust/issues/348#issuecomment-2076518611 > Hi, @Fokko About the read projection part, currently we can convert parquet files into arrow streams, but there are some limitations: it only support primitive types, and schema

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

2024-04-25 Thread via GitHub
aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2076512208 I ran the integration test using my AWS account and the `testUpdateTable()` was successful. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] OSSFileIO not compatible with aliyun-sdk-oss higher than 3.11.3 [iceberg]

2024-04-25 Thread via GitHub
769484623 commented on issue #9934: URL: https://github.com/apache/iceberg/issues/9934#issuecomment-2076510336 > Currently the version being used in Iceberg is > > https://github.com/apache/iceberg/blob/b714978583f5ba32ec34396cb82b0a8f100bf837/gradle/libs.versions.toml#L24 > >

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-25 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1578924531 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Build: Bump pytest-mock from 3.12.0 to 3.14.0 [iceberg-python]

2024-04-25 Thread via GitHub
HonahX merged PR #658: URL: https://github.com/apache/iceberg-python/pull/658 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Build: Bump rich from 13.7.0 to 13.7.1 [iceberg-python]

2024-04-25 Thread via GitHub
HonahX merged PR #648: URL: https://github.com/apache/iceberg-python/pull/648 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

<    1   2