Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

nastra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400147481 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -104,12 +96,18 @@ protected void doRefresh() { content

[I] Spark: inconsistency in rewrite data and summary [iceberg]

Fokko opened a new issue, #7463: URL: https://github.com/apache/iceberg/issues/7463 ### Apache Iceberg version 1.2.1 (latest release) ### Query engine Spark ### Please describe the bug  Make sure to set the number of partitions to one, to avoid having

Re: [I] PARSE_SYNTAX_ERROR was encountered for 'ALTER TABLE ... CREATE BRANCH ... [iceberg]

nastra commented on issue #9116: URL: https://github.com/apache/iceberg/issues/9116#issuecomment-1820372169 @jackskmok it appears that you're missing `--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` -- This is an automated message from the

Re: [PR] Core: Remove synchronization from BitmapPositionDeleteIndex [iceberg]

nastra commented on PR #9119: URL: https://github.com/apache/iceberg/pull/9119#issuecomment-1820369632 thanks @aokolnychyi for fixing this and @amogh-jahagirdar for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core: Remove synchronization from BitmapPositionDeleteIndex [iceberg]

nastra merged PR #9119: URL: https://github.com/apache/iceberg/pull/9119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Data: Always use delete index for position deletes [iceberg]

amogh-jahagirdar commented on code in PR #9117: URL: https://github.com/apache/iceberg/pull/9117#discussion_r1400123389 ## data/src/main/java/org/apache/iceberg/data/DeleteFilter.java: ## @@ -245,18 +242,9 @@ private CloseableIterable applyPosDeletes(CloseableIterable records)

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

mobley-trent commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1400115507 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400110138 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -104,12 +96,18 @@ protected void doRefresh() { content

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400108586 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400104301 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -267,4 +306,23 @@ static DataFile makeDataFile(Table icebergTable, String

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400100247 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieViewCatalog.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] feat: Add support for `UnboundPartitionSpec`. [iceberg-rust]

my-vegetable-has-exploded commented on issue #98: URL: https://github.com/apache/iceberg-rust/issues/98#issuecomment-1820337516 > Hi, @my-vegetable-has-exploded I mean I don't want to add `Option` in current `PartitionSpec`, but it's ok to add it in `UnboundPartitonSpec`. get it!,

Re: [I] feat: Add support for `UnboundPartitionSpec`. [iceberg-rust]

liurenjie1024 commented on issue #98: URL: https://github.com/apache/iceberg-rust/issues/98#issuecomment-1820300963 Hi, @my-vegetable-has-exploded I mean I don't want to add `Option` in current `PartitionSpec`, but it's ok to add it in `UnboundPartitonSpec`. -- This is an automated

Re: [PR] Core: Remove synchronization from BitmapPositionDeleteIndex [iceberg]

aokolnychyi closed pull request #9119: Core: Remove synchronization from BitmapPositionDeleteIndex URL: https://github.com/apache/iceberg/pull/9119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1820198393 Does the data analyst connect to a remote already running cluster? Do we assume the jar is populated on executors but not supplied by the analyst in the local shell? -- This is an

Re: [PR] Core: Remove synchronization from BitmapPositionDeleteIndex [iceberg]

aokolnychyi commented on PR #9119: URL: https://github.com/apache/iceberg/pull/9119#issuecomment-1820181900 @nastra @wypoon, could you check whether my analysis was correct? Let me know if I missed something. cc @amogh-jahagirdar @singhpk234 @Fokko as well -- This is an automated

[PR] Core: Remove synchronization from BitmapPositionDeleteIndex [iceberg]

aokolnychyi opened a new pull request, #9119: URL: https://github.com/apache/iceberg/pull/9119 This PR removes synchronization from `BitmapPositionDeleteIndex` added in PR #8805. We switched to `ParallelIterable` to parallelize reading of deletes in the original change. However,

[I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]

irshadcc opened a new issue, #9118: URL: https://github.com/apache/iceberg/issues/9118 ### Feature Request / Improvement During the snapshot commit process of MergingSnapshotProducer, Iceberg tries to merge the manifest files to increase the planning performance. During operations

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

liurenjie1024 commented on PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#issuecomment-1820111558 > As I mentioned earlier, having the REST Docker images would make testing a lot easier. I've tracked this in #100, and will do it after this get merged. This pr is already

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

liurenjie1024 commented on code in PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1399952861 ## crates/catalog/rest/src/catalog.rs: ## @@ -1017,31 +1109,31 @@ mod tests { .with_summary(Summary { operation:

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

liurenjie1024 commented on code in PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1399952745 ## crates/catalog/rest/src/catalog.rs: ## @@ -1092,4 +1184,375 @@ mod tests { config_mock.assert_async().await;

Re: [PR] Spark 3.5: Honor Spark conf spark.sql.files.maxPartitionBytes in read split [iceberg]

manuzhang commented on PR #8922: URL: https://github.com/apache/iceberg/pull/8922#issuecomment-1820108319 We have similar requests to set platform level split size to reduce read RPCs to HDFS. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Spark: inconsistency in rewrite data and summary [iceberg]

github-actions[bot] closed issue #7463: Spark: inconsistency in rewrite data and summary URL: https://github.com/apache/iceberg/issues/7463 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Spark: inconsistency in rewrite data and summary [iceberg]

github-actions[bot] commented on issue #7463: URL: https://github.com/apache/iceberg/issues/7463#issuecomment-1820009219 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] iceberg table query bug [iceberg]

github-actions[bot] commented on issue #7640: URL: https://github.com/apache/iceberg/issues/7640#issuecomment-1820009144 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] Core: Always use delete index for position deletes [iceberg]

aokolnychyi commented on PR #9117: URL: https://github.com/apache/iceberg/pull/9117#issuecomment-1819963551 cc @amogh-jahagirdar @nastra @RussellSpitzer @flyrain @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Docs: Remove UNIQUE keyword as it is not supported in Flink [iceberg]

aokolnychyi commented on PR #9046: URL: https://github.com/apache/iceberg/pull/9046#issuecomment-1819964784 cc @pvary @stevenzwu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Always use delete index for position deletes [iceberg]

aokolnychyi commented on code in PR #9117: URL: https://github.com/apache/iceberg/pull/9117#discussion_r1399860140 ## data/src/main/java/org/apache/iceberg/data/DeleteFilter.java: ## @@ -245,18 +242,9 @@ private CloseableIterable applyPosDeletes(CloseableIterable records) {

Re: [PR] Core: Always use delete index for position deletes [iceberg]

aokolnychyi commented on code in PR #9117: URL: https://github.com/apache/iceberg/pull/9117#discussion_r1399860140 ## data/src/main/java/org/apache/iceberg/data/DeleteFilter.java: ## @@ -245,18 +242,9 @@ private CloseableIterable applyPosDeletes(CloseableIterable records) {

Re: [PR] Spark: Add spark.sql.iceberg.locality.enabled to control whether fetch locali… [iceberg]

aokolnychyi merged PR #9101: URL: https://github.com/apache/iceberg/pull/9101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Spark: Add spark.sql.iceberg.locality.enabled to control whether fetch locali… [iceberg]

aokolnychyi commented on PR #9101: URL: https://github.com/apache/iceberg/pull/9101#issuecomment-1819950516 Thanks, @zhongyujiang! Thanks for reviewing, @singhpk234! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.10 [iceberg-python]

dependabot[bot] opened a new pull request, #155: URL: https://github.com/apache/iceberg-python/pull/155 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.4.8 to 9.4.10. Release notes Sourced from

Re: [PR] Spark 3.3, 3.4: Backport fix for metadata delete condition check for branches [iceberg]

amogh-jahagirdar merged PR #9115: URL: https://github.com/apache/iceberg/pull/9115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Spark 3.3, 3.4: Backport fix for metadata delete condition check for branches [iceberg]

amogh-jahagirdar commented on PR #9115: URL: https://github.com/apache/iceberg/pull/9115#issuecomment-1819881061 Thanks for reviews @danielcweeks @singhpk234 ! merging since this was a straightforward backport -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1350603878 ## data/src/main/java/org/apache/iceberg/data/DeleteLoader.java: ## @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1399806008 ## data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1399804623 ## core/src/main/java/org/apache/iceberg/deletes/PositionDeleteIndex.java: ## @@ -44,4 +44,14 @@ public interface PositionDeleteIndex { /** Returns true if

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1399800368 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -452,6 +454,59 @@ private static void checkSchemaCompatibility( } } + public static

[I] PARSE_SYNTAX_ERROR was encountered for 'ALTER TABLE ... CREATE BRANCH ... [iceberg]

jackskmok opened a new issue, #9116: URL: https://github.com/apache/iceberg/issues/9116 I am trying to testing the create branch feature with the table with AWS Glue catalog , but hit the error as title. Here is the message: >>> spark.sql("alter table iceberg.athena_data create

Re: [PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.10 [iceberg]

nastra merged PR #9114: URL: https://github.com/apache/iceberg/pull/9114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Use Strings to verify null and empty string [iceberg]

ndrluis commented on PR #9090: URL: https://github.com/apache/iceberg/pull/9090#issuecomment-1819760036 @nastra @Fokko Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Use Strings to verify null and empty string [iceberg]

ndrluis commented on PR #9090: URL: https://github.com/apache/iceberg/pull/9090#issuecomment-1819759619 @nastra @ndrluis Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] iceberg-aliyun: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on issue #9081: URL: https://github.com/apache/iceberg/issues/9081#issuecomment-1819705926 @lisirrx great, feel free to go ahead and open a PR once you have something and tag me for a review -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Update Slack URL [iceberg-docs]

Fokko merged PR #294: URL: https://github.com/apache/iceberg-docs/pull/294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Update Slack URL [iceberg-docs]

Fokko opened a new pull request, #294: URL: https://github.com/apache/iceberg-docs/pull/294 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

syun64 commented on PR #8976: URL: https://github.com/apache/iceberg/pull/8976#issuecomment-1819503960 Incorporated suggested changes @nastra . Thank you for the reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] iceberg-aliyun: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

lisirrx commented on issue #9081: URL: https://github.com/apache/iceberg/issues/9081#issuecomment-1819356677 @nastra Hi, I have read the discussion in #7160 and read some packages which are already changed to JUnit5. I want to have a try. -- This is an automated message from the Apache

Re: [I] iceberg reports an error after upgrading to 1.4.2 [iceberg]

huanghanyu-vungle commented on issue #9018: URL: https://github.com/apache/iceberg/issues/9018#issuecomment-1819237615 ` def saveReport(data: DataFrame, targetTable: String, isPublisherReport: Boolean): Unit = { val tmpTableName = "__tmp"

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1399328620 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache

Re: [PR] Nessie: reimplement namespace operations [iceberg]

dimas-b commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1819117195 I'm fine with using `ClientSide*` builders for Namespace manipulation. However, are we not getting the benefit of refactoring catalog code to be more concise and straight-forward

Re: [I] feat: Add support for `UnboundPartitionSpec`. [iceberg-rust]

my-vegetable-has-exploded commented on issue #98: URL: https://github.com/apache/iceberg-rust/issues/98#issuecomment-1819116596 Sorry for my misunderstanding and thanks for your patience @liurenjie1024. Without ```option```, UnboundPartitionSpec and UnboundPartitionField may like this?

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1399218687 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + +

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

Xuanwo commented on code in PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1399106512 ## crates/catalog/rest/src/catalog.rs: ## @@ -446,6 +490,24 @@ impl RestCatalog { Ok(()) } + +fn load_file_io( +, +

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1399025309 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1399024202 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software

Re: [PR] Replace i64 with DateTime [iceberg-rust]

liurenjie1024 commented on PR #94: URL: https://github.com/apache/iceberg-rust/pull/94#issuecomment-1818822751 The failure is caused by upgrades of `uuid`. You can solve it by fixing version as [this commit]() -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Replace i64 with DateTime [iceberg-rust]

Fokko commented on PR #94: URL: https://github.com/apache/iceberg-rust/pull/94#issuecomment-1818816616 Looks like we have a sad clippy: ``` ``` error: argument to `panic!()` in a const context must have type `` --> crates/catalog/rest/src/catalog.rs:985:13 | 985 |

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

Fokko commented on code in PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1398974623 ## crates/catalog/rest/src/catalog.rs: ## @@ -586,16 +648,46 @@ mod _serde { pub(super) metadata: TableMetadata, pub(super) config: Option>, } +

Re: [PR] Build: Bump rich from 13.6.0 to 13.7.0 [iceberg-python]

Fokko merged PR #151: URL: https://github.com/apache/iceberg-python/pull/151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Override useCommitCoordinator to false [iceberg]

jbonofre commented on PR #9017: URL: https://github.com/apache/iceberg/pull/9017#issuecomment-1818750776 Thanks for the report. Let me take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Update table metadata [iceberg-python]

Fokko commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1398855771 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] Update table metadata [iceberg-python]

Fokko commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1398847839 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.9 [iceberg]

dependabot[bot] closed pull request #9108: Build: Bump mkdocs-material from 9.4.8 to 9.4.9 URL: https://github.com/apache/iceberg/pull/9108 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.9 [iceberg]

dependabot[bot] commented on PR #9108: URL: https://github.com/apache/iceberg/pull/9108#issuecomment-1818551723 Superseded by #9114. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.10 [iceberg]

dependabot[bot] opened a new pull request, #9114: URL: https://github.com/apache/iceberg/pull/9114 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.4.8 to 9.4.10. Release notes Sourced from

Re: [PR] Build: Bump mkdocs-material from 9.4.8 to 9.4.9 [iceberg]

nastra commented on PR #9108: URL: https://github.com/apache/iceberg/pull/9108#issuecomment-1818542037 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] chore(deps): Update itertools requirement from 0.11 to 0.12 [iceberg-rust]

Fokko merged PR #102: URL: https://github.com/apache/iceberg-rust/pull/102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] chore(deps): Update opendal requirement from 0.41 to 0.42 [iceberg-rust]

Fokko merged PR #101: URL: https://github.com/apache/iceberg-rust/pull/101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: