Re: [PR] Support identifier warehouses [iceberg-rust]

liurenjie1024 merged PR #308: URL: https://github.com/apache/iceberg-rust/pull/308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

liurenjie1024 commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1552969104 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,363 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: Project transform [iceberg-rust]

liurenjie1024 commented on PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#issuecomment-2039007641 cc @Fokko Do you have other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] feat: Project transform [iceberg-rust]

liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1548808318 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,323 @@ impl Transform { _ => self == other, } } + +/// Projects a giv

Re: [I] cannot insert value in hive command shell [iceberg]

github-actions[bot] commented on issue #2442: URL: https://github.com/apache/iceberg/issues/2442#issuecomment-2038481456 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Can snapshot has an optional name? [iceberg]

github-actions[bot] commented on issue #2231: URL: https://github.com/apache/iceberg/issues/2231#issuecomment-2038481247 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Can snapshot has an optional name? [iceberg]

github-actions[bot] closed issue #2231: Can snapshot has an optional name? URL: https://github.com/apache/iceberg/issues/2231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] flink 1.12.0 cannot run iceberg batch mode [iceberg]

github-actions[bot] closed issue #2225: flink 1.12.0 cannot run iceberg batch mode URL: https://github.com/apache/iceberg/issues/2225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] flink 1.12.0 cannot run iceberg batch mode [iceberg]

github-actions[bot] commented on issue #2225: URL: https://github.com/apache/iceberg/issues/2225#issuecomment-2038481227 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] What is the meaning of `delete_rows_count` and `delete_data_count_file ` at manifest [iceberg]

github-actions[bot] commented on issue #2445: URL: https://github.com/apache/iceberg/issues/2445#issuecomment-2038481477 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Add `AlwaysTrue` and `AlwaysFalse` to `Predicate` [iceberg-rust]

sdd commented on PR #319: URL: https://github.com/apache/iceberg-rust/pull/319#issuecomment-2038369738 merged this into https://github.com/apache/iceberg-rust/pull/320 as it is a bit pointless on its own -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add `AlwaysTrue` and `AlwaysFalse` to `Predicate` [iceberg-rust]

sdd closed pull request #319: Add `AlwaysTrue` and `AlwaysFalse` to `Predicate` URL: https://github.com/apache/iceberg-rust/pull/319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

sdd commented on PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#issuecomment-2038350682 PTAL @liurenjie1024 and @marvinlanhenke - extracted from https://github.com/apache/iceberg-rust/pull/241 and added tests -- This is an automated message from the Apache Git Service. To r

Re: [I] Support partial overwrites [iceberg-python]

whynick1 commented on issue #268: URL: https://github.com/apache/iceberg-python/issues/268#issuecomment-2038339351 > @whynick1 This should be done once #554 is in. Would you like to provide one of the metadata tables? See #511 @Fokko In that case, can I take https://github.com/apache

Re: [PR] Add a few getters that are required for the evaluating manifests in TableScan [iceberg-rust]

sdd closed pull request #318: Add a few getters that are required for the evaluating manifests in TableScan URL: https://github.com/apache/iceberg-rust/pull/318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] OpenAPI: Fix additionalProperties for SnapshotSummary [iceberg]

Fokko merged PR #9838: URL: https://github.com/apache/iceberg/pull/9838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] OpenAPI: Fix additionalProperties for SnapshotSummary [iceberg]

Fokko commented on code in PR #9838: URL: https://github.com/apache/iceberg/pull/9838#discussion_r1552476063 ## open-api/rest-catalog-open-api.py: ## @@ -171,7 +171,6 @@ class SortOrder(BaseModel): class Summary(BaseModel): operation: Literal['append', 'replace', 'overwr

Re: [PR] Add a few getters that are required for the evaluating manifests in TableScan [iceberg-rust]

sdd commented on PR #318: URL: https://github.com/apache/iceberg-rust/pull/318#issuecomment-2038244514 @liurenjie1024 and @marvinlanhenke - small PR split out from https://github.com/apache/iceberg-rust/pull/241 ready for review :-) -- This is an automated message from the Apache Git Ser

Re: [PR] Add `AlwaysTrue` and `AlwaysFalse` to `Predicate` [iceberg-rust]

sdd commented on PR #319: URL: https://github.com/apache/iceberg-rust/pull/319#issuecomment-2038240842 @liurenjie1024 and @marvinlanhenke - small PR split out from https://github.com/apache/iceberg-rust/pull/241 ready for review! -- This is an automated message from the Apache Git Servic

[PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd opened a new pull request, #323: URL: https://github.com/apache/iceberg-rust/pull/323 This PR was broken out of https://github.com/apache/iceberg-rust/pull/241 as that PR was getting too large. It depends on https://github.com/apache/iceberg-rust/pull/322, and integrates the `Man

[PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd opened a new pull request, #322: URL: https://github.com/apache/iceberg-rust/pull/322 This PR has been broken out of https://github.com/apache/iceberg-rust/pull/241 as that PR was getting too large. It depends on https://github.com/apache/iceberg-rust/pull/320 and https://github

Re: [PR] Build: Bump junit from 5.10.1 to 5.10.2 [iceberg]

Fokko commented on PR #9699: URL: https://github.com/apache/iceberg/pull/9699#issuecomment-2038140183 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Build: Bump calcite from 1.10.0 to 1.35.0 [iceberg]

dependabot[bot] commented on PR #8239: URL: https://github.com/apache/iceberg/pull/8239#issuecomment-2038141263 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency by

Re: [PR] Build: Bump calcite from 1.10.0 to 1.35.0 [iceberg]

Fokko commented on PR #8239: URL: https://github.com/apache/iceberg/pull/8239#issuecomment-2038141211 Duplicate of https://github.com/apache/iceberg/pull/9042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Build: Bump calcite from 1.10.0 to 1.35.0 [iceberg]

Fokko closed pull request #8239: Build: Bump calcite from 1.10.0 to 1.35.0 URL: https://github.com/apache/iceberg/pull/8239 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.7 [iceberg]

Fokko merged PR #9912: URL: https://github.com/apache/iceberg/pull/9912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[PR] add `InclusiveProjection` Visitor [iceberg-rust]

sdd opened a new pull request, #321: URL: https://github.com/apache/iceberg-rust/pull/321 This PR has been broken out of https://github.com/apache/iceberg-rust/pull/241 as it was getting too large. The `InclusiveProjection` is used in the process of filtering manifest files in table

[PR] Add `BoundPredicateEvaluator` trait [iceberg-rust]

sdd opened a new pull request, #320: URL: https://github.com/apache/iceberg-rust/pull/320 This trait is used for `BoundPredicate` visitors that evaluate the predicate to a boolean, and associated tests. It has been broken out of the https://github.com/apache/iceberg-rust/pull/241 PR as that

[PR] Add a few getters that are required for the evaluating manifests in TableScan [iceberg-rust]

sdd opened a new pull request, #318: URL: https://github.com/apache/iceberg-rust/pull/318 This small PR has been broken out of https://github.com/apache/iceberg-rust/pull/241 as that PR was getting too large. It should be pretty uncontroversial, adding some getters that are required

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

amogh-jahagirdar merged PR #9953: URL: https://github.com/apache/iceberg/pull/9953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

amogh-jahagirdar commented on PR #9953: URL: https://github.com/apache/iceberg/pull/9953#issuecomment-2038104819 Thanks for the reviews @ajantha-bhat and @Fokko ! Merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Add Struct Accessors to BoundReferences [iceberg-rust]

sdd opened a new pull request, #317: URL: https://github.com/apache/iceberg-rust/pull/317 First PR to come out of breaking up https://github.com/apache/iceberg-rust/pull/241. Adds `StructAccessor`, which is added to `BoundReference` as a means of retrieving a field's value from a (po

Re: [PR] REST: fix spurious warning when shutting down refresh executor [iceberg]

amogh-jahagirdar commented on PR #10087: URL: https://github.com/apache/iceberg/pull/10087#issuecomment-2038064291 Thanks for fixing this @adutra! Merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] REST: fix spurious warning when shutting down refresh executor [iceberg]

amogh-jahagirdar merged PR #10087: URL: https://github.com/apache/iceberg/pull/10087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

Fokko commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1552302617 ## pyiceberg/manifest.py: ## @@ -283,31 +277,12 @@ def __repr__(self) -> str: } -@singledispatch -def partition_field_to_data_file_partition_field(partition_fie

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

amogh-jahagirdar commented on code in PR #9953: URL: https://github.com/apache/iceberg/pull/9953#discussion_r1552280909 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestS3FileIO.java: ## @@ -377,6 +384,50 @@ public void testResolvingFileIOLoad() { Assertions.assertThat(re

Re: [PR] Add entries metadata table [iceberg-python]

Fokko merged PR #551: URL: https://github.com/apache/iceberg-python/pull/551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add entries metadata table [iceberg-python]

Fokko commented on PR #551: URL: https://github.com/apache/iceberg-python/pull/551#issuecomment-2037933309 Good one @HonahX, just added it 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Add samples for listing table.files and table.partitions [iceberg-python]

moryachok commented on issue #578: URL: https://github.com/apache/iceberg-python/issues/578#issuecomment-2037890575 Hey @Fokko, that would be awesome, let me know when it merged and I will test it on my side. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Support partial overwrites [iceberg-python]

Fokko commented on issue #268: URL: https://github.com/apache/iceberg-python/issues/268#issuecomment-2037867614 Closing this since it is a duplicate of #511. A partial overwrite is essentially a delete + append. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Support partial overwrites [iceberg-python]

Fokko closed issue #268: Support partial overwrites URL: https://github.com/apache/iceberg-python/issues/268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Support partial overwrites [iceberg-python]

Fokko commented on issue #268: URL: https://github.com/apache/iceberg-python/issues/268#issuecomment-2037866632 @whynick1 This should be done once https://github.com/apache/iceberg-python/issues/554 is in. Would you like to provide one of the metadata tables? See https://github.com/apache/

Re: [I] Add samples for listing table.files and table.partitions [iceberg-python]

Fokko commented on issue #578: URL: https://github.com/apache/iceberg-python/issues/578#issuecomment-2037863395 @moryachok It looks like someone else is already working on files, how about partitions? I'll merge https://github.com/apache/iceberg-python/pull/551 later today and then it shoul

Re: [PR] Parquet: Make row-group filters cooperate to filter [iceberg]

cccs-jc commented on PR #6893: URL: https://github.com/apache/iceberg/pull/6893#issuecomment-2037778990 It would be great to revive your PR. I think it's the best approach and it's a major improvement over the current implementation. The query speed is much faster with this fix. I ha

Re: [PR] Docs: Add gc.enabled property [iceberg]

nk1506 closed pull request #9231: Docs: Add gc.enabled property URL: https://github.com/apache/iceberg/pull/9231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: Add gc.enabled property [iceberg]

nk1506 commented on PR #9231: URL: https://github.com/apache/iceberg/pull/9231#issuecomment-2037765782 Closing this as this might nor required on doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 commented on PR #9682: URL: https://github.com/apache/iceberg/pull/9682#issuecomment-2037762988 Common code abstraction on Hive-Table/View has been parked for later. BaseMetadata for Table and View might not be required. We can fulfil the same using different strategy. -- This is

Re: [PR] Core: Common metadata for TableMetadata and ViewMetadata [iceberg]

nk1506 closed pull request #9682: Core: Common metadata for TableMetadata and ViewMetadata URL: https://github.com/apache/iceberg/pull/9682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Support partial overwrites [iceberg-python]

whynick1 commented on issue #268: URL: https://github.com/apache/iceberg-python/issues/268#issuecomment-2037755644 @Fokko if nobody is already looking at this, would love to try and take a stab at this? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nk1506 closed pull request #8907: Hive: Add View support for HIVE catalog URL: https://github.com/apache/iceberg/pull/8907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nk1506 commented on PR #8907: URL: https://github.com/apache/iceberg/pull/8907#issuecomment-2037749273 Closing this PR as same has been addressed with ongoing [PR](https://github.com/apache/iceberg/pull/9852) . -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Docs: Document support for binary in truncate transform [iceberg]

TheNeuralBit commented on code in PR #10079: URL: https://github.com/apache/iceberg/pull/10079#discussion_r1552085818 ## format/spec.md: ## @@ -245,19 +245,19 @@ For example, a file may be written with schema `1: a int, 2: b string, 3: c doub Tables may also define a propert

[PR] Remove `record_fields` from the `Record` class [iceberg-python]

Fokko opened a new pull request, #580: URL: https://github.com/apache/iceberg-python/pull/580 First step towards https://github.com/apache/iceberg-python/issues/579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[I] Refactor the `Record` by removing the schema [iceberg-python]

Fokko opened a new issue, #579: URL: https://github.com/apache/iceberg-python/issues/579 ### Feature Request / Improvement The `Record` class in Typedef should not carry a schema as it does now. This is mostly for testing purposes, but it is very tempting to rely on this schema, whil

Re: [PR] Fix `rewrite_position_delete_files` result file set [iceberg]

bk-mz commented on PR #9945: URL: https://github.com/apache/iceberg/pull/9945#issuecomment-2036840825 @nastra hey, yes, you are correct. sorry for that. unfortunately I can't setup proper infrastructure to ensure this test requirements. I've found this bug in prod during produ

Re: [PR] Hive, JDBC: Add null check before matching the error message [iceberg]

nastra commented on code in PR #10082: URL: https://github.com/apache/iceberg/pull/10082#discussion_r1551437288 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcTableOperations.java: ## @@ -138,7 +138,7 @@ public void doCommit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Hive, JDBC: Add null check before matching the error message [iceberg]

nastra commented on code in PR #10082: URL: https://github.com/apache/iceberg/pull/10082#discussion_r1551398196 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java: ## @@ -59,29 +59,19 @@ public HiveClientPool(int poolSize, Configuration conf) { @Overr

Re: [PR] Hive, JDBC: Add null check before matching the error message [iceberg]

nk1506 commented on code in PR #10082: URL: https://github.com/apache/iceberg/pull/10082#discussion_r1551350882 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java: ## @@ -59,29 +59,19 @@ public HiveClientPool(int poolSize, Configuration conf) { @Overr

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

carlosescura commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2036615576 @rahil-c is there any possibility to continue working on this PR? Many of us would really appreciate it. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Hive, JDBC: Add null check before matching the error message [iceberg]

nastra commented on code in PR #10082: URL: https://github.com/apache/iceberg/pull/10082#discussion_r1551269821 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java: ## @@ -59,29 +59,19 @@ public HiveClientPool(int poolSize, Configuration conf) { @Overr

Re: [I] Add samples for listing table.files and table.partitions [iceberg-python]

Fokko commented on issue #578: URL: https://github.com/apache/iceberg-python/issues/578#issuecomment-2036498291 Hey @moryachok Thanks for raising this. This is currently being worked on in https://github.com/apache/iceberg-python/issues/511 The files one should be rather straightforw

Re: [PR] Move writes to Transaction [iceberg-python]

Fokko commented on PR #571: URL: https://github.com/apache/iceberg-python/pull/571#issuecomment-2036374729 Thanks for working on this @syun64 and @HonahX for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Move writes to Transaction [iceberg-python]

Fokko merged PR #571: URL: https://github.com/apache/iceberg-python/pull/571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Move writes to Transaction [iceberg-python]

Fokko commented on PR #571: URL: https://github.com/apache/iceberg-python/pull/571#issuecomment-2036373338 The methods on the table were added as shorthands indeed. I'm open to removing those methods and letting everything go through a transaction. This way we have a single way to do things