Re: [PR] Spark: Added merge schema as spark configuration [iceberg]

manuzhang commented on PR #9397: URL: https://github.com/apache/iceberg/pull/9397#issuecomment-1873675429 @Aleena-M-Georgy you need to propose your change against latest `spark/v3.5`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Core, Spark 3.3: Add FileRewriter API [iceberg]

manuzhang commented on code in PR #7175: URL: https://github.com/apache/iceberg/pull/7175#discussion_r1439159500 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Build: Bump actions/labeler from 4 to 5 [iceberg]

panbingkun commented on PR #9331: URL: https://github.com/apache/iceberg/pull/9331#issuecomment-1873595594 Friendly ping @nastra, can you take some time to help review this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Spark 3.5: Support filtering with buckets in RewriteDataFilesProcedure [iceberg]

manuzhang opened a new pull request, #9396: URL: https://github.com/apache/iceberg/pull/9396 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Correct schema behavior [iceberg-python]

amogh-jahagirdar commented on code in PR #247: URL: https://github.com/apache/iceberg-python/pull/247#discussion_r1439114924 ## pyiceberg/table/__init__.py: ## @@ -942,15 +942,16 @@ def snapshot(self) -> Optional[Snapshot]: return self.table.current_snapshot() de

Re: [PR] Allow filtering on newly added columns [iceberg-python]

amogh-jahagirdar commented on code in PR #246: URL: https://github.com/apache/iceberg-python/pull/246#discussion_r1439112448 ## pyiceberg/expressions/visitors.py: ## @@ -906,7 +906,16 @@ def visit_bound_predicate(self, predicate: BoundPredicate[L]) -> BooleanExpressi def t

Re: [I] stream api how to only update a column ? [iceberg]

github-actions[bot] commented on issue #6901: URL: https://github.com/apache/iceberg/issues/6901#issuecomment-1873537298 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] when use beeline read iceberg table on HDFS, cast error [iceberg]

github-actions[bot] closed issue #6918: when use beeline read iceberg table on HDFS, cast error URL: https://github.com/apache/iceberg/issues/6918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] when use beeline read iceberg table on HDFS, cast error [iceberg]

github-actions[bot] commented on issue #6918: URL: https://github.com/apache/iceberg/issues/6918#issuecomment-1873537291 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] 数据合并 [iceberg]

github-actions[bot] commented on issue #7092: URL: https://github.com/apache/iceberg/issues/7092#issuecomment-1873537276 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] 数据合并 [iceberg]

github-actions[bot] closed issue #7092: 数据合并 URL: https://github.com/apache/iceberg/issues/7092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-u

[PR] Build: Bump pytest from 7.4.3 to 7.4.4 [iceberg-python]

dependabot[bot] opened a new pull request, #248: URL: https://github.com/apache/iceberg-python/pull/248 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.3 to 7.4.4. Release notes Sourced from https://github.com/pytest-dev/pytest/releases";>pytest's releases. pyt

[PR] Correct schema behavior [iceberg-python]

Fokko opened a new pull request, #247: URL: https://github.com/apache/iceberg-python/pull/247 When we alter the schema, we want to use the latest schema by default, except when you select a specific snapshot that has a schema-id. -- This is an automated message from the Apache Git Service

Re: [PR] chore(deps): Update opendal requirement from 0.43 to 0.44 [iceberg-rust]

Fokko merged PR #142: URL: https://github.com/apache/iceberg-rust/pull/142 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] chore(deps): Bump peaceiris/actions-gh-pages from 3.9.2 to 3.9.3 [iceberg-rust]

Fokko merged PR #143: URL: https://github.com/apache/iceberg-rust/pull/143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Data: Support reading default values from generic Avro readers [iceberg]

rdblue commented on PR #6004: URL: https://github.com/apache/iceberg/pull/6004#issuecomment-1873442005 Now that #9366 is in, I think we should perform a similar refactor for the Iceberg generics and add default support that way. It should be much simpler! -- This is an automated message f

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on PR #9366: URL: https://github.com/apache/iceberg/pull/9366#issuecomment-1873441376 Thanks for reviewing, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue merged PR #9366: URL: https://github.com/apache/iceberg/pull/9366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] Allow filtering on newly added columns [iceberg-python]

Fokko opened a new pull request, #246: URL: https://github.com/apache/iceberg-python/pull/246 Resolves #217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Glue catalog commit table [iceberg-python]

Fokko commented on PR #140: URL: https://github.com/apache/iceberg-python/pull/140#issuecomment-1873429589 Thanks @HonahX for working on this, and thanks @pdames, @nicor88, and @jackye1995 for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Glue catalog commit table [iceberg-python]

Fokko merged PR #140: URL: https://github.com/apache/iceberg-python/pull/140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Implement Glue Table commit [iceberg-python]

Fokko closed issue #29: Implement Glue Table commit URL: https://github.com/apache/iceberg-python/issues/29 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

Fokko commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439066996 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r143909 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [I] bug: The current snapshot with id 0 will be skip. [iceberg-python]

MehulBatra commented on issue #232: URL: https://github.com/apache/iceberg-python/issues/232#issuecomment-1873421764 I will take care of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439066126 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064941 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader i

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439065448 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -69,62 +99,107 @@ public void setRowPositionSupplier(Supplier posSupplier) { @Override

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064941 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader i

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064577 ## core/src/main/java/org/apache/iceberg/avro/Avro.java: ## @@ -683,23 +698,34 @@ public ReadBuilder classLoader(ClassLoader classLoader) { return this; }

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on PR #9384: URL: https://github.com/apache/iceberg/pull/9384#issuecomment-1873414556 Looks good overall. Thanks for adding this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439063090 ## core/src/main/java/org/apache/iceberg/io/ClusteredPositionDeleteWriter.java: ## @@ -46,17 +49,39 @@ public ClusteredPositionDeleteWriter( OutputFileFactory fil

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439063090 ## core/src/main/java/org/apache/iceberg/io/ClusteredPositionDeleteWriter.java: ## @@ -46,17 +49,39 @@ public ClusteredPositionDeleteWriter( OutputFileFactory fil

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439062877 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439062877 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439061281 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -71,14 +83,44 @@ public DeleteWriteResult result() { @Override public

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439062562 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439062436 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439061554 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -93,9 +135,9 @@ private DeleteWriteResult writeDeletes() throws IOException

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439061281 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -71,14 +83,44 @@ public DeleteWriteResult result() { @Override public

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439060980 ## core/src/main/java/org/apache/iceberg/deletes/SortingPositionOnlyDeleteWriter.java: ## @@ -60,7 +72,7 @@ public void write(PositionDelete positionDelete) { @Over

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439060046 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "histor

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439059974 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "histor

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

rdblue commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1439059745 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -708,4 +709,15 @@ private long sparkAdvisoryPartitionSize() { private double shu

Re: [PR] Glue catalog commit table [iceberg-python]

HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1438999074 ## tests/conftest.py: ## @@ -1586,7 +1586,7 @@ def fixture_aws_credentials() -> Generator[None, None, None]: os.environ.pop("AWS_DEFAULT_REGION") -MOTO_SE

Re: [PR] Glue catalog commit table [iceberg-python]

HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1438987608 ## pyiceberg/catalog/__init__.py: ## @@ -587,8 +590,34 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> ToOutputFile.tab