Re: [I] Correct names in the ManifestList [iceberg-rust]

liurenjie1024 commented on issue #354: URL: https://github.com/apache/iceberg-rust/issues/354#issuecomment-2078683994 > @Fokko, can you please assign this to me? Thanks! Done, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580510837 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580506460 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580505826 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 commented on PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#issuecomment-2078666082 Thanks @sdd for this pr, and @Xuanwo @marvinlanhenke for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 merged PR #323: URL: https://github.com/apache/iceberg-rust/pull/323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add Pagination To List Apis [iceberg]

rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1580469059 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Add Pagination To List Apis [iceberg]

rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1580469059 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1580446821 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1580444775 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078535487 Ok this is reproducing via the github actions build on my public fork https://github.com/jkolash/iceberg/actions/runs/8842101257/job/24280206652 ``` TestDataFrameWriterV2 >

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 commented on PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#issuecomment-2078490131 Let's wait a moment to see if others have comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580334216 ## crates/iceberg/src/scan.rs: ## @@ -158,8 +196,24 @@ impl TableScan { .await?; // Generate data file stream -let mut

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1580294047 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/BaseDeltaTaskWriter.java: ## @@ -56,7 +58,8 @@ abstract class BaseDeltaTaskWriter extends

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on PR #10200: URL: https://github.com/apache/iceberg/pull/10200#issuecomment-2078403034 > Also there is a behavioural change that the previous write rolled deletes @pvary, I don't think that threshold makes sense in this scenario. It was needed because we kept

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1580291026 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -109,18 +112,34 @@ protected abstract class BaseEqualityDeltaWriter implements Closeable

[PR] [draft] Parquet page skipping (using filtered row groups) [iceberg]

wypoon opened a new pull request, #10228: URL: https://github.com/apache/iceberg/pull/10228 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Hive documentation not updated on website [iceberg]

github-actions[bot] commented on issue #2715: URL: https://github.com/apache/iceberg/issues/2715#issuecomment-2078371964 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Add support for TimeType, ListType, MapType and StructType in the ArrowReader [iceberg]

github-actions[bot] commented on issue #2485: URL: https://github.com/apache/iceberg/issues/2485#issuecomment-2078371768 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add support for TimeType, ListType, MapType and StructType in the ArrowReader [iceberg]

github-actions[bot] closed issue #2485: Add support for TimeType, ListType, MapType and StructType in the ArrowReader URL: https://github.com/apache/iceberg/issues/2485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Build: Bump getdaft from 0.2.16 to 0.2.21 [iceberg-python]

dependabot[bot] opened a new pull request, #662: URL: https://github.com/apache/iceberg-python/pull/662 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.16 to 0.2.21. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases;>getdaft's releases.

[PR] Build: Bump griffe from 0.40.1 to 0.44.0 [iceberg-python]

dependabot[bot] opened a new pull request, #660: URL: https://github.com/apache/iceberg-python/pull/660 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 0.40.1 to 0.44.0. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases;>griffe's releases.

[PR] Build: Bump pyspark from 3.5.0 to 3.5.1 [iceberg-python]

dependabot[bot] opened a new pull request, #659: URL: https://github.com/apache/iceberg-python/pull/659 Bumps [pyspark](https://github.com/apache/spark) from 3.5.0 to 3.5.1. Commits https://github.com/apache/spark/commit/fd86f85e181fc2dc0f50a096855acf83a6cc5d9c;>fd86f85

[PR] Build: Bump mkdocstrings from 0.24.0 to 0.24.3 [iceberg-python]

dependabot[bot] opened a new pull request, #661: URL: https://github.com/apache/iceberg-python/pull/661 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.24.0 to 0.24.3. Release notes Sourced from

[I] How to reinitialize spark catalog on an ongoing spark session with iceberg [iceberg]

vasudhamsr opened a new issue, #10227: URL: https://github.com/apache/iceberg/issues/10227 ### Query engine Spark ### Question We have a scenario where the catalog object keeps changing and the custom FileIO underneath as well. But the loadTable function

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

stevenzwu commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1580202486 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -130,7 +131,9 @@ private ScanContext( this.watermarkColumn =

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

stevenzwu commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1580202486 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -130,7 +131,9 @@ private ScanContext( this.watermarkColumn =

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

stevenzwu commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1580188262 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[I] AWS: Creating a Glue table with Lake Formation enabled fails [iceberg]

nickdelnano opened a new issue, #10226: URL: https://github.com/apache/iceberg/issues/10226 ### Apache Iceberg version 1.4.3 ### Query engine Flink ### Please describe the bug  Hi, I am observing issues when creating an Iceberg table with a Glue

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078200948 hmm I think this may be related to the spark version we are using as I tested on spark-3.4.1 and didn't see the issue but see it on our 3.4.2 -- This is an automated message

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078157080 If/when there is a PR I can test it on my side. where I have exhaustive type testing. ``` java.lang.ClassCastException: class java.lang.Byte cannot be cast to class

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078150463 ```kotlin val df = spark.sql("""select inline(array(from_json('{"b":82}', 'struct')))""") df.show() ``` ``` +---+ | b| +---+ | 82|

Re: [PR] Core: Avro writers use BlockingBinaryEncoder to enable array/map size calculations. [iceberg]

Fokko commented on PR #8625: URL: https://github.com/apache/iceberg/pull/8625#issuecomment-2078143286 Yes, this is still top of mind! I'm going to see what's needed and make sure that it will be included in the next Avro release! -- This is an automated message from the Apache Git

Re: [PR] Build: Test Apache Avro 1.11.2 [iceberg]

Fokko closed pull request #7922: Build: Test Apache Avro 1.11.2 URL: https://github.com/apache/iceberg/pull/7922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

wolfeidau commented on PR #59: URL: https://github.com/apache/iceberg-go/pull/59#issuecomment-2078141567 @zeroshade I will dig into this in the next few days, I have implemented the builder for tables but need a way to have a generic path builder. Given I have time off next week I

Re: [PR] Support partial deletes [iceberg-python]

MehulBatra commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1580084812 ## pyiceberg/table/__init__.py: ## @@ -235,6 +242,10 @@ class TableProperties: WRITE_PARTITION_SUMMARY_LIMIT = "write.summary.partition-limit"

Re: [PR] Support partial deletes [iceberg-python]

MehulBatra commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1580084812 ## pyiceberg/table/__init__.py: ## @@ -235,6 +242,10 @@ class TableProperties: WRITE_PARTITION_SUMMARY_LIMIT = "write.summary.partition-limit"

Re: [PR] Support partial deletes [iceberg-python]

MehulBatra commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1580084812 ## pyiceberg/table/__init__.py: ## @@ -235,6 +242,10 @@ class TableProperties: WRITE_PARTITION_SUMMARY_LIMIT = "write.summary.partition-limit"

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078099689 @Fokko Thanks for the quick response I will try to write up a code snippet reproducing the issue. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

Fokko commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078097905 Hey @jkolash Thanks for reporting this. The behavior should stay the same, due to the logic here:

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580049448 ## crates/iceberg/src/scan.rs: ## @@ -158,8 +196,24 @@ impl TableScan { .await?; // Generate data file stream -let mut entries =

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580039092 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +240,25 @@ impl TableScan { .boxed()) } +fn create_manifest_evaluator( +id: i32, +

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1580038792 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +240,25 @@ impl TableScan { .boxed()) } +fn create_manifest_evaluator( +id: i32, +

[I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash opened a new issue, #10225: URL: https://github.com/apache/iceberg/issues/10225 ### Apache Iceberg version 1.5.0 ### Query engine Spark ### Please describe the bug  The removal of the code ```java private static PrimitiveWriter

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580024955 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1580019899 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579945951 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579940862 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579930294 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579919751 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579897701 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579883806 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

stevenzwu commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1579871219 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -40,24 +44,27 @@ import org.apache.iceberg.FileFormat;

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2077790169 > @epgif can you please address the test failures? Done. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

stevenzwu commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1579860068 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579827548 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

tshauck commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579818649 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2077663647 > I've left some comment to improve, but it looks great! I'll invite datafusion community to help review. @liurenjie1024 Thanks for the review. I fixed most of the

Re: [PR] Update 1.5.1 release notes [iceberg]

nastra merged PR #10224: URL: https://github.com/apache/iceberg/pull/10224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Update 1.5.1 release notes [iceberg]

nastra commented on PR #10224: URL: https://github.com/apache/iceberg/pull/10224#issuecomment-2077658431 LGTM, thanks @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579755826 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

viirya commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2077637850 Thanks @liurenjie1024. The roadmaps doc looks good to me. I added a few items under DataFusion integration. Feel free to modify it. Thanks. -- This is an automated message from

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

Xuanwo commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579739285 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [PR] Update 1.5.1 release notes [iceberg]

amogh-jahagirdar commented on code in PR #10224: URL: https://github.com/apache/iceberg/pull/10224#discussion_r1579724014 ## site/docs/releases.md: ## @@ -73,6 +73,21 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.1 release

Re: [PR] Update 1.5.1 release notes [iceberg]

nastra commented on code in PR #10224: URL: https://github.com/apache/iceberg/pull/10224#discussion_r1579716696 ## site/docs/releases.md: ## @@ -73,6 +73,21 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.1 release +Apache

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579654225 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579601294 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Sql catalog [iceberg-rust]

liurenjie1024 commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1579259139 ## crates/catalog/sql/Cargo.toml: ## @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] AWS: Fix TestGlueCatalogTable#testCreateTable [iceberg]

aajisaka commented on PR #10221: URL: https://github.com/apache/iceberg/pull/10221#issuecomment-2077332098 Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579520268 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579472707 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

bryanck commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1579467601 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/IcebergWriterFactoryTest.java: ## @@ -83,4 +90,26 @@ public void

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579466191 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [I] `field-id`'s missing in generated Avro files [iceberg-rust]

Fokko commented on issue #353: URL: https://github.com/apache/iceberg-rust/issues/353#issuecomment-2077166551 @s-akhtar-baig Certainly  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Correct names in the ManifestList [iceberg-rust]

s-akhtar-baig commented on issue #354: URL: https://github.com/apache/iceberg-rust/issues/354#issuecomment-2077159735 @Fokko, can you please assign this to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] `field-id`'s missing in generated Avro files [iceberg-rust]

s-akhtar-baig commented on issue #353: URL: https://github.com/apache/iceberg-rust/issues/353#issuecomment-2077159481 @Fokko, can you please assign this to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579449633 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579449633 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

[I] `field-id`'s missing in generated Avro files [iceberg-rust]

Fokko opened a new issue, #353: URL: https://github.com/apache/iceberg-rust/issues/353 Looks like the `field-id` is missing in the Avro metadata generated by iceberg-rust. This should conform the spec: https://iceberg.apache.org/spec/#avro ``` avro-tools getschema

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579426812 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579420662 ## crates/iceberg/src/transaction.rs: ## @@ -121,6 +166,270 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579412576 ## crates/iceberg/src/transaction.rs: ## @@ -95,6 +104,42 @@ impl<'a> Transaction<'a> { Ok(self) } +/// Creates a fast append action. +pub fn

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579406252 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[I] Empty snapshot ID should be `Null` instead of `-1` [iceberg-rust]

Fokko opened a new issue, #352: URL: https://github.com/apache/iceberg-rust/issues/352 This is an old bug from Java. Where the Snapshot was set to -1 instead of `None`:

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579402269 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579401733 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579401183 ## crates/e2e_test/tests/append_data_file_test.rs: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [I] How to move Iceberg table from one location to another [iceberg]

cccs-jc commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2077081125 @ksmatharoo, thank you for providing your code to replace the paths within the iceberg metadata. I've implemented a similar solution using Python's fastavro to handle Avro

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579375646 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579372414 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579369755 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1579368639 ## crates/e2e_test/testdata/docker-compose.yaml: ## @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

nastra commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2077042155 @epgif can you please address the test failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1579326259 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

  1   2   >