Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560470048 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List

Re: [I] ACL when using DynamoDb based Catalog [iceberg]

2024-04-10 Thread via GitHub
munendrasn closed issue #6763: ACL when using DynamoDb based Catalog URL: https://github.com/apache/iceberg/issues/6763 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] AWS: Changes to DynamoDb Catalog to support ACL on Namespace Entries [iceberg]

2024-04-10 Thread via GitHub
munendrasn closed pull request #6862: AWS: Changes to DynamoDb Catalog to support ACL on Namespace Entries URL: https://github.com/apache/iceberg/pull/6862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
ajantha-bhat commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2048960308 @dimas-b : catalog created by spark and catalog created by the user is using the same same `JavaHttpClient` object? Isn't the culprit is the `Static` class

Re: [I] Implement `Closable` interface for class `HiveCatalog` and `HiveClientPool` [iceberg]

2024-04-10 Thread via GitHub
pvary commented on issue #10100: URL: https://github.com/apache/iceberg/issues/10100#issuecomment-2048947358 @yuqi1129: Caffeine suggests using Java9 system scheduler if possible. https://github.com/ben-manes/caffeine/wiki/Cleanup Would it work for you, or the specific classpath

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-10 Thread via GitHub
pvary commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1560439436 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/TestChangeLogTable.java: ## @@ -98,7 +98,7 @@ public void before() { @Override public void clean() {

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-10 Thread via GitHub
pvary commented on code in PR #10112: URL: https://github.com/apache/iceberg/pull/10112#discussion_r1560437038 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/FlinkTestBase.java: ## @@ -126,4 +126,18 @@ protected void dropCatalog(String catalogName, boolean

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-10 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1560434336 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,386 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-10 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1560433436 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,386 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-10 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1560432153 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,357 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-10 Thread via GitHub
kevinjqliu commented on PR #590: URL: https://github.com/apache/iceberg-python/pull/590#issuecomment-2048858719 cc @Fokko / @HonahX -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [BUG] Valid column characters fail on to_arrow() or to_pandas() ArrowInvalid: No match for FieldRef.Name [iceberg-python]

2024-04-10 Thread via GitHub
kevinjqliu commented on issue #584: URL: https://github.com/apache/iceberg-python/issues/584#issuecomment-2048853777 I've added [apache/iceberg/#10120](https://github.com/apache/iceberg/issues/10120) as a feature request to allow writing column names with special characters. I've

Re: [PR] Sanitized special character column name before writing to parquet [iceberg-python]

2024-04-10 Thread via GitHub
kevinjqliu commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1560381631 ## pyiceberg/io/pyarrow.py: ## @@ -1772,12 +1772,13 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT ) def

[I] [feature request] Allow Java Iceberg library to write parquet files with special character column names [iceberg]

2024-04-10 Thread via GitHub
kevinjqliu opened a new issue, #10120: URL: https://github.com/apache/iceberg/issues/10120 ### Feature Request / Improvement Based on discussions from [iceberg-python/#584](https://github.com/apache/iceberg-python/issues/584), we found that the Java Iceberg library "sanitizes" and

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
Xuanwo commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1560378281 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1560377462 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Sanitized special character column before writing to parquet [iceberg-python]

2024-04-10 Thread via GitHub
kevinjqliu commented on code in PR #590: URL: https://github.com/apache/iceberg-python/pull/590#discussion_r1560374756 ## column_name_test.py: ## @@ -0,0 +1,87 @@ +import os.path + +from pyiceberg.catalog.sql import SqlCatalog +import pyarrow as pa +import pandas as pd + + +def

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-10 Thread via GitHub
vteja11 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1560368628 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,357 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Spark-3.5: Support CTAS and RTAS to preserve schema nullability. [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on code in PR #10074: URL: https://github.com/apache/iceberg/pull/10074#discussion_r1560322347 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkCatalogOperations.java: ## @@ -34,6 +38,41 @@ public class TestSparkCatalogOperations

Re: [PR] Spark-3.5: Support CTAS and RTAS to preserve schema nullability. [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on code in PR #10074: URL: https://github.com/apache/iceberg/pull/10074#discussion_r1560320292 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/BaseCatalog.java: ## @@ -21,19 +21,31 @@ import org.apache.iceberg.spark.procedures.SparkProcedures;

Re: [PR] Spark-3.5: Support CTAS and RTAS to preserve schema nullability. [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on code in PR #10074: URL: https://github.com/apache/iceberg/pull/10074#discussion_r1560282149 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/BaseCatalog.java: ## @@ -21,19 +21,31 @@ import org.apache.iceberg.spark.procedures.SparkProcedures;

Re: [PR] Docs: Fix inconsistency in branching and tagging scenario [iceberg]

2024-04-10 Thread via GitHub
bitsondatadev commented on code in PR #9968: URL: https://github.com/apache/iceberg/pull/9968#discussion_r1560201975 ## docs/docs/branching.md: ## @@ -49,20 +49,21 @@ Tags can be used for retaining important historical snapshots for auditing purpo The above diagram

Re: [I] Add support for TimeType, ListType, MapType and StructType in the ArrowReader [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] commented on issue #2485: URL: https://github.com/apache/iceberg/issues/2485#issuecomment-2048630301 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Flink : add watermark support for streaming read [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] closed issue #2264: Flink : add watermark support for streaming read URL: https://github.com/apache/iceberg/issues/2264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Flink : add Computed Column support for iceberg [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] closed issue #2262: Flink : add Computed Column support for iceberg URL: https://github.com/apache/iceberg/issues/2262 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Add an action to compact delete files [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] closed issue #1027: Add an action to compact delete files URL: https://github.com/apache/iceberg/issues/1027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Flink : add watermark support for streaming read [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] commented on issue #2264: URL: https://github.com/apache/iceberg/issues/2264#issuecomment-2048630149 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add an action to compact delete files [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] commented on issue #1027: URL: https://github.com/apache/iceberg/issues/1027#issuecomment-2048630123 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Flink : add Computed Column support for iceberg [iceberg]

2024-04-10 Thread via GitHub
github-actions[bot] commented on issue #2262: URL: https://github.com/apache/iceberg/issues/2262#issuecomment-2048630136 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560187855 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -36,7 +36,7 @@ public class TestListTablesResponse extends

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560187545 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -34,7 +34,7 @@ public class TestListNamespacesResponse extends

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560180427 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -68,7 +68,7 @@ import org.junit.jupiter.api.Test; public abstract class CatalogTests { -

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
pseudomuto commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2048599779 Well, that worked like a charm. That is indeed the issue. I can't thank you guys enough! I'd like to give something back for this. Can I take a stab at

Re: [PR] Spark 3.4: Fix system function pushdown in CoW row-level commands [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on PR #10119: URL: https://github.com/apache/iceberg/pull/10119#issuecomment-2048595286 I excluded the unnecessary branch in `ReplaceStaticInvoke` and adapted the tests. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Core: Allow manifest file cache to be configurable [iceberg]

2024-04-10 Thread via GitHub
singhpk234 commented on PR #10118: URL: https://github.com/apache/iceberg/pull/10118#issuecomment-2048575243 interesting, whats the use case for this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
pseudomuto commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2048568071 Hmm...I did notice that when I wrote it. My assumption was that the SparkActions would be creating tasks that would instantiate their own instance of the catalog via Spark

Re: [I] [BUG] Valid column characters fail on to_arrow() or to_pandas() ArrowInvalid: No match for FieldRef.Name [iceberg-python]

2024-04-10 Thread via GitHub
gwindes commented on issue #584: URL: https://github.com/apache/iceberg-python/issues/584#issuecomment-2048558338 @kevinjqliu @Fokko I verified that the fix from @kevinjqliu does indeed fix the pyarrow issue I was seeing with the column names I have. Attached some photos of the

Re: [PR] Spark 3.4: Fix system function pushdown in CoW row-level commands [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on PR #10119: URL: https://github.com/apache/iceberg/pull/10119#issuecomment-2048536387 I forgot 3.4 is a bit different, I may need to make some changes. Let me see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2024-04-10 Thread via GitHub
aokolnychyi commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-2048532659 Sorry for the delay, @tmnd1991. The last several months were really busy. I think a simple cherry-pick to 3.4 should be enough. The original change was for 3.5. Could you take a look

Re: [PR] Flink: Adds support for Flink 1.19 version [iceberg]

2024-04-10 Thread via GitHub
rodmeneses commented on PR #10112: URL: https://github.com/apache/iceberg/pull/10112#issuecomment-2048507129 cc: @pvary @stevenzwu @mas-chen please take a look at your earliest convenience. Thanks -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560093868 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,39 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
dimas-b commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2048477509 Well, that `try (var catalog = new NessieCatalog()) {...}` block _will_ close the Nessie client associated with the table that is loaded from this catalog. I believe the

Re: [PR] preserve field order in `Schema.select` [iceberg-python]

2024-04-10 Thread via GitHub
felixscherz closed pull request #242: preserve field order in `Schema.select` URL: https://github.com/apache/iceberg-python/pull/242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560039876 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2330,46 @@ public void multipleDiffsAgainstMultipleTablesLastFails() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560038289 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2330,46 @@ public void multipleDiffsAgainstMultipleTablesLastFails() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1560030414 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -224,6 +226,7 @@ public void initialize(String name, Map unresolved) {

[PR] Core: Allow manifest file cache to be configurable [iceberg]

2024-04-10 Thread via GitHub
tdcmeehan opened a new pull request, #10118: URL: https://github.com/apache/iceberg/pull/10118 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: Fix links of `Get Started` and `Community` sections in footer [iceberg]

2024-04-10 Thread via GitHub
Fokko merged PR #10098: URL: https://github.com/apache/iceberg/pull/10098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Spark: readStream from Iceberg doesn't progress anymore after running Maintenance (rewrite_data_files and rewrite_manifests) [iceberg]

2024-04-10 Thread via GitHub
singhpk234 commented on issue #10117: URL: https://github.com/apache/iceberg/issues/10117#issuecomment-2048372396 can you please try with this change : https://github.com/apache/iceberg/pull/8980 ? this is released in iceberg 1.5 -- This is an automated message from the Apache Git

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
pseudomuto commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2048366352 Sure thing! I've removed imports and a few things for brevity ```java private static final String NESSIE_PREFIX = "spark.sql.catalog.nessie."; Table

Re: [PR] StructType field `optional` by default [iceberg-python]

2024-04-10 Thread via GitHub
Fokko commented on code in PR #592: URL: https://github.com/apache/iceberg-python/pull/592#discussion_r1559924873 ## tests/test_types.py: ## @@ -560,13 +560,13 @@ def test_deserialization_struct() -> None: def test_str_struct(simple_struct: StructType) -> None: -assert

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559923936 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,97 @@ public void testFilterAndRemovePrefix() {

Re: [I] [feat request] Add `metadata_log_entries` metadata table [iceberg-python]

2024-04-10 Thread via GitHub
Fokko commented on issue #594: URL: https://github.com/apache/iceberg-python/issues/594#issuecomment-2048232027 Thanks for creating this issue @kevinjqliu, I've also assigned you in https://github.com/apache/iceberg-python/issues/511 -- This is an automated message from the Apache Git

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
amogh-jahagirdar commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559919696 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,97 @@ public void testFilterAndRemovePrefix() {

Re: [I] Add metadata tables [iceberg-python]

2024-04-10 Thread via GitHub
syun64 commented on issue #511: URL: https://github.com/apache/iceberg-python/issues/511#issuecomment-2048227788 Hi @Fokko could I pick up the Partitions table? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1558712520 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559757955 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
pseudomuto commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2047986951 Just tried with matching server version (0.78.0). Same result unfortunately. ``` java.lang.NullPointerException: Cannot invoke

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
Xuanwo commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559754777 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559751858 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
Xuanwo commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559727594 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-04-10 Thread via GitHub
wgtmac commented on PR #9772: URL: https://github.com/apache/iceberg/pull/9772#issuecomment-2047925627 @nastra Gentle ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-10 Thread via GitHub
RussellSpitzer commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1559706583 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +237,42 @@ public void

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-10 Thread via GitHub
manuzhang commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1559705114 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +237,42 @@ public void

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559675925 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559672626 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
Xuanwo commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559671945 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1559655890 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -232,4 +237,42 @@ public void

Re: [I] manifest list missing error after commit failed exception [iceberg]

2024-04-10 Thread via GitHub
flisboac commented on issue #9406: URL: https://github.com/apache/iceberg/issues/9406#issuecomment-2047837859 I've also come across this error. The difference is tha0, AFAICT, I don't need to rollback the table when that error happens. ```text Traceback (most recent call last):

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559638330 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,114 @@ public void testFilterAndRemovePrefix() {

Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #10037: URL: https://github.com/apache/iceberg/pull/10037#discussion_r1559638434 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ProcedureUtil.java: ## @@ -51,4 +56,29 @@ static String statsFileLocation(String

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559627285 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,114 @@ public void testFilterAndRemovePrefix() {

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
dimas-b commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2047816727 In non-oauth2 modes, there's no log message on closing the client, unfortunately. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559623826 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -278,14 +286,26 @@ public void setConf(Object newConf) { @Override public List

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559618368 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-10 Thread via GitHub
ZENOTME commented on code in PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#discussion_r1559618368 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559612907 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -36,7 +36,7 @@ public class TestListTablesResponse extends

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1559609293 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559607741 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,39 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559611564 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListNamespacesResponse.java: ## @@ -34,7 +34,7 @@ public class TestListNamespacesResponse extends

[I] is:issue is:open spark readStream from Iceberg doesn't progress anymore after running Maintenance (rewrite_data_files and rewrite_manifests) [iceberg]

2024-04-10 Thread via GitHub
tenstriker opened a new issue, #10117: URL: https://github.com/apache/iceberg/issues/10117 ### Apache Iceberg version 1.4.3 ### Query engine Spark ### Please describe the bug  Spark version : 3.5_2.12 We have iceberg spark streaming read job reading

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559605425 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2330,46 @@ public void multipleDiffsAgainstMultipleTablesLastFails() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559608101 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +151,39 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559602390 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -68,7 +68,7 @@ import org.junit.jupiter.api.Test; public abstract class CatalogTests { -

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1559602987 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2330,46 @@ public void multipleDiffsAgainstMultipleTablesLastFails() {

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559603081 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,114 @@ public void testFilterAndRemovePrefix() {

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559581343 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,114 @@ public void testFilterAndRemovePrefix() {

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-04-10 Thread via GitHub
Fokko commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2047703495 @swapdewalkar Thanks for picking this up! I've just assigned it to you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-04-10 Thread via GitHub
swapdewalkar commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2047668021 I am trying to working on this, is it possible to assign it to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
pseudomuto commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2047655639 > These settings are not needed on the client (engine) side. We have Nessie running behind IAP in Google and the auth token is added to the spark properties before the

Re: [I] Remove `unwrap()` in `ManifestListWriter.close()` [iceberg-rust]

2024-04-10 Thread via GitHub
tabmatfournier commented on issue #177: URL: https://github.com/apache/iceberg-rust/issues/177#issuecomment-2047652334 Is this issue still valid given https://github.com/apache/iceberg-rust/pull/185 ? This seems addressed. Apologies for the noise, surfing good first issues for something

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
dimas-b commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2047587036 Side note @pseudomuto : > spark.sql.catalog.nessie.quarkus.oidc.auth-server-url=https://accounts.google.com > spark.sql.catalog.nessie.quarkus.oidc.client-id=

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-10 Thread via GitHub
Fokko commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1559451439 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An optional

Re: [I] NPE During RewriteDataFiles Action with Nessie [iceberg]

2024-04-10 Thread via GitHub
dimas-b commented on issue #10110: URL: https://github.com/apache/iceberg/issues/10110#issuecomment-2047483276 Not sure it is critical in this case, but Iceberg 1.4.3 [includes](https://github.com/apache/iceberg/blob/apache-iceberg-1.4.3/gradle/libs.versions.toml#L50) Nessie Client 0.71.0,

Re: [PR] Open-api: update prefix param description [iceberg]

2024-04-10 Thread via GitHub
nastra commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1559391038 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An optional

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
ajantha-bhat commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559372303 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -54,7 +54,31 @@ enum SchemaVersion { static final String TABLE_RECORD_TYPE = "TABLE";

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
ajantha-bhat commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559369352 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,97 @@ public void testFilterAndRemovePrefix() {

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559369000 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -54,7 +54,31 @@ enum SchemaVersion { static final String TABLE_RECORD_TYPE = "TABLE";

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559367373 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,97 @@ public void testFilterAndRemovePrefix() {

Re: [PR] Core: Fix JDBC Catalog table commit when migrating from schema V0 to V1 [iceberg]

2024-04-10 Thread via GitHub
jbonofre commented on code in PR #10111: URL: https://github.com/apache/iceberg/pull/10111#discussion_r1559367008 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +51,97 @@ public void testFilterAndRemovePrefix() {

  1   2   >