Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1582017255 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1582017255 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] Build: Bump mkdocs-material from 9.5.18 to 9.5.19 [iceberg-python]

2024-04-27 Thread via GitHub
Fokko merged PR #663: URL: https://github.com/apache/iceberg-python/pull/663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Build: Bump datamodel-code-generator from 0.25.5 to 0.25.6 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10242: URL: https://github.com/apache/iceberg/pull/10242 Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.25.5 to 0.25.6. Release notes Sourced from

[PR] Build: Bump mkdocs-material from 9.5.18 to 9.5.19 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10241: URL: https://github.com/apache/iceberg/pull/10241 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.18 to 9.5.19. Release notes Sourced from

[PR] Build: Bump software.amazon.awssdk:bom from 2.25.35 to 2.25.40 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10240: URL: https://github.com/apache/iceberg/pull/10240 Bumps software.amazon.awssdk:bom from 2.25.35 to 2.25.40. [![Dependabot compatibility

[PR] Build: Bump com.gorylenko.gradle-git-properties:gradle-git-properties from 2.4.1 to 2.4.2 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10239: URL: https://github.com/apache/iceberg/pull/10239 Bumps com.gorylenko.gradle-git-properties:gradle-git-properties from 2.4.1 to 2.4.2. [![Dependabot compatibility

[PR] Build: Bump nessie from 0.79.0 to 0.80.0 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10237: URL: https://github.com/apache/iceberg/pull/10237 Bumps `nessie` from 0.79.0 to 0.80.0. Updates `org.projectnessie.nessie:nessie-client` from 0.79.0 to 0.80.0 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.22 to 1.2.23 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10238: URL: https://github.com/apache/iceberg/pull/10238 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.22 to 1.2.23. Release notes Sourced from

[PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.26.1 to 2.27.0 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10236: URL: https://github.com/apache/iceberg/pull/10236 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.26.1 to 2.27.0. Release notes Sourced from

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#issuecomment-2081289882 @liurenjie1024 Thanks for review. Sorry for late. I addressed the comments by rewriting the visitors using the new API. I replied with another questions. -- This is an automated

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581996145 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581995193 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994781 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994749 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994744 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [I] Spark SQL extension syntax for identifier field update [iceberg]

2024-04-27 Thread via GitHub
github-actions[bot] commented on issue #2724: URL: https://github.com/apache/iceberg/issues/2724#issuecomment-2081259388 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Support for low latency writes to iceberg table [iceberg]

2024-04-27 Thread via GitHub
github-actions[bot] commented on issue #2723: URL: https://github.com/apache/iceberg/issues/2723#issuecomment-2081259380 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] Clean backport with:Flink: Backport #10207 to v1.18 and v1.17 [iceberg]

2024-04-27 Thread via GitHub
pvary commented on PR #10235: URL: https://github.com/apache/iceberg/pull/10235#issuecomment-2081230888 Thanks for the review and the merge @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Build: Bump mkdocstrings from 0.24.0 to 0.24.3 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] closed pull request #661: Build: Bump mkdocstrings from 0.24.0 to 0.24.3 URL: https://github.com/apache/iceberg-python/pull/661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump mkdocstrings from 0.24.0 to 0.24.3 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] commented on PR #661: URL: https://github.com/apache/iceberg-python/pull/661#issuecomment-2081165993 Superseded by #664. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump griffe from 0.40.1 to 0.44.0 [iceberg-python]

2024-04-27 Thread via GitHub
Fokko merged PR #660: URL: https://github.com/apache/iceberg-python/pull/660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Build: Bump mkdocstrings from 0.24.0 to 0.25.0 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #664: URL: https://github.com/apache/iceberg-python/pull/664 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.24.0 to 0.25.0. Release notes Sourced from

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-27 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581362714 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866578 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866578 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866390 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,110 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' +

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1581860457 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -136,6 +137,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog

Re: [PR] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #10233: URL: https://github.com/apache/iceberg/pull/10233#discussion_r1581856455 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -166,23 +178,106 @@ public void deletePrefix(String prefix) { @Override public

Re: [PR] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #10233: URL: https://github.com/apache/iceberg/pull/10233#discussion_r1581856096 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -166,23 +178,106 @@ public void deletePrefix(String prefix) { @Override public

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-04-27 Thread via GitHub
yabola commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-2081026195 @sujithjay could you also take a look since you implements `startWith`~ Spark on parquet also support `endsWith` and `contains`

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851587 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851445 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -167,6 +168,67 @@ public void testInitialize() {

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851517 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851484 ## core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851465 ## core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581695111 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -56,26 +69,38 @@ public R run(Action action, boolean retry) throws E, InterruptedExc

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
kevinjqliu commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080922162 feel free to tag in the PR for review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
SebastianoMeneghin commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080910925 Sure, I can! If it makes sense for you, I will haha -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
kevinjqliu commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080872516 It should be a 1 line change, would you like to contribute? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-04-27 Thread via GitHub
bryanck commented on code in PR #10173: URL: https://github.com/apache/iceberg/pull/10173#discussion_r1581844369 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -28,6 +28,8 @@ import java.util.Set; import java.util.UUID;

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-04-27 Thread via GitHub
bryanck commented on code in PR #10173: URL: https://github.com/apache/iceberg/pull/10173#discussion_r1581844026 ## kafka-connect/build.gradle: ## @@ -43,6 +43,7 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") { implementation

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
SebastianoMeneghin commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080830949 Should I fix the problem or will you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Clean backport with:Flink: Backport #10207 to v1.18 and v1.17 [iceberg]

2024-04-27 Thread via GitHub
stevenzwu merged PR #10235: URL: https://github.com/apache/iceberg/pull/10235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Tracking issues of OpenDAL API changes [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo commented on issue #356: URL: https://github.com/apache/iceberg-rust/issues/356#issuecomment-2080446957 cc @Fokko @liurenjie1024 @ZENOTME to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[I] Tracking issues of OpenDAL API changes [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo opened a new issue, #356: URL: https://github.com/apache/iceberg-rust/issues/356 Hi, iceberger! OpenDAL's coming v0.46 release will have API changes that could affect our project. ## Changes I list the major changes that we need to take care: ### OpenDAL Reader

Re: [PR] docs: installation of the new `iceberg_catalog_rest` added to the docs [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo commented on PR #355: URL: https://github.com/apache/iceberg-rust/pull/355#issuecomment-2080421284 Also cc @Fokko and @liurenjie1024 for a review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above