Re: [PR] Move `zlib` and `unicode` licenses to `allow` [iceberg-rust]

2024-08-19 Thread via GitHub
liurenjie1024 merged PR #566: URL: https://github.com/apache/iceberg-rust/pull/566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722738419 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722735848 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722723867 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722719791 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Spark 3.5: Fix incorrect catalog loaded in TestCreateActions [iceberg]

2024-08-19 Thread via GitHub
manuzhang commented on code in PR #10952: URL: https://github.com/apache/iceberg/pull/10952#discussion_r1722688458 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java: ## @@ -186,6 +186,8 @@ public void before() { public void after() thro

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-08-19 Thread via GitHub
wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2297933397 @pvary I have added `testFlinkScenario1` and `testFlinkScenario2` to `TestChangelogReader`. Please check the expected results. (I will rename the tests later with more descriptive names.

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-19 Thread via GitHub
wmoustafa commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1722654876 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-19 Thread via GitHub
wmoustafa commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1722654876 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-19 Thread via GitHub
wmoustafa commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1722654876 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [I] How to use equality delete in Iceberg v2 table [iceberg]

2024-08-19 Thread via GitHub
link3280 commented on issue #6196: URL: https://github.com/apache/iceberg/issues/6196#issuecomment-2297901577 > > @singhpk234 is it worth documenting this so that it's clearer for users? > > Actually it took me a while to find this issue to confirm that Equality deletes are not suppor

Re: [PR] [For testing only] Testing the BaseIncrementalChangelogScan implementation from #9888 [iceberg]

2024-08-19 Thread via GitHub
manuzhang commented on PR #10954: URL: https://github.com/apache/iceberg/pull/10954#issuecomment-2297874007 @wypoon Thanks for creating this. It's been a while since I worked on #9888. I will revisit it and go through your PR as well. Hopefully we can push this feature for users this time.

Re: [PR] Ensure that RestCatalog passes user config to FileIO [iceberg-rust]

2024-08-19 Thread via GitHub
liurenjie1024 merged PR #476: URL: https://github.com/apache/iceberg-rust/pull/476 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Flink: backport PR #10777 from 1.19 to 1.18 for sink test refactoring. [iceberg]

2024-08-19 Thread via GitHub
stevenzwu commented on code in PR #10965: URL: https://github.com/apache/iceberg/pull/10965#discussion_r1722564325 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestBase.java: ## @@ -122,7 +123,23 @@ protected void assertSameElements(String message, Iterable expec

Re: [PR] Spark Action to Analyze table [iceberg]

2024-08-19 Thread via GitHub
karuppayya commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1722548584 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestComputeTableStatsAction.java: ## @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Refactor PyArrow DataFiles Projection functions [iceberg-python]

2024-08-19 Thread via GitHub
sungwy commented on PR #1043: URL: https://github.com/apache/iceberg-python/pull/1043#issuecomment-2297755108 > LGTM! I added a few comments. @Fokko @HonahX do you mind taking a look when you get a chance? Yes, would love to get your blessings on this refactoring @Fokko and @HonahX -

Re: [PR] Access delegation [iceberg-python]

2024-08-19 Thread via GitHub
sungwy commented on code in PR #1033: URL: https://github.com/apache/iceberg-python/pull/1033#discussion_r1722545650 ## pyiceberg/catalog/rest.py: ## @@ -532,7 +534,7 @@ def _config_headers(self, session: Session) -> None: session.headers["Content-type"] = "application/

Re: [PR] Update create table doc to clarify ID re-assignment [iceberg-python]

2024-08-19 Thread via GitHub
sungwy merged PR #1072: URL: https://github.com/apache/iceberg-python/pull/1072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Bump mkdocs-material from 9.5.31 to 9.5.32 [iceberg-python]

2024-08-19 Thread via GitHub
sungwy merged PR #1075: URL: https://github.com/apache/iceberg-python/pull/1075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Bump moto from 5.0.12 to 5.0.13 [iceberg-python]

2024-08-19 Thread via GitHub
sungwy merged PR #1077: URL: https://github.com/apache/iceberg-python/pull/1077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Bump griffe from 1.0.0 to 1.1.0 [iceberg-python]

2024-08-19 Thread via GitHub
sungwy merged PR #1076: URL: https://github.com/apache/iceberg-python/pull/1076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [I] How to write to a bucket-partitioned table using PySpark? [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on issue #5977: URL: https://github.com/apache/iceberg/issues/5977#issuecomment-2297732147 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Not able to process hundred of RowData types [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on issue #5958: URL: https://github.com/apache/iceberg/issues/5958#issuecomment-2297732114 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Docs: Using FlinkSQL for time travel in batch mode [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5951: URL: https://github.com/apache/iceberg/pull/5951#issuecomment-2297732073 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] refactor(flink): update deprecated code in flink [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5954: URL: https://github.com/apache/iceberg/pull/5954#issuecomment-2297732091 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Refactor commit lock mechanism from HiveTableOperations [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5877: URL: https://github.com/apache/iceberg/pull/5877#issuecomment-2297732001 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Core: Rollback compaction on conflicts [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5888: URL: https://github.com/apache/iceberg/pull/5888#issuecomment-2297732020 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Core: Use Table Partitioning sort with manual Sort Order in Rewrite Data Files [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] closed pull request #4941: Core: Use Table Partitioning sort with manual Sort Order in Rewrite Data Files URL: https://github.com/apache/iceberg/pull/4941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] Not able to process hundred of RowData types [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] closed issue #5958: Not able to process hundred of RowData types URL: https://github.com/apache/iceberg/issues/5958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Spark: Iceberg bug 5935 fix where some methods of Spark3Util do not set current session in spark's threadlocal [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5959: URL: https://github.com/apache/iceberg/pull/5959#issuecomment-2297732129 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Read Iceberg Table Bug(cannot find field start_date from [org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObj) [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on issue #5945: URL: https://github.com/apache/iceberg/issues/5945#issuecomment-2297732055 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] HuaweiCloud: Add support for HuaweiCloud OBS [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #5942: URL: https://github.com/apache/iceberg/pull/5942#issuecomment-2297732033 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Flink write iceberg bug(org.apache.iceberg.exceptions.NotFoundException) [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on issue #5846: URL: https://github.com/apache/iceberg/issues/5846#issuecomment-2297731988 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Use Table Partitioning sort with manual Sort Order in Rewrite Data Files [iceberg]

2024-08-19 Thread via GitHub
github-actions[bot] commented on PR #4941: URL: https://github.com/apache/iceberg/pull/4941#issuecomment-2297731713 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-19 Thread via GitHub
aokolnychyi commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1722504135 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-19 Thread via GitHub
aokolnychyi commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1722501925 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

[PR] Bump moto from 5.0.12 to 5.0.13 [iceberg-python]

2024-08-19 Thread via GitHub
dependabot[bot] opened a new pull request, #1077: URL: https://github.com/apache/iceberg-python/pull/1077 Bumps [moto](https://github.com/getmoto/moto) from 5.0.12 to 5.0.13. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

[PR] Bump griffe from 1.0.0 to 1.1.0 [iceberg-python]

2024-08-19 Thread via GitHub
dependabot[bot] opened a new pull request, #1076: URL: https://github.com/apache/iceberg-python/pull/1076 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 1.0.0 to 1.1.0. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

Re: [PR] OpenAPI: Add RemovePartitionSpecsUpdate REST update type [iceberg]

2024-08-19 Thread via GitHub
aokolnychyi commented on code in PR #10846: URL: https://github.com/apache/iceberg/pull/10846#discussion_r1722488735 ## open-api/rest-catalog-open-api.yaml: ## @@ -2265,6 +2265,7 @@ components: remove-statistics: '#/components/schemas/RemoveStatisticsUpdate'

Re: [PR] OpenAPI: Add RemovePartitionSpecsUpdate REST update type [iceberg]

2024-08-19 Thread via GitHub
aokolnychyi commented on code in PR #10846: URL: https://github.com/apache/iceberg/pull/10846#discussion_r1722488735 ## open-api/rest-catalog-open-api.yaml: ## @@ -2265,6 +2265,7 @@ components: remove-statistics: '#/components/schemas/RemoveStatisticsUpdate'

[PR] Bump mkdocs-material from 9.5.31 to 9.5.32 [iceberg-python]

2024-08-19 Thread via GitHub
dependabot[bot] opened a new pull request, #1075: URL: https://github.com/apache/iceberg-python/pull/1075 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.31 to 9.5.32. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

Re: [PR] Flink: FLIP-27 IcebergSource builder missed a couple of configs compared to old FlinkSource: expose locality and plan parallelism [iceberg]

2024-08-19 Thread via GitHub
stevenzwu merged PR #10957: URL: https://github.com/apache/iceberg/pull/10957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-19 Thread via GitHub
jackye1995 commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1722436641 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encod

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-19 Thread via GitHub
jackye1995 commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1722436641 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -194,15 +192,34 @@ public static String decodeString(String encoded) { * @return UTF-8 encod

Re: [PR] OpenAPI: Add RemovePartitionSpecsUpdate REST update type [iceberg]

2024-08-19 Thread via GitHub
amogh-jahagirdar commented on code in PR #10846: URL: https://github.com/apache/iceberg/pull/10846#discussion_r1722391302 ## open-api/rest-catalog-open-api.yaml: ## @@ -2265,6 +2265,7 @@ components: remove-statistics: '#/components/schemas/RemoveStatisticsUpdate'

Re: [PR] OpenAPI: Add RemovePartitionSpecsUpdate REST update type [iceberg]

2024-08-19 Thread via GitHub
amogh-jahagirdar commented on code in PR #10846: URL: https://github.com/apache/iceberg/pull/10846#discussion_r1722391302 ## open-api/rest-catalog-open-api.yaml: ## @@ -2265,6 +2265,7 @@ components: remove-statistics: '#/components/schemas/RemoveStatisticsUpdate'

Re: [PR] Flink: put everything together for range distribution in Flink sink [iceberg]

2024-08-19 Thread via GitHub
stevenzwu merged PR #10859: URL: https://github.com/apache/iceberg/pull/10859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] refactor: some more small refactors [iceberg-go]

2024-08-19 Thread via GitHub
zeroshade commented on PR #130: URL: https://github.com/apache/iceberg-go/pull/130#issuecomment-2297407754 CC @Fokko @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Spec: Minor modifications for v3 [iceberg]

2024-08-19 Thread via GitHub
rdblue commented on PR #10948: URL: https://github.com/apache/iceberg/pull/10948#issuecomment-2297312087 Thanks for reviewing, everyone. I've opened a thread on the dev list to merge this. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[I] Support writes to Bucket Partitioned Tables [iceberg-python]

2024-08-19 Thread via GitHub
sungwy opened a new issue, #1074: URL: https://github.com/apache/iceberg-python/issues/1074 ### Feature Request / Improvement We'd like to support writes to Bucket partitioned tables by leveraging pyiceberg_core package of iceberg-rust. - [ ] Expose Bucket Transform through PyO3 on

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722260283 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,335 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Ex

Re: [PR] Expose Bucket Transform to Python Binding [iceberg-rust]

2024-08-19 Thread via GitHub
sungwy commented on PR #556: URL: https://github.com/apache/iceberg-rust/pull/556#issuecomment-2297298410 Thank you for the reviews @Xuanwo - I've taken all of your feedback and refactored the code to use the arrow::pyarrow crate -- This is an automated message from the Apache Git Service

Re: [PR] Ensure that RestCatalog passes user config to FileIO [iceberg-rust]

2024-08-19 Thread via GitHub
sdd commented on code in PR #476: URL: https://github.com/apache/iceberg-rust/pull/476#discussion_r1722209176 ## crates/catalog/rest/src/catalog.rs: ## @@ -504,8 +504,15 @@ impl Catalog for RestCatalog { .query::(request) .await?; +let config

Re: [PR] Post Release 0.7.1 version updates [iceberg-python]

2024-08-19 Thread via GitHub
sungwy merged PR #1073: URL: https://github.com/apache/iceberg-python/pull/1073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

[PR] Post Release 0.7.1 version updates [iceberg-python]

2024-08-19 Thread via GitHub
sungwy opened a new pull request, #1073: URL: https://github.com/apache/iceberg-python/pull/1073 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722175695 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,335 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Ex

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722173257 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,335 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Ex

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722169292 ## crates/catalog/sql/src/catalog.rs: ## @@ -141,21 +142,30 @@ impl SqlCatalog { } /// SQLX Any does not implement PostgresSQL bindings, so we have t

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722170737 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,335 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Ex

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-19 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1722169865 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,335 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Ex

Re: [PR] Ensure that RestCatalog passes user config to FileIO [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on code in PR #476: URL: https://github.com/apache/iceberg-rust/pull/476#discussion_r1722149100 ## crates/catalog/rest/src/catalog.rs: ## @@ -504,8 +504,15 @@ impl Catalog for RestCatalog { .query::(request) .await?; +let conf

Re: [I] Iceberg Spark streaming skips rows of data [iceberg]

2024-08-19 Thread via GitHub
singhpk234 commented on issue #10156: URL: https://github.com/apache/iceberg/issues/10156#issuecomment-2297093657 @cccs-jc no i wasn't i tried this unit test : ``` @TestTemplate public void testResumingStreamReadFromCheckpointWithStreamFromTimestamp() throws Exception {

Re: [PR] Python: Use hatch for dependency management [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo merged PR #572: URL: https://github.com/apache/iceberg-rust/pull/572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-08-19 Thread via GitHub
stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1722057725 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java: ## @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software

Re: [PR] Docs: Update MRAP endpoint and add more docs [iceberg]

2024-08-19 Thread via GitHub
singhpk234 commented on PR #9362: URL: https://github.com/apache/iceberg/pull/9362#issuecomment-2297005056 Thanks @amogh-jahagirdar really appreciate it ! No problem at all, can totally understand ! -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Docs: Update MRAP endpoint and add more docs [iceberg]

2024-08-19 Thread via GitHub
amogh-jahagirdar merged PR #9362: URL: https://github.com/apache/iceberg/pull/9362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Docs: Update MRAP endpoint and add more docs [iceberg]

2024-08-19 Thread via GitHub
amogh-jahagirdar commented on PR #9362: URL: https://github.com/apache/iceberg/pull/9362#issuecomment-2297000551 Sorry @singhpk234 that this somehow missed getting merged a long time ago! I think we can just go ahead and merge. -- This is an automated message from the Apache Git Service.

Re: [PR] Expose Bucket Transform to Python Binding [iceberg-rust]

2024-08-19 Thread via GitHub
sungwy commented on code in PR #556: URL: https://github.com/apache/iceberg-rust/pull/556#discussion_r1722066450 ## .github/workflows/bindings_python_ci.yml: ## @@ -78,6 +78,6 @@ jobs: shell: bash run: | set -e - pip install dist/pyiceberg_c

[PR] Python: Use hatch for dependency management [iceberg-rust]

2024-08-19 Thread via GitHub
sungwy opened a new pull request, #572: URL: https://github.com/apache/iceberg-rust/pull/572 Currently, the dependency management is a bit awkward because we duplicate the list of optional dependencies in `pyproject.toml` under `[project.optional-dependencies]`, and then in `.github/workfl

Re: [PR] Docs: Add documentation for Rate limiting in Spark Structured Streaming [iceberg]

2024-08-19 Thread via GitHub
singhpk234 commented on PR #8044: URL: https://github.com/apache/iceberg/pull/8044#issuecomment-2296979743 This has been raised in community a couple of times as at this point one has to read code to see it's supported : Relevant slack threads where this being documented could have helpe

Re: [PR] Docs: Update MRAP endpoint and add more docs [iceberg]

2024-08-19 Thread via GitHub
singhpk234 commented on PR #9362: URL: https://github.com/apache/iceberg/pull/9362#issuecomment-2296969461 cc @nastra @Fokko. S3 Access point team PM and SA team requested the following doc change request. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1722038556 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsGenerator.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] SQL Catalog Implementation? [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo closed issue #205: SQL Catalog Implementation? URL: https://github.com/apache/iceberg-rust/issues/205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] SQL Catalog Implementation? [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #205: URL: https://github.com/apache/iceberg-rust/issues/205#issuecomment-2296938105 The SQL catalog work has been started. Let's create a new tracking issue instead. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] ci: Security audit failed due to rsa. [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #221: URL: https://github.com/apache/iceberg-rust/issues/221#issuecomment-2296935953 No actions to take so far, closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Integrate with datafusion [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #242: URL: https://github.com/apache/iceberg-rust/issues/242#issuecomment-2296934785 Basic integration has been added in https://github.com/apache/iceberg-rust/pull/324. We can create a new tracking issues for missing pieces. -- This is an automated messag

Re: [I] Integrate with datafusion [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo closed issue #242: Integrate with datafusion URL: https://github.com/apache/iceberg-rust/issues/242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Add gcs support in FileIO [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo closed issue #239: Add gcs support in FileIO URL: https://github.com/apache/iceberg-rust/issues/239 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Add gcs support in FileIO [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #239: URL: https://github.com/apache/iceberg-rust/issues/239#issuecomment-2296933470 Been added in https://github.com/apache/iceberg-rust/pull/520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Scan does not work as expected [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #495: URL: https://github.com/apache/iceberg-rust/issues/495#issuecomment-2296930200 I believe this should have been fixed. Please feel free to open new issues if still exists. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] Scan does not work as expected [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo closed issue #495: Scan does not work as expected URL: https://github.com/apache/iceberg-rust/issues/495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Should we use to_avro instead of `RawLiteral` indirection layer [iceberg-rust]

2024-08-19 Thread via GitHub
Xuanwo commented on issue #562: URL: https://github.com/apache/iceberg-rust/issues/562#issuecomment-2296893923 > I think it's > > `Literal -> RawLiteral -> Bytes` vs `Literal -> avro::Value -> Bytes` I see. Thanks! -- This is an automated message from the Apache Git Service.

Re: [I] Should we use to_avro instead of `RawLiteral` indirection layer [iceberg-rust]

2024-08-19 Thread via GitHub
xxchan commented on issue #562: URL: https://github.com/apache/iceberg-rust/issues/562#issuecomment-2296891098 I think it's `Literal -> RawLiteral -> Bytes` vs `Literal -> avro::Value -> Bytes` -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721988655 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721980729 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721979347 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Fix NPE in Resolving FileIo when conf is not set [iceberg]

2024-08-19 Thread via GitHub
munendrasn commented on PR #10872: URL: https://github.com/apache/iceberg/pull/10872#issuecomment-2296858187 @nastra Updated, reverted the last commit in the PR fe256b411363d1f884ce7a51a07f53b8fb9a797b -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721978424 ## data/src/jmh/java/org/apache/iceberg/PartitionStatsGeneratorBenchmark.java: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721975924 ## core/src/main/java/org/apache/iceberg/data/IdentityPartitionConverters.java: ## @@ -48,6 +50,13 @@ public static Object convertConstant(Type type, Object v

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.27.0 to 1.30.4 [iceberg-go]

2024-08-19 Thread via GitHub
nastra commented on PR #128: URL: https://github.com/apache/iceberg-go/pull/128#issuecomment-2296846237 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Fix NPE in Resolving FileIo when conf is not set [iceberg]

2024-08-19 Thread via GitHub
nastra commented on PR #10872: URL: https://github.com/apache/iceberg/pull/10872#issuecomment-2296845324 > @nastra Thank you for the review, please let me know if I should moved the `null` check to `getConf` or should we create new `Configuration` if not set? @munendrasn yes I think i

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721966029 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721957929 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] feat(table/scanner): Initial pass for planning a scan and returning the files to use [iceberg-go]

2024-08-19 Thread via GitHub
zeroshade commented on PR #118: URL: https://github.com/apache/iceberg-go/pull/118#issuecomment-2296822491 @nastra All rebased already :smile: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-19 Thread via GitHub
RussellSpitzer commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1721954085 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Fix NPE in Resolving FileIo when conf is not set [iceberg]

2024-08-19 Thread via GitHub
munendrasn commented on PR #10872: URL: https://github.com/apache/iceberg/pull/10872#issuecomment-2296807136 @nastra Thank you for the review, please let me know if I should moved the `null` check to `getConf` or should we create new `Configuration` if not set? -- This is an automated mes

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.27.0 to 1.30.4 [iceberg-go]

2024-08-19 Thread via GitHub
zeroshade commented on PR #128: URL: https://github.com/apache/iceberg-go/pull/128#issuecomment-2296788984 @Fokko @nastra could one of you please have dependabot rebase these PRs? I'm not a committer so I don't have permissions to make it rebase myself -- This is an automated message from

Re: [PR] feat(table/scanner): Initial pass for planning a scan and returning the files to use [iceberg-go]

2024-08-19 Thread via GitHub
nastra commented on PR #118: URL: https://github.com/apache/iceberg-go/pull/118#issuecomment-2296770709 @zeroshade could you please rebase this one now that all the other PRs are merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.27.2 to 2.27.7 [iceberg]

2024-08-19 Thread via GitHub
nastra merged PR #10961: URL: https://github.com/apache/iceberg/pull/10961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] feat(manifest): fix partition data map [iceberg-go]

2024-08-19 Thread via GitHub
nastra merged PR #124: URL: https://github.com/apache/iceberg-go/pull/124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

  1   2   >