Re: [PR] WIP: TableMetadataBuilder [iceberg-rust]

2024-09-02 Thread via GitHub
c-thiel commented on PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#issuecomment-2325737930 Fixes https://github.com/apache/iceberg-rust/issues/232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Flink: Fix compile warning [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat commented on PR #11072: URL: https://github.com/apache/iceberg/pull/11072#issuecomment-2325691307 @pvary: Done. I just compiled for default modules previously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Materialized View Spec [iceberg]

2024-09-02 Thread via GitHub
JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741477865 ## format/view-spec.md: ## @@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when t

Re: [PR] Materialized View Spec [iceberg]

2024-09-02 Thread via GitHub
JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741476428 ## format/view-spec.md: ## @@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki Writers create view metadata files

Re: [I] Iceberg table not able to read data from S3 after few hours using Athena . [iceberg]

2024-09-02 Thread via GitHub
yashgangrades commented on issue #9684: URL: https://github.com/apache/iceberg/issues/9684#issuecomment-2325634504 I am also facing the same issue. I am using Glue to create an Iceberg table from raw data in S3, but when I run a `SELECT *` query, it returns 0 results. -- This is an automa

Re: [PR] Docs: Fix Flink 1.20 support versions [iceberg]

2024-09-02 Thread via GitHub
pvary commented on PR #11065: URL: https://github.com/apache/iceberg/pull/11065#issuecomment-2325609769 @stevenzwu: do we similarly revert the shuffling write documentation then, until the release happens? -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Flink: Fix compile warning [iceberg]

2024-09-02 Thread via GitHub
pvary commented on PR #11072: URL: https://github.com/apache/iceberg/pull/11072#issuecomment-2325607580 @ajantha-bhat: do we have the same issue in 1.19? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-09-02 Thread via GitHub
amogh-jahagirdar commented on PR #10962: URL: https://github.com/apache/iceberg/pull/10962#issuecomment-2325605201 I'll take a pass over this PR tomorrow! thanks for your patience @jasonf20 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Kafka: runtime integration test failure or flaky [iceberg]

2024-09-02 Thread via GitHub
stevenzwu commented on issue #11046: URL: https://github.com/apache/iceberg/issues/11046#issuecomment-2325573357 cc @bryanck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-09-02 Thread via GitHub
amitgilad3 commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2325561162 Hey @szehon-ho @nastra @nk1506 - i fixed you suggestions , can you please review again. thanks -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Materialized View Spec [iceberg]

2024-09-02 Thread via GitHub
bennychow commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741392915 ## format/view-spec.md: ## @@ -158,6 +173,59 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when

Re: [PR] Materialized View Spec [iceberg]

2024-09-02 Thread via GitHub
bennychow commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1741391714 ## format/view-spec.md: ## @@ -42,12 +42,24 @@ An atomic swap of one view metadata file for another provides the basis for maki Writers create view metadata fil

Re: [PR] Docs: Add Databend docs url to sidebar [iceberg]

2024-09-02 Thread via GitHub
PsiACE commented on PR #10834: URL: https://github.com/apache/iceberg/pull/10834#issuecomment-2325537000 cc @amogh-jahagirdar @Fokko Can you guys help review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] Flink: Fix compile warning [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat opened a new pull request, #11072: URL: https://github.com/apache/iceberg/pull/11072 `./gradlew clean build -x test -x integrationTest --no-build-cache` results in compile warnings on the newly added code. `iceberg/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink

Re: [PR] open-api: Fix compile warnings for testFixtures [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat commented on code in PR #11071: URL: https://github.com/apache/iceberg/pull/11071#discussion_r1741345469 ## gradle/libs.versions.toml: ## @@ -177,6 +178,7 @@ slf4j-simple = { module = "org.slf4j:slf4j-simple", version.ref = "slf4j" } snowflake-jdbc = { module = "n

[PR] open-api: Fix compile warnings for testFixtures [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat opened a new pull request, #11071: URL: https://github.com/apache/iceberg/pull/11071 `./gradlew clean build -x test -x integrationTest --no-build-cache` results in compile warnings for open-api module. https://github.com/user-attachments/assets/27a8f97e-8a48-45f9-812a-b7

Re: [PR] chore(deps): Bump crate-ci/typos from 1.24.1 to 1.24.3 [iceberg-rust]

2024-09-02 Thread via GitHub
dependabot[bot] commented on PR #595: URL: https://github.com/apache/iceberg-rust/pull/595#issuecomment-2325424978 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump crate-ci/typos from 1.24.1 to 1.24.3 [iceberg-rust]

2024-09-02 Thread via GitHub
liurenjie1024 commented on PR #595: URL: https://github.com/apache/iceberg-rust/pull/595#issuecomment-2325424957 We can't merge this as it involves typo fix, will open an issue to track this. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] chore(deps): Bump crate-ci/typos from 1.24.1 to 1.24.3 [iceberg-rust]

2024-09-02 Thread via GitHub
liurenjie1024 closed pull request #595: chore(deps): Bump crate-ci/typos from 1.24.1 to 1.24.3 URL: https://github.com/apache/iceberg-rust/pull/595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] arrow/schema:new func `convert_schema` for `ArrowSchemaConverter` [iceberg-rust]

2024-09-02 Thread via GitHub
liurenjie1024 commented on PR #539: URL: https://github.com/apache/iceberg-rust/pull/539#issuecomment-2325423680 Hi, @AndreMouche Sorry for late reply. It seems that the benchmark result shows that there is no critial performance change? > Now it seems that we could use ArrowSc

Re: [I] Review new ImmutablesReferenceEquality error-prone check [iceberg]

2024-09-02 Thread via GitHub
danielhumanmod commented on issue #10855: URL: https://github.com/apache/iceberg/issues/10855#issuecomment-2325421244 Hi @findepi , I’ve shared an initial conclusion in my previous comment regarding the usage of `ImmutablesReferenceEquality`. Do you think any further action is needed on thi

[I] Unable to query iceberg table , getting unable to open manifest file "org.apache.avro.InvalidAvroMagicException: Not an Avro data file" [iceberg]

2024-09-02 Thread via GitHub
chennurchaitanya opened a new issue, #11070: URL: https://github.com/apache/iceberg/issues/11070 select * from $files Caused by: org.apache.iceberg.exceptions.RuntimeIOException: Failed to open file: s3a://siem-datamarts/datamarts/dm_bigipasm/metadata/a5cfd565-eb2e-4aa3-b39f-73d79214

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-2325404746 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] Optimize `plan_files` with filter in case whe it is fully evaluated on Iceberg metadata [iceberg-python]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #491: URL: https://github.com/apache/iceberg-python/issues/491#issuecomment-2325404736 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] Merge Small File Error [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7919: URL: https://github.com/apache/iceberg/issues/7919#issuecomment-2325403822 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [Feature Request] Inspect partitions Metadata for Tables with Many Partitions [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7892: URL: https://github.com/apache/iceberg/issues/7892#issuecomment-2325403794 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Data files name collision written by Spark Streaming job after it's restart [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7890: URL: https://github.com/apache/iceberg/issues/7890#issuecomment-2325403778 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] DataFrame inconsistency after MERGE operation [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7863: URL: https://github.com/apache/iceberg/issues/7863#issuecomment-2325403726 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Aliyun: Migrate tests to junit5 for aliyun client factory [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on PR #7853: URL: https://github.com/apache/iceberg/pull/7853#issuecomment-2325403716 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] PartitionSpec field name should be consistent for bucket and trunc in $partitions metadata table [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7849: URL: https://github.com/apache/iceberg/issues/7849#issuecomment-2325403695 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Add param to limit manifest parallel reader queue size [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on PR #7844: URL: https://github.com/apache/iceberg/pull/7844#issuecomment-2325403682 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Iceberg requiredNumOfPartitions method [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7918: URL: https://github.com/apache/iceberg/issues/7918#issuecomment-2325403806 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] API: Convert Evaluator in expression to use a Comparator [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on PR #7883: URL: https://github.com/apache/iceberg/pull/7883#issuecomment-2325403745 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] missing option in remove_orphan_files (prefix mismatch) [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7884: URL: https://github.com/apache/iceberg/issues/7884#issuecomment-2325403763 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Partition Filter returns incorrect results for decimal partition columns with trailing 0's [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7882: URL: https://github.com/apache/iceberg/issues/7882#issuecomment-2325403735 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] delete with clause IN [iceberg]

2024-09-02 Thread via GitHub
github-actions[bot] commented on issue #7850: URL: https://github.com/apache/iceberg/issues/7850#issuecomment-2325403704 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[PR] Bump mkdocs-material from 9.5.33 to 9.5.34 [iceberg-python]

2024-09-02 Thread via GitHub
dependabot[bot] opened a new pull request, #1126: URL: https://github.com/apache/iceberg-python/pull/1126 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.33 to 9.5.34. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

[PR] Bump mkdocs-autorefs from 1.1.0 to 1.2.0 [iceberg-python]

2024-09-02 Thread via GitHub
dependabot[bot] opened a new pull request, #1125: URL: https://github.com/apache/iceberg-python/pull/1125 Bumps [mkdocs-autorefs](https://github.com/mkdocstrings/autorefs) from 1.1.0 to 1.2.0. Release notes Sourced from https://github.com/mkdocstrings/autorefs/releases";>mkdocs-aut

[PR] Bump mkdocstrings from 0.25.2 to 0.26.0 [iceberg-python]

2024-09-02 Thread via GitHub
dependabot[bot] opened a new pull request, #1124: URL: https://github.com/apache/iceberg-python/pull/1124 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.25.2 to 0.26.0. Release notes Sourced from https://github.com/mkdocstrings/mkdocstrings/releases";>mkd

Re: [PR] Add metadata tables for `data_files` and `delete_files` [iceberg-python]

2024-09-02 Thread via GitHub
soumya-ghosh commented on code in PR #1066: URL: https://github.com/apache/iceberg-python/pull/1066#discussion_r1741247300 ## pyiceberg/table/__init__.py: ## @@ -4308,7 +4308,9 @@ def history(self) -> "pa.Table": return pa.Table.from_pylist(history, schema=history_sch

Re: [PR] Add metadata tables for `data_files` and `delete_files` [iceberg-python]

2024-09-02 Thread via GitHub
soumya-ghosh commented on code in PR #1066: URL: https://github.com/apache/iceberg-python/pull/1066#discussion_r1741244086 ## mkdocs/docs/api.md: ## @@ -844,6 +844,19 @@ readable_metrics: [ [6.0989]] ``` + Review Comment: Okay, removing it. I was not sure what this did i

Re: [PR] Add metadata tables for `data_files` and `delete_files` [iceberg-python]

2024-09-02 Thread via GitHub
soumya-ghosh commented on code in PR #1066: URL: https://github.com/apache/iceberg-python/pull/1066#discussion_r1741239066 ## tests/integration/test_inspect_table.py: ## @@ -672,126 +672,141 @@ def test_inspect_files( # append more data tbl.append(arrow_table_with_null

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-09-02 Thread via GitHub
ebyhr commented on PR #11064: URL: https://github.com/apache/iceberg/pull/11064#issuecomment-2325282952 @RussellSpitzer Can you review this PR when you have time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-02 Thread via GitHub
a-agmon commented on PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#issuecomment-2325251287 Thanks much for the review and comments @liurenjie1024 I refactored the code to be based on `TreeNodeVisitor` now, and all unit tests there. I initially thought that `TreeNodeVi

Re: [I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-09-02 Thread via GitHub
BTheunissen commented on issue #1104: URL: https://github.com/apache/iceberg-python/issues/1104#issuecomment-2325218390 @kevinjqliu Definitely fair enough that the reason for deprecation being that the catalog settings are generally exposed as a `Dict[str, str]` and the botocore.Session ob

Re: [PR] Manifest list encryption [iceberg]

2024-09-02 Thread via GitHub
rdblue commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1741145261 ## core/src/main/java/org/apache/iceberg/encryption/KeyEncryptionKey.java: ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Manifest list encryption [iceberg]

2024-09-02 Thread via GitHub
rdblue commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1741144386 ## api/src/main/java/org/apache/iceberg/ManifestListFile.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-09-02 Thread via GitHub
rdblue commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2325121721 I opened https://github.com/jacobmarble/apache-iceberg/pull/2 to fix the remaining issue, which is that there was no check that prevented the new type from being used in v1 or v2 tables.

Re: [I] Concurrent writes failures [iceberg-python]

2024-09-02 Thread via GitHub
bikeshedder commented on issue #1084: URL: https://github.com/apache/iceberg-python/issues/1084#issuecomment-2325085387 > FYI, according to the docs, "SQLite is not built for concurrency, you should use this catalog for exploratory or development purposes." https://py.iceberg.apache.org/co

Re: [I] Consolidate FileIO [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #310: URL: https://github.com/apache/iceberg-python/issues/310#issuecomment-2325085196 Moving forward, I suggest that we choose only one interface, but make the choice of using which underlying implementation configurable (based on the fact that [both implementat

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324865049 Its configurable via the write properties. See this comment https://github.com/apache/iceberg-python/issues/1041#issuecomment-2323380629 -- This is an automated mess

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

2024-09-02 Thread via GitHub
jurossiar commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2324860967 I'm wondering if you have any update on this issue? -> Previous comments: https://github.com/apache/iceberg/issues/9960#issuecomment-2197375635 I've just tried using: spark-v

Re: [I] Adjust the "table_exists" behavior in the REST Catalog [iceberg-python]

2024-09-02 Thread via GitHub
ndrluis closed issue #1018: Adjust the "table_exists" behavior in the REST Catalog URL: https://github.com/apache/iceberg-python/issues/1018 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324857080 Reading on table spec, I just realised that there is a field `location` in https://iceberg.apache.org/spec/#table-metadata-fields that specifies a base location of the table

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324831712 yep! There's definitely opportunities to consolidate the two. I opened #310 with some details. -- This is an automated message from the Apache Git Service. To respo

[I] Consolidate FileIO [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu opened a new issue, #310: URL: https://github.com/apache/iceberg-python/issues/310 ### Feature Request / Improvement **Can we consolidate and standardize FileIO to the PyArrow implementation?** There are currently two different FileIO implementations, `ARROW_FILE_IO`

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324824803 Also reading on here: https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec There might be some opportunity that we simplify the

Re: [PR] fix: fixing tests to work with s3Express [iceberg]

2024-09-02 Thread via GitHub
stubz151 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1740960324 ## aws/src/integration/java/org/apache/iceberg/aws/s3/TestS3FileIOIntegration.java: ## @@ -113,17 +116,23 @@ public static void beforeClass() { AwsIntegTestUtil

Re: [PR] fix: fixing tests to work with s3Express [iceberg]

2024-09-02 Thread via GitHub
stubz151 commented on PR #11021: URL: https://github.com/apache/iceberg/pull/11021#issuecomment-2324814314 Just reset the history this pr got a bit messy sorry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] fix: fixing tests to work with s3Express [iceberg]

2024-09-02 Thread via GitHub
stubz151 closed pull request #11021: fix: fixing tests to work with s3Express URL: https://github.com/apache/iceberg/pull/11021 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] CLI list not working [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1122: URL: https://github.com/apache/iceberg-python/issues/1122#issuecomment-2324805260 Thanks, putting uri in front of commands worked! Another thing is ``` -> % pyiceberg list --help URI missing, please provide using --uri, the

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324794360 @kevinjqliu I think resolving fs at file level should make the API cleaner. I would say one benefit one might choose fs on table level is to reuse that fs instance for perfo

Re: [I] CLI list not working [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1122: URL: https://github.com/apache/iceberg-python/issues/1122#issuecomment-2324786219 > ➜ pyiceberg --help Usage: pyiceberg [OPTIONS] COMMAND [ARGS]... can you try ``` pyiceberg --uri http://localhost:8181 list ``` from https://py.icebe

Re: [I] Implement `TableProviderFactory` for a `IcebergTableFactory` [iceberg-rust]

2024-09-02 Thread via GitHub
yukkit commented on issue #586: URL: https://github.com/apache/iceberg-rust/issues/586#issuecomment-2324782434 @liurenjie1024 Thank you so much for your feedback! I agree with your points, and I will start working on it based on this approach. -- This is an automated message from the Apac

Re: [I] CLI list not working [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1122: URL: https://github.com/apache/iceberg-python/issues/1122#issuecomment-2324776148 Looks like the catalog inference code requires the URI to include `http` in front https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc3

[I] New PR label [ready for review] [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu opened a new issue, #1123: URL: https://github.com/apache/iceberg-python/issues/1123 ### Feature Request / Improvement To streamline the PR review process, I propose adding a new label like "ready for review" to signal that the PR is waiting for reviewers. -- This i

Re: [PR] fix: fixing tests to work with s3Express [iceberg]

2024-09-02 Thread via GitHub
stubz151 commented on code in PR #11021: URL: https://github.com/apache/iceberg/pull/11021#discussion_r1740931594 ## aws/src/integration/java/org/apache/iceberg/aws/AwsIntegTestUtil.java: ## @@ -127,6 +129,39 @@ public static void cleanS3Bucket(S3Client s3, String bucketName, S

Re: [PR] Core: Refactor ZOrderByteUtils [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat commented on PR #10624: URL: https://github.com/apache/iceberg/pull/10624#issuecomment-2324764167 @RussellSpitzer: I have updated it today. Sorry for delay on this. I lost track of this PR notification. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Spec: Fix table of content generation [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat commented on PR #11067: URL: https://github.com/apache/iceberg/pull/11067#issuecomment-2324762847 @RussellSpitzer: I have updated it today. Sorry for delay on this. I lost track of this PR notification. -- This is an automated message from the Apache Git Service. To respond

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324762218 Generally, this problem should go away if we re-evaluate `fs` and `io` each time a file is read and written. Or other words, we should stop passing the `io` parameter arou

Re: [PR] fix: fixing tests to work with s3Express [iceberg]

2024-09-02 Thread via GitHub
stubz151 commented on PR #11021: URL: https://github.com/apache/iceberg/pull/11021#issuecomment-2324759204 > > I can add the backslash to the ListPrefix request. Do we want to do it for all s3 buckets(general + directory) this will be a breaking change, or would we prefer to add it in like

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324758344 > I dont think fixing SqlCatalog alone is the proper answer to this bug. The io layer seems to me ill written and has to be fixed somewhere in the uppper level (e.g. Fsspe

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
kevinjqliu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324750207 Thanks for taking a look at this @TiansuYu > why we are implementing a custom I think custom scheme parsing avoids picking one library over another (`fsspec`

[PR] Spec: Fix rendering of partition stats file spec [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat opened a new pull request, #11068: URL: https://github.com/apache/iceberg/pull/11068 Even though the indentation in `.md` file was correct, for website (mkdocs), need to use required syntax for rendering. Hence, adjusted it accordingly. before: https://githu

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-09-02 Thread via GitHub
ebyhr commented on PR #11064: URL: https://github.com/apache/iceberg/pull/11064#issuecomment-2324566532 CI hit https://github.com/apache/iceberg/issues/10172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324566016 Read my comment [here](https://gist.github.com/kevinjqliu/647808faba256855639e91dd58243082?permalink_comment_id=5175413#gistcomment-5175413) for the cause of the issue.

Re: [PR] Spec: Minor modifications for v3 [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat commented on code in PR #10948: URL: https://github.com/apache/iceberg/pull/10948#discussion_r1740747491 ## format/spec.md: ## @@ -113,9 +122,9 @@ Tables do not require random-access writes. Once written, data and metadata file Tables do not require rename, except

[PR] Spec: Fix table of content generation [iceberg]

2024-09-02 Thread via GitHub
ajantha-bhat opened a new pull request, #11067: URL: https://github.com/apache/iceberg/pull/11067 Table of contents in the [spec web page](https://iceberg.apache.org/spec/) is not generated for `spec` and its subsection after https://github.com/apache/iceberg/pull/10948. It is super hard to

Re: [PR] feat: support create partition table for non REST catalog [iceberg-rust]

2024-09-02 Thread via GitHub
FANNG1 commented on PR #577: URL: https://github.com/apache/iceberg-rust/pull/577#issuecomment-2324436354 > Thanks @FANNG1 for this pr. However I think there are some prepartion work before we can actually finished this pr. If we can narrow down the goal of this pr to change type for `Unbou

Re: [PR] feat: support create partition table for non REST catalog [iceberg-rust]

2024-09-02 Thread via GitHub
FANNG1 commented on code in PR #577: URL: https://github.com/apache/iceberg-rust/pull/577#discussion_r1740693686 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -301,12 +302,7 @@ impl TableMetadataBuilder { } = table_creation; let partition_specs = match

Re: [PR] feat: support create partition table for non REST catalog [iceberg-rust]

2024-09-02 Thread via GitHub
FANNG1 commented on code in PR #577: URL: https://github.com/apache/iceberg-rust/pull/577#discussion_r1740691810 ## crates/iceberg/tests/partition_table_test.rs: ## @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Openlineage support [iceberg]

2024-09-02 Thread via GitHub
pawel-big-lebowski closed pull request #9285: Openlineage support URL: https://github.com/apache/iceberg/pull/9285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2024-09-02 Thread via GitHub
manuzhang commented on issue #11016: URL: https://github.com/apache/iceberg/issues/11016#issuecomment-2324276412 This might be better asked in the `#vendor-cloudera` Slack channel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[I] Flaky test `TestCopyOnWriteMerge > testMergeWithSnapshotIsolation()` [iceberg]

2024-09-02 Thread via GitHub
manuzhang opened a new issue, #11066: URL: https://github.com/apache/iceberg/issues/11066 ### Apache Iceberg version main (development) ### Query engine Spark ### Please describe the bug 🐞 ``` 16:30:41 TestCopyOnWriteMerge > testMergeWithSnapshotIsolation

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-02 Thread via GitHub
liurenjie1024 commented on code in PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#discussion_r1740552449 ## crates/integrations/datafusion/src/physical_plan/predicate_converter.rs: ## @@ -0,0 +1,156 @@ +// Licensed to the Apache Software Foundation (ASF) under one

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324181879 I will have a look this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate [iceberg-rust]

2024-09-02 Thread via GitHub
a-agmon commented on PR #588: URL: https://github.com/apache/iceberg-rust/pull/588#issuecomment-2324171975 Hey @liurenjie1024 , I think this one is ready to go :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[I] CLI list not working [iceberg-python]

2024-09-02 Thread via GitHub
TiansuYu opened a new issue, #1122: URL: https://github.com/apache/iceberg-python/issues/1122 ### Apache Iceberg version 0.7.1 (latest release) ### Please describe the bug 🐞 I am currently trying out the pyiceberg CLI, and found this command seems not working as expected

Re: [PR] feat: support create partition table for non REST catalog [iceberg-rust]

2024-09-02 Thread via GitHub
liurenjie1024 commented on code in PR #577: URL: https://github.com/apache/iceberg-rust/pull/577#discussion_r1740498084 ## crates/iceberg/tests/partition_table_test.rs: ## @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic