[PR] Spark: Remove support for Spark 3.2 [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat opened a new pull request, #9295: URL: https://github.com/apache/iceberg/pull/9295 PR contains two commits: a) Removes the spark-3.2 mention from the docs and configs (outside spark/v3.2 folder). b) To blindly delete the spark/v3.2 folder. -- This is an automated messa

Re: [PR] Spark 3.5: Add Spark application id to summary of RewriteDataFilesSparkAction [iceberg]

2023-12-13 Thread via GitHub
manuzhang commented on PR #9273: URL: https://github.com/apache/iceberg/pull/9273#issuecomment-1855330976 The failed check is due to a flaky test. I created https://github.com/apache/iceberg/pull/9294 to fix it. -- This is an automated message from the Apache Git Service. To respond to th

[PR] Test: Fix flaky tests which concurrently modify HashSet [iceberg]

2023-12-13 Thread via GitHub
manuzhang opened a new pull request, #9294: URL: https://github.com/apache/iceberg/pull/9294 As found in https://github.com/apache/iceberg/actions/runs/7191749083/job/19586958583?pr=9273 ``` TestRemoveOrphanFilesAction3 > orphanedFileRemovedWithParallelTasks FAILED java.lang

Re: [PR] WIP: Glue catalog commit [iceberg-python]

2023-12-13 Thread via GitHub
HonahX commented on PR #140: URL: https://github.com/apache/iceberg-python/pull/140#issuecomment-1855276027 I'm quite busy at the moment, but I'm aiming to complete this by the end of the week. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-13 Thread via GitHub
manuzhang commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1426265402 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java: ## @@ -68,6 +68,18 @@ public Boolean asBoolean(ProcedureParameter param, Bo

Re: [PR] WIP: Glue catalog commit [iceberg-python]

2023-12-13 Thread via GitHub
HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426255789 ## pyiceberg/catalog/glue.py: ## @@ -247,8 +278,50 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons Raises:

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426257091 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_m

Re: [PR] WIP: Glue catalog commit [iceberg-python]

2023-12-13 Thread via GitHub
HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426255789 ## pyiceberg/catalog/glue.py: ## @@ -247,8 +278,50 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons Raises:

Re: [PR] WIP: Glue catalog commit [iceberg-python]

2023-12-13 Thread via GitHub
HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426255789 ## pyiceberg/catalog/glue.py: ## @@ -247,8 +278,50 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons Raises:

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426255077 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifes

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426250459 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_m

Re: [PR] WIP: Glue catalog commit [iceberg-python]

2023-12-13 Thread via GitHub
HonahX commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426238613 ## pyiceberg/catalog/glue.py: ## @@ -177,6 +191,23 @@ def _create_glue_table(self, database_name: str, table_name: str, table_input: T except self.glue.ex

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2023-12-13 Thread via GitHub
geruh commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1426221963 ## open-api/rest-catalog-open-api.yaml: ## @@ -2367,6 +2368,21 @@ components: items: type: string +AppendFilesUpdate: + allOf: +

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426215180 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifes

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426215180 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifes

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426207729 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_m

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426184104 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifes

Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on issue #119: URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855134729 If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,format%2

Re: [PR] Core: Use properties while initializing default HadoopFileIO for Hadoop catalog. [iceberg]

2023-12-13 Thread via GitHub
agrawalreetika commented on code in PR #9283: URL: https://github.com/apache/iceberg/pull/9283#discussion_r1426167674 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopCatalog.java: ## @@ -112,11 +112,11 @@ public void initialize(String name, Map properties) { this.war

Re: [I] Flink related document improvement request [iceberg]

2023-12-13 Thread via GitHub
911432 commented on issue #7395: URL: https://github.com/apache/iceberg/issues/7395#issuecomment-1855109087 not-stale -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Spark 3.5: Parallelize file listing in add_files procedure [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1426150782 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -351,6 +351,8 @@ private Map summary(TableMetadata previous) { SnapshotSummary.

Re: [I] Read ManifestList, Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on issue #36: URL: https://github.com/apache/iceberg-rust/issues/36#issuecomment-1855092587 Close by #79 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Read ManifestList, Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 closed issue #36: Read ManifestList, Manifest URL: https://github.com/apache/iceberg-rust/issues/36 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] Discussion: The design of in memory model of iceberg spec. [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 closed issue #3: Discussion: The design of in memory model of iceberg spec. URL: https://github.com/apache/iceberg-rust/issues/3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Discussion: The design of in memory model of iceberg spec. [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on issue #3: URL: https://github.com/apache/iceberg-rust/issues/3#issuecomment-1855091407 Close for now. Feel free to reopen when necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Design of Serialization/Deserialization [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 closed issue #2: Design of Serialization/Deserialization URL: https://github.com/apache/iceberg-rust/issues/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Design of Serialization/Deserialization [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on issue #2: URL: https://github.com/apache/iceberg-rust/issues/2#issuecomment-1855090734 Close this as we have finished discussion. Feel free to open if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426141277 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_m

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426135812 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifes

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1426129491 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -1120,15 +1236,17 @@ mod test { #[tokio::test] async fn test_manifest_list_writer_v2() { -let ex

Re: [I] Support bucket transform on multiple data columns [iceberg]

2023-12-13 Thread via GitHub
pdames commented on issue #5626: URL: https://github.com/apache/iceberg/issues/5626#issuecomment-1854969584 +1 to adding support for this feature. On my side, the principal use-case where this is beneficial is when hashing a composite primary key column where the cardinality of each individ

[PR] Adding Snowflake's public documentation [iceberg-docs]

2023-12-13 Thread via GitHub
scottteal opened a new pull request, #297: URL: https://github.com/apache/iceberg-docs/pull/297 Updating vendor description links with latest public documentation from Snowflake. Adding Snowflake's primary docs link for Iceberg to the sidebar. -- This is an automated message from the Apac

[I] Spark Rewrite Write Performance [iceberg]

2023-12-13 Thread via GitHub
W-I-D-EE opened a new issue, #9293: URL: https://github.com/apache/iceberg/issues/9293 ### Query engine Spark 3.2.3 ### Question When compacting small parquet files i noticed it seems slow. Writing a single ~256mb parquet file to HDFS is taking 4-5 mins (total time inclu

Re: [PR] Add name-mapping [iceberg-python]

2023-12-13 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1426017219 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,204 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [I] Support Delete & Update iceberg table with Flink [iceberg]

2023-12-13 Thread via GitHub
github-actions[bot] closed issue #7311: Support Delete & Update iceberg table with Flink URL: https://github.com/apache/iceberg/issues/7311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Support Delete & Update iceberg table with Flink [iceberg]

2023-12-13 Thread via GitHub
github-actions[bot] commented on issue #7311: URL: https://github.com/apache/iceberg/issues/7311#issuecomment-1854885763 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Sort order id not being respected [iceberg]

2023-12-13 Thread via GitHub
github-actions[bot] commented on issue #7600: URL: https://github.com/apache/iceberg/issues/7600#issuecomment-1854885694 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Sort order id not being respected [iceberg]

2023-12-13 Thread via GitHub
github-actions[bot] closed issue #7600: Sort order id not being respected URL: https://github.com/apache/iceberg/issues/7600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2023-12-13 Thread via GitHub
jackye1995 commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1425987620 ## open-api/rest-catalog-open-api.yaml: ## @@ -2367,6 +2368,21 @@ components: items: type: string +AppendFilesUpdate: + allOf: +

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2023-12-13 Thread via GitHub
jackye1995 commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1425987620 ## open-api/rest-catalog-open-api.yaml: ## @@ -2367,6 +2368,21 @@ components: items: type: string +AppendFilesUpdate: + allOf: +

[PR] Build: Bump mypy-boto3-glue from 1.33.5 to 1.34.0 [iceberg-python]

2023-12-13 Thread via GitHub
dependabot[bot] opened a new pull request, #213: URL: https://github.com/apache/iceberg-python/pull/213 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.33.5 to 1.34.0. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commits";

[PR] Rest append files [iceberg]

2023-12-13 Thread via GitHub
geruh opened a new pull request, #9292: URL: https://github.com/apache/iceberg/pull/9292 Hi All, I have created this smaller PR focused more on the AppendFiles operations. These changes are stemming from this PR: [#9237](https://github.com/apache/iceberg/pull/9237). @jackye19

[PR] Add name-mapping [iceberg-python]

2023-12-13 Thread via GitHub
Fokko opened a new pull request, #212: URL: https://github.com/apache/iceberg-python/pull/212 All the things to (de)serialize the name-mapping, and all the neccessary visitors and such -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] WIP: first pass at `UnboundTransform` [iceberg-python]

2023-12-13 Thread via GitHub
jayceslesar commented on PR #209: URL: https://github.com/apache/iceberg-python/pull/209#issuecomment-1854687084 @Fokko this isnt completely done, but would appreciate a review + some guidance on next steps once you run linting and see whats up haha -- This is an automated message from th

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-13 Thread via GitHub
ismailsimsek commented on PR #8340: URL: https://github.com/apache/iceberg/pull/8340#issuecomment-1854653706 Thank you all for reviewing it, @amogh-jahagirdar should we also merge it to 1.4.x branch, for next release? created PR for it if makes sense https://github.com/apache/iceberg/pul

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425822122 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -474,68 +421,52 @@ private Collection loadDeleteFiles() { DeleteFileIndex build() {

Re: [I] [JdbcCatalog] Issue with Namespace Exists [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar closed issue #8832: [JdbcCatalog] Issue with Namespace Exists URL: https://github.com/apache/iceberg/issues/8832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] [JDBC Catalog on PostgreSQL] Undesired query result from namespace with underscore in the name [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar closed issue #8321: [JDBC Catalog on PostgreSQL] Undesired query result from namespace with underscore in the name URL: https://github.com/apache/iceberg/issues/8321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar merged PR #8340: URL: https://github.com/apache/iceberg/pull/8340 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar commented on PR #8340: URL: https://github.com/apache/iceberg/pull/8340#issuecomment-1854514510 Merging, thanks again for this fix @ismailsimsek and your follow-through. Thanks for reviews @dramaticlly @nastra @rdblue -- This is an automated message from the Apache Git S

Re: [I] Iceberg table querying and joining issues [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar closed issue #9287: Iceberg table querying and joining issues URL: https://github.com/apache/iceberg/issues/9287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Iceberg table querying and joining issues [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar commented on issue #9287: URL: https://github.com/apache/iceberg/issues/9287#issuecomment-1854489654 This seems like an AWS Athena issue, could you report the issue to them? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Remove Spark-3.2 support for Iceberg 1.5.0 and onwards. [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat commented on issue #9290: URL: https://github.com/apache/iceberg/issues/9290#issuecomment-1854477223 I will have a PR by tomorrow (based on the community discussion conclusion). -- This is an automated message from the Apache Git Service. To respond to the message, please log

[I] Remove Spark-3.2 support for Iceberg 1.5.0 and onwards. [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat opened a new issue, #9290: URL: https://github.com/apache/iceberg/issues/9290 Based on the mailing list discussion https://lists.apache.org/thread/ttonfgrfb3ssrhl38kc7hljqmf5phlfw -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Core: Use properties while initializing default HadoopFileIO for Hadoop catalog. [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9283: URL: https://github.com/apache/iceberg/pull/9283#discussion_r1425696907 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopCatalog.java: ## @@ -112,11 +112,11 @@ public void initialize(String name, Map properties) { this.w

Re: [PR] Implementation of Glue View Catalog [iceberg]

2023-12-13 Thread via GitHub
amogh-jahagirdar closed pull request #5085: Implementation of Glue View Catalog URL: https://github.com/apache/iceberg/pull/5085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Support 'schema.name-mapping.default' Column Projection property [iceberg-python]

2023-12-13 Thread via GitHub
syun64 commented on issue #202: URL: https://github.com/apache/iceberg-python/issues/202#issuecomment-1854378926 @Fokko Absolutely 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Support 'schema.name-mapping.default' Column Projection property [iceberg-python]

2023-12-13 Thread via GitHub
Fokko commented on issue #202: URL: https://github.com/apache/iceberg-python/issues/202#issuecomment-1854334634 @syun64 Thanks for raising this, is this something that you're interested in contributing? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] AWS access denied from metadata file on S3 [iceberg-python]

2023-12-13 Thread via GitHub
Fokko closed issue #201: AWS access denied from metadata file on S3 URL: https://github.com/apache/iceberg-python/issues/201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] AWS access denied from metadata file on S3 [iceberg-python]

2023-12-13 Thread via GitHub
Fokko commented on issue #201: URL: https://github.com/apache/iceberg-python/issues/201#issuecomment-1854326828 Awesome, thanks for letting me know! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2023-12-13 Thread via GitHub
harshith-bolar-rapido commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1854204500 Noticed that this happens when the input folders are partitioned by date. Ex: `mmdd=20230304` Here's a minimal reproducible example. @RussellSpitzer @nastra

Re: [PR] Replace deprecated RuntimeIOException with UncheckedIOException [iceberg]

2023-12-13 Thread via GitHub
jbonofre commented on code in PR #4776: URL: https://github.com/apache/iceberg/pull/4776#discussion_r1425565613 ## api/src/main/java/org/apache/iceberg/Files.java: ## @@ -24,8 +24,8 @@ import java.io.RandomAccessFile; import java.nio.file.Paths; import org.apache.iceberg.exce

Re: [PR] Replace deprecated RuntimeIOException with UncheckedIOException [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat commented on code in PR #4776: URL: https://github.com/apache/iceberg/pull/4776#discussion_r1425558869 ## api/src/main/java/org/apache/iceberg/Files.java: ## @@ -24,8 +24,8 @@ import java.io.RandomAccessFile; import java.nio.file.Paths; import org.apache.iceberg.

Re: [PR] Core: Add Catalog Transactions API [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #6948: URL: https://github.com/apache/iceberg/pull/6948#discussion_r1425557919 ## core/src/main/java/org/apache/iceberg/catalog/CatalogTransaction.java: ## @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat commented on code in PR #6887: URL: https://github.com/apache/iceberg/pull/6887#discussion_r142508 ## api/src/main/java/org/apache/iceberg/exceptions/FileHandlingException.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

2023-12-13 Thread via GitHub
ajantha-bhat commented on code in PR #6887: URL: https://github.com/apache/iceberg/pull/6887#discussion_r1425552569 ## api/src/main/java/org/apache/iceberg/exceptions/FileHandlingException.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

2023-12-13 Thread via GitHub
Fokko commented on code in PR #6887: URL: https://github.com/apache/iceberg/pull/6887#discussion_r1425537371 ## api/src/main/java/org/apache/iceberg/exceptions/FileHandlingException.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425513915 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1210,34 +1182,53 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425445565 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1211,33 +1183,48 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425443702 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425406151 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [I] AWS access denied from metadata file on S3 [iceberg-python]

2023-12-13 Thread via GitHub
itaise commented on issue #201: URL: https://github.com/apache/iceberg-python/issues/201#issuecomment-1853965470 Hi @Fokko, The first suggestion did not work, but passing the credentials explicitly like in your second suggestion worked. Thank you very much! -- This is an automated

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-12-13 Thread via GitHub
cccs-jc commented on PR #8980: URL: https://github.com/apache/iceberg/pull/8980#issuecomment-1853964957 so I did more digging. On our production tables I search for all manifests which have a `existing_data_files_count > 0` and `added_data_files_count > 0` and I find none. This leads me to

Re: [PR] Replace deprecated RuntimeIOException with UncheckedIOException [iceberg]

2023-12-13 Thread via GitHub
jbonofre commented on code in PR #4776: URL: https://github.com/apache/iceberg/pull/4776#discussion_r1425380109 ## api/src/main/java/org/apache/iceberg/Files.java: ## @@ -24,8 +24,8 @@ import java.io.RandomAccessFile; import java.nio.file.Paths; import org.apache.iceberg.exce

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425379313 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,97 +524,214 @@ private Iterable>> deleteManifestRea } } - // a group of indexe

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425377180 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,93 +513,187 @@ private Iterable>> deleteManifestRea } } - // a group of indexe

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425376250 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -474,68 +421,52 @@ private Collection loadDeleteFiles() { DeleteFileIndex build() {

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425372419 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,93 +513,187 @@ private Iterable>> deleteManifestRea } } - // a group of indexe

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425375381 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,93 +513,187 @@ private Iterable>> deleteManifestRea } } - // a group of indexe

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425372419 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,93 +513,187 @@ private Iterable>> deleteManifestRea } } - // a group of indexe

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425370701 ## core/src/main/java/org/apache/iceberg/util/ArrayUtil.java: ## @@ -320,4 +320,29 @@ public static boolean isStrictlyAscending(long[] array) { return true;

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425370277 ## core/src/main/java/org/apache/iceberg/util/ArrayUtil.java: ## @@ -320,4 +320,29 @@ public static boolean isStrictlyAscending(long[] array) { return true;

Re: [PR] Core: Lookup targeted position deletes by path [iceberg]

2023-12-13 Thread via GitHub
aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425368136 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -474,68 +421,52 @@ private Collection loadDeleteFiles() { DeleteFileIndex build() {

Re: [PR] Core: Use properties while initializing default HadoopFileIO for Hadoop catalog. [iceberg]

2023-12-13 Thread via GitHub
yingsu00 commented on PR #9283: URL: https://github.com/apache/iceberg/pull/9283#issuecomment-1853905363 PR message "to Initialise" -> to initialize -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425315871 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-12-13 Thread via GitHub
pvary commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1853832988 > When you say table commits, does that mean the entire table? Not the whole table data, but the table metadata. Starting from the end of the history (most recent), it goes throu

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-12-13 Thread via GitHub
FranMorilloAWS commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1853806769 Im facing issue that when flink restarts it is not able to commit to all tables.. Any reason for this? -- This is an automated message from the Apache Git Service. To respo

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-12-13 Thread via GitHub
FranMorilloAWS commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1853776664 Do we need to add an operator uid, to make sure state compatibility? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

2023-12-13 Thread via GitHub
FranMorilloAWS commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1853771914 When you say table commits, does that mean the entire table?. Also could there be a scenario that because of state mis allignment, that is not able to get the highest checkpo

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1425146761 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Serializabl

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko merged PR #79: URL: https://github.com/apache/iceberg-rust/pull/79 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] test: Replace binary avro file by generating it on the fly. [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko commented on issue #70: URL: https://github.com/apache/iceberg-rust/issues/70#issuecomment-1853637160 I would love to see this before the release, there is also some interesting data here: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425129671 -- This is an automated

[I] Support `initial-default` when reading Avro [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko opened a new issue, #119: URL: https://github.com/apache/iceberg-rust/issues/119 I would suggest adding support for `initial-default` to read V1 Avro files into V2 objects in memory. For example, V1 does not have sequence numbers, and then need to be set to `0`. This is importa

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
ZENOTME commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425139874 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1864 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425137422 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1864 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425129671 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1821 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425117160 ## crates/iceberg/src/spec/manifest.rs: ## @@ -0,0 +1,1821 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: suport read/write Manifest [iceberg-rust]

2023-12-13 Thread via GitHub
Fokko commented on code in PR #79: URL: https://github.com/apache/iceberg-rust/pull/79#discussion_r1425109506 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -1120,15 +1236,17 @@ mod test { #[tokio::test] async fn test_manifest_list_writer_v2() { -let expe

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1425096603 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -219,81 +208,6 @@ public void testInitializeCatalogWithProperties() { .isEqu

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-12-13 Thread via GitHub
nk1506 commented on PR #8918: URL: https://github.com/apache/iceberg/pull/8918#issuecomment-1853577807 Thanks @ajantha-bhat and @nastra for recommending configurable approach for `HiveMetastoreExtension` and fixing this. Although I had to make a small [fix](https://github.com/apache/iceber

  1   2   >