Re: [PR] Core:Simplify newTableMetadata method in TableMetadata class [iceberg]

2024-06-17 Thread via GitHub
wangyinsheng commented on code in PR #10528: URL: https://github.com/apache/iceberg/pull/10528#discussion_r1643914605 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -74,11 +74,7 @@ public static TableMetadata newTableMetadata( public static TableMetadata n

Re: [PR] Core:Simplify newTableMetadata method in TableMetadata class [iceberg]

2024-06-17 Thread via GitHub
wangyinsheng commented on code in PR #10528: URL: https://github.com/apache/iceberg/pull/10528#discussion_r1643914605 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -74,11 +74,7 @@ public static TableMetadata newTableMetadata( public static TableMetadata n

[PR] Core:Simplify newTableMetadata method in TableMetadata class [iceberg]

2024-06-17 Thread via GitHub
DaqianLiao opened a new pull request, #10528: URL: https://github.com/apache/iceberg/pull/10528 This pr simplifies the newTableMetadata method call in Iceberg by removing the redundant TableMetadata class. We can create `TableMetadata` object by using method 'org.apache.iceberg.TableMeta

Re: [PR] Flink: move flush operation from prepareSnapshotPreBarrier to snapshotState [iceberg]

2024-06-17 Thread via GitHub
zhongqishang commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2175073951 > @zhongqishang: Why would this solve the issue? I am trying to solve this problem. This scenario is hard to reproduce. I want to re-express the background of the probl

Re: [I] flink datastream api write iceberg table, finding exception: iceberg Caused by: org.apache.thrift.transport.TTransportException [iceberg]

2024-06-17 Thread via GitHub
pvary commented on issue #10527: URL: https://github.com/apache/iceberg/issues/10527#issuecomment-2174998452 @hashmapybx: Seems like an issue with the HMS communication. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Flink: move flush operation from prepareSnapshotPreBarrier to snapshotState [iceberg]

2024-06-17 Thread via GitHub
pvary commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2174995958 @zhongqishang: Why would this solve the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[I] flink datastream api write iceberg table, finding exception: iceberg Caused by: org.apache.thrift.transport.TTransportException [iceberg]

2024-06-17 Thread via GitHub
hashmapybx opened a new issue, #10527: URL: https://github.com/apache/iceberg/issues/10527 ### Query engine https://github.com/apache/iceberg/assets/26193074/975f0966-a8da-4613-a9ac-220c33194d9a";> ### Question java.lang.RuntimeException: Failed to get table info from

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643733387 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
liurenjie1024 commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643677799 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643673503 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643673503 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] Core: pushdown data_file.content when filter manifests in entries table [iceberg]

2024-06-17 Thread via GitHub
szehon-ho commented on code in PR #10203: URL: https://github.com/apache/iceberg/pull/10203#discussion_r1643661857 ## core/src/main/java/org/apache/iceberg/BaseEntriesTable.java: ## @@ -192,37 +188,46 @@ public Boolean notNaN(BoundReference ref) { @Override publ

Re: [PR] Core: pushdown data_file.content when filter manifests in entries table [iceberg]

2024-06-17 Thread via GitHub
szehon-ho commented on code in PR #10203: URL: https://github.com/apache/iceberg/pull/10203#discussion_r1643654326 ## core/src/main/java/org/apache/iceberg/BaseEntriesTable.java: ## @@ -192,37 +188,46 @@ public Boolean notNaN(BoundReference ref) { @Override publ

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
liurenjie1024 commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1643647234 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174739993 Something like ``` spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ spark.sql.catalog.spark_catalog=org.apache.iceber

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174734872 Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your confi

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2024-06-17 Thread via GitHub
amoghjaha-db commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174733735 Ah sorry I forgot with the SessionCatalog you need to actually call your catalog name "spark_catalog". Please replace AwsDataCatalog in your catalog name in all your configura

Re: [PR] Create rollback and set snapshot APIs [iceberg-python]

2024-06-17 Thread via GitHub
HonahX commented on code in PR #758: URL: https://github.com/apache/iceberg-python/pull/758#discussion_r1643598048 ## pyiceberg/table/__init__.py: ## @@ -2010,6 +2016,84 @@ def create_branch( self._requirements += requirement return self +def rollback_to_

Re: [PR] Create rollback and set snapshot APIs [iceberg-python]

2024-06-17 Thread via GitHub
HonahX commented on code in PR #758: URL: https://github.com/apache/iceberg-python/pull/758#discussion_r1643598048 ## pyiceberg/table/__init__.py: ## @@ -2010,6 +2016,84 @@ def create_branch( self._requirements += requirement return self +def rollback_to_

Re: [I] Deduplication support in RewriteDataFilesAction [iceberg]

2024-06-17 Thread via GitHub
github-actions[bot] commented on issue #2764: URL: https://github.com/apache/iceberg/issues/2764#issuecomment-2174665984 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Token expiration issued when using WORKER_POOL with Spark Thrift Server [iceberg]

2024-06-17 Thread via GitHub
github-actions[bot] commented on issue #2753: URL: https://github.com/apache/iceberg/issues/2753#issuecomment-2174665923 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support TIME_MILLIS in Arrow code [iceberg]

2024-06-17 Thread via GitHub
github-actions[bot] commented on issue #2755: URL: https://github.com/apache/iceberg/issues/2755#issuecomment-2174665951 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
grantatspothero commented on PR #10523: URL: https://github.com/apache/iceberg/pull/10523#issuecomment-2174646888 Found a problem with the approach. This assumption is incorrect for all `SnapshotProducer` > no orphaned manifests could exist if no retries have occurred. This

Re: [PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
grantatspothero commented on code in PR #10523: URL: https://github.com/apache/iceberg/pull/10523#discussion_r1643557930 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -423,21 +423,23 @@ public void commit() { try { LOG.info("Committed snapshot

Re: [PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
grantatspothero commented on code in PR #10523: URL: https://github.com/apache/iceberg/pull/10523#discussion_r1643557191 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -563,6 +565,14 @@ protected boolean canInheritSnapshotId() { return canInheritSnaps

[PR] Bump mkdocs-material from 9.5.26 to 9.5.27 [iceberg-python]

2024-06-17 Thread via GitHub
dependabot[bot] opened a new pull request, #826: URL: https://github.com/apache/iceberg-python/pull/826 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.26 to 9.5.27. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mk

[PR] Bump mypy-boto3-glue from 1.34.126 to 1.34.128 [iceberg-python]

2024-06-17 Thread via GitHub
dependabot[bot] opened a new pull request, #825: URL: https://github.com/apache/iceberg-python/pull/825 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.126 to 1.34.128. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commi

[PR] Bump griffe from 0.45.3 to 0.46.1 [iceberg-python]

2024-06-17 Thread via GitHub
dependabot[bot] opened a new pull request, #824: URL: https://github.com/apache/iceberg-python/pull/824 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 0.45.3 to 0.46.1. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

[PR] Bump urllib3 from 1.26.18 to 1.26.19 [iceberg-python]

2024-06-17 Thread via GitHub
dependabot[bot] opened a new pull request, #823: URL: https://github.com/apache/iceberg-python/pull/823 Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.18 to 1.26.19. Release notes Sourced from https://github.com/urllib3/urllib3/releases";>urllib3's releases. 1

Re: [PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10523: URL: https://github.com/apache/iceberg/pull/10523#discussion_r1643433781 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -423,21 +423,23 @@ public void commit() { try { LOG.info("Committed snapshot {} ({})

Re: [PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10523: URL: https://github.com/apache/iceberg/pull/10523#discussion_r1643432562 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -563,6 +565,14 @@ protected boolean canInheritSnapshotId() { return canInheritSnapshotId;

Re: [PR] Run Flink tests on Java 17 too [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10477: URL: https://github.com/apache/iceberg/pull/10477#issuecomment-2174334087 re: dropping 8 -- i started some work on a side for this: https://github.com/apache/iceberg/pull/10518 Let me know when time comes to ship the changes. -- This is an automated mes

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-17 Thread via GitHub
jbonofre commented on code in PR #10494: URL: https://github.com/apache/iceberg/pull/10494#discussion_r1643348264 ## core/src/main/java/org/apache/iceberg/io/ContentCache.java: ## @@ -147,10 +147,23 @@ public InputFile tryCache(InputFile input) { return input; } + /**

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on PR #9502: URL: https://github.com/apache/iceberg/pull/9502#issuecomment-2174271818 One more thing, we're also leaking V3 spec features into the codebase. Should we guard that with a flag? -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #10494: URL: https://github.com/apache/iceberg/pull/10494#discussion_r1643337565 ## core/src/main/java/org/apache/iceberg/io/ContentCache.java: ## @@ -147,10 +147,23 @@ public InputFile tryCache(InputFile input) { return input; } + /** +

Re: [PR] Remove redundant -XX:+IgnoreUnrecognizedVMOptions [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10475: URL: https://github.com/apache/iceberg/pull/10475#issuecomment-2174255012 thanks @Fokko for the merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Remove redundant -XX:+IgnoreUnrecognizedVMOptions [iceberg]

2024-06-17 Thread via GitHub
Fokko merged PR #10475: URL: https://github.com/apache/iceberg/pull/10475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643326216 ## core/src/test/java/org/apache/iceberg/avro/TestReadDefaultValues.java: ## @@ -0,0 +1,237 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643322829 ## core/src/test/java/org/apache/iceberg/avro/TestReadDefaultValues.java: ## @@ -0,0 +1,237 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[PR] spec: Fix formatting of Default values [iceberg]

2024-06-17 Thread via GitHub
Fokko opened a new pull request, #10525: URL: https://github.com/apache/iceberg/pull/10525 The list doesn't seem to work with a `-`, a `*` seems to do the trick (also used below). ![image](https://github.com/apache/iceberg/assets/1134248/0a853650-6adf-42c8-b1b1-aca3076d1c71) --

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643308001 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -155,6 +162,41 @@ public ValueReader record(Type partner, Schema record, List> f retur

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643308001 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -155,6 +162,41 @@ public ValueReader record(Type partner, Schema record, List> f retur

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643299107 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -155,6 +162,41 @@ public ValueReader record(Type partner, Schema record, List> f retur

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-17 Thread via GitHub
Fokko commented on code in PR #9502: URL: https://github.com/apache/iceberg/pull/9502#discussion_r1643294155 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -155,6 +162,41 @@ public ValueReader record(Type partner, Schema record, List> f retur

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2024-06-17 Thread via GitHub
salahdjb commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2174018221 @amogh-jahagirdar thanks for your reply, I switched to SparkSessionCatalog as you suggested, I'm getting the following error: `Caused by: java.lang.NullPointerException:

Re: [PR] Build: Update NOTICE to include copyright for 2024 [iceberg]

2024-06-17 Thread via GitHub
szehon-ho merged PR #10471: URL: https://github.com/apache/iceberg/pull/10471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Update NOTICE to include copyright for 2024 [iceberg]

2024-06-17 Thread via GitHub
szehon-ho commented on PR #10471: URL: https://github.com/apache/iceberg/pull/10471#issuecomment-2173969027 Merged, thanks @caicancai , and @nastra , @jbonofre for additional review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2173880721 @salahdjb Using the SparkSessionCatalog instead of just the SparkCatalog should work. SparkSessionCatalog will delegate to the built in catalog for non-iceberg tables. Try

[PR] Core: Skip uncommitted manifest cleanup if no retries have occurred [iceberg]

2024-06-17 Thread via GitHub
grantatspothero opened a new pull request, #10523: URL: https://github.com/apache/iceberg/pull/10523 Context from slack thread: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1718381807647999 > We are ingesting streaming data using a java service that does iceberg FastAppend

Re: [PR] Run CI checks on all supported JDKs [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10473: URL: https://github.com/apache/iceberg/pull/10473#issuecomment-2173791978 rebased to resolve conflicts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar commented on PR #10373: URL: https://github.com/apache/iceberg/pull/10373#issuecomment-2173768003 Sorry for the delayed reply everyone, just getting back into this PR. @stevenzwu >I like the introduction of the CleanableFailure marker interface for this purpos

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10494: URL: https://github.com/apache/iceberg/pull/10494#discussion_r1643023850 ## core/src/main/java/org/apache/iceberg/io/ContentCache.java: ## @@ -147,10 +147,23 @@ public InputFile tryCache(InputFile input) { return input; } + /**

Re: [PR] Build: Upgrade to gradle 8.8 [iceberg]

2024-06-17 Thread via GitHub
jbonofre commented on PR #8486: URL: https://github.com/apache/iceberg/pull/8486#issuecomment-2173749565 @nastra as discussed, I identified the problem with `gradle-revapi` plugin: 1. On current `main` branch (with Gradle 8.1.1 and `gradle-revapi` 1.7.0), we can see: ``` Old

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] Require Assertions to be statically imported [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10517: URL: https://github.com/apache/iceberg/pull/10517#issuecomment-2173718901 rebased & updated for https://github.com/apache/iceberg/pull/10500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

[I] `DataFrameWriterV2` fails to write to Iceberg table with schema having nested array[struct] fields [iceberg]

2024-06-17 Thread via GitHub
pasha-i opened a new issue, #10522: URL: https://github.com/apache/iceberg/issues/10522 ### Apache Iceberg version 1.5.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Hello, We have an existing working Spark Scala job (Spark 3.2.0

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-17 Thread via GitHub
jacobmarble commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2173664606 @nastra is there anything else to do on this pull request before merging? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Require Assertions to be statically imported [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10517: URL: https://github.com/apache/iceberg/pull/10517#issuecomment-2173661691 no worries, thanks for reviewing! all the static import were added with intellij's "import statically" action i need to rebase to resolve conflicts with https://github.com/apache/i

Re: [PR] API, Spark 3.3: Remove all usages of deprecated AssertHelpers [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10500: URL: https://github.com/apache/iceberg/pull/10500#issuecomment-2173661631 thanks for the merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
ZENOTME commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642947050 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

[PR] Ignore iceberg-build.properties file loading exception [iceberg]

2024-06-17 Thread via GitHub
EugeneChung opened a new pull request, #10520: URL: https://github.com/apache/iceberg/pull/10520 - The IcebergBuild code says it's not critical that even if it fails to load iceberg-build.properties, it just ignores the failure and set information as unknown. - But every first call of ic

Re: [PR] Build: Upgrade to gradle 8.8 [iceberg]

2024-06-17 Thread via GitHub
jbonofre commented on PR #8486: URL: https://github.com/apache/iceberg/pull/8486#issuecomment-2173602323 NB: the `gradle-revapi` plugin used in this PR is one in preparation containing the detection fix. -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
liurenjie1024 commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642925017 ## crates/iceberg/src/spec/values.rs: ## @@ -2334,9 +2453,20 @@ mod _serde { }, RawLiteralEnum::Float(v) => match ty {

Re: [PR] Spark: Use bulk deletes in rewrite manifests action [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar merged PR #10343: URL: https://github.com/apache/iceberg/pull/10343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Use bulk deletes in rewrite manifests action [iceberg]

2024-06-17 Thread via GitHub
amogh-jahagirdar commented on PR #10343: URL: https://github.com/apache/iceberg/pull/10343#issuecomment-2173588791 Forgot I had this PR up :) thanks for the reviews @nastra @dramaticlly ! I went with just using deleteFiles from the BaseSparkAction, that will also log on failed deletes. Spa

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
liurenjie1024 commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642916388 ## crates/iceberg/src/spec/values.rs: ## @@ -105,6 +108,115 @@ pub struct Datum { literal: PrimitiveLiteral, } +impl Serialize for Datum { +fn seri

Re: [PR] feat: make BoundPredicate,Datum serializable [iceberg-rust]

2024-06-17 Thread via GitHub
liurenjie1024 commented on code in PR #406: URL: https://github.com/apache/iceberg-rust/pull/406#discussion_r1642914543 ## crates/iceberg/src/spec/values.rs: ## @@ -105,6 +108,115 @@ pub struct Datum { literal: PrimitiveLiteral, } +impl Serialize for Datum { +fn seri

Re: [PR] Require Assertions to be statically imported [iceberg]

2024-06-17 Thread via GitHub
nastra commented on PR #10517: URL: https://github.com/apache/iceberg/pull/10517#issuecomment-2173540780 This might take a few days to review as I just made it through the first 100 files -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[PR] Drop support for Java 8 [iceberg]

2024-06-17 Thread via GitHub
findepi opened a new pull request, #10518: URL: https://github.com/apache/iceberg/pull/10518 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Remove redundant -XX:+IgnoreUnrecognizedVMOptions [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10475: URL: https://github.com/apache/iceberg/pull/10475#issuecomment-2173488608 the flag seems to have been added in https://github.com/apache/iceberg/pull/7391/commits/8dd29caf687e2f490b47a9f0519f2514d6b4150c -- This is an automated message from the Apache Git S

Re: [PR] Run revapi workflow on workflow/build system changes [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10485: URL: https://github.com/apache/iceberg/pull/10485#discussion_r1642866065 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Co

Re: [PR] API, Spark 3.3: Remove all usages of deprecated AssertHelpers [iceberg]

2024-06-17 Thread via GitHub
nastra merged PR #10500: URL: https://github.com/apache/iceberg/pull/10500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Encryption integration and test [iceberg]

2024-06-17 Thread via GitHub
ggershinsky commented on code in PR #5544: URL: https://github.com/apache/iceberg/pull/5544#discussion_r1642836390 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -472,10 +501,22 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-06-17 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1642821801 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-17 Thread via GitHub
ajantha-bhat commented on code in PR #10494: URL: https://github.com/apache/iceberg/pull/10494#discussion_r1642819432 ## core/src/main/java/org/apache/iceberg/io/ContentCache.java: ## @@ -147,10 +147,23 @@ public InputFile tryCache(InputFile input) { return input; } +

Re: [PR] Manifest list encryption [iceberg]

2024-06-17 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1642815825 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +201,24 @@ private void cacheManifests(FileIO fileIO) { if (allManifests == null) {

Re: [PR] Manifest list encryption [iceberg]

2024-06-17 Thread via GitHub
ggershinsky commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1642815825 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +201,24 @@ private void cacheManifests(FileIO fileIO) { if (allManifests == null) {

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-06-17 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1642810642 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Run Flink tests on Java 17 too [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10477: URL: https://github.com/apache/iceberg/pull/10477#discussion_r1642808292 ## .github/workflows/spark-ci.yml: ## @@ -69,7 +69,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: -jvm: [8, 11] +jvm: [8, 11, 17]

Re: [PR] Require Assertions to be statically imported [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10517: URL: https://github.com/apache/iceberg/pull/10517#issuecomment-2173382245 > currently based on #10511, #10512 rebased, no longer the case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Import the right assertThatThrownBy method from assertj [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10512: URL: https://github.com/apache/iceberg/pull/10512#issuecomment-2173380090 thank you @nastra for the merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Merge job definitions in spark-ci.yml [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10513: URL: https://github.com/apache/iceberg/pull/10513#issuecomment-2173378709 thank you @nastra for the merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Merge job definitions in spark-ci.yml [iceberg]

2024-06-17 Thread via GitHub
nastra merged PR #10513: URL: https://github.com/apache/iceberg/pull/10513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Fix ancestors_of exception for tables with >=2000 snapshots [iceberg-python]

2024-06-17 Thread via GitHub
ndrluis commented on PR #821: URL: https://github.com/apache/iceberg-python/pull/821#issuecomment-2173350905 Hi @syun64, this is the stacktrace ``` Traceback (most recent call last): File "/Users/magus/Workspace/oss/iceberg-python/test_view.py", line 18, in print(list(

Re: [PR] Rename test helper to deconflict with Assertions [iceberg]

2024-06-17 Thread via GitHub
nastra merged PR #10511: URL: https://github.com/apache/iceberg/pull/10511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Import the right assertThatThrownBy method from assertj [iceberg]

2024-06-17 Thread via GitHub
nastra merged PR #10512: URL: https://github.com/apache/iceberg/pull/10512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Remove all usages of deprecated AssertHelpers [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10500: URL: https://github.com/apache/iceberg/pull/10500#discussion_r1642764522 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -643,10 +642,9 @@ public void testSingleCommitWithRewriteFai

Re: [PR] Run Flink tests on Java 17 too [iceberg]

2024-06-17 Thread via GitHub
nastra commented on code in PR #10477: URL: https://github.com/apache/iceberg/pull/10477#discussion_r1642760847 ## .github/workflows/spark-ci.yml: ## @@ -69,7 +69,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: -jvm: [8, 11] +jvm: [8, 11, 17]

Re: [PR] Require Assertions to be statically imported [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10517: URL: https://github.com/apache/iceberg/pull/10517#issuecomment-2173282414 currently based on https://github.com/apache/iceberg/pull/10511, https://github.com/apache/iceberg/pull/10512 -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Remove all usages of deprecated AssertHelpers [iceberg]

2024-06-17 Thread via GitHub
findepi commented on code in PR #10500: URL: https://github.com/apache/iceberg/pull/10500#discussion_r1642744095 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -688,23 +688,23 @@ public void invalidDataImport()

Re: [PR] Flink 1.17: Supports batch queries using time ranges [iceberg]

2024-06-17 Thread via GitHub
pvary commented on PR #7362: URL: https://github.com/apache/iceberg/pull/7362#issuecomment-2173276028 Hopefully one last comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Flink 1.17: Supports batch queries using time ranges [iceberg]

2024-06-17 Thread via GitHub
pvary commented on code in PR #7362: URL: https://github.com/apache/iceberg/pull/7362#discussion_r1642738841 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java: ## @@ -146,6 +185,7 @@ enum ScanMode { @VisibleForTesting static ScanMode

Re: [I] iceberg support tiimestamp in nanosecond (timestamp_ns data type) [iceberg]

2024-06-17 Thread via GitHub
anuhewa commented on issue #10510: URL: https://github.com/apache/iceberg/issues/10510#issuecomment-2173270785 @nastra Noted and thanks a lot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Add support for orc format [iceberg-python]

2024-06-17 Thread via GitHub
MehulBatra commented on code in PR #790: URL: https://github.com/apache/iceberg-python/pull/790#discussion_r1642734352 ## pyiceberg/io/pyarrow.py: ## @@ -912,6 +916,9 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType: return TimestamptzType()

Re: [PR] Flink: handle rescale properly and refactor statistics [iceberg]

2024-06-17 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1642704253 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/RangePartitioner.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Test spark-ci.yml run on spark-ci.yml changes [iceberg]

2024-06-17 Thread via GitHub
findepi closed pull request #10516: Test spark-ci.yml run on spark-ci.yml changes URL: https://github.com/apache/iceberg/pull/10516 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] Test spark-ci.yml run on spark-ci.yml changes [iceberg]

2024-06-17 Thread via GitHub
findepi opened a new pull request, #10516: URL: https://github.com/apache/iceberg/pull/10516 debug why https://github.com/apache/iceberg/pull/10481 did have spark-ci run -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Fix CI script inclusion of release branches (v2) [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10515: URL: https://github.com/apache/iceberg/pull/10515#issuecomment-2173139218 Once this is merged, the changes need to be backported to existing active 1.x.x branches. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Fix CI script inclusion of release branches [iceberg]

2024-06-17 Thread via GitHub
findepi commented on PR #10514: URL: https://github.com/apache/iceberg/pull/10514#issuecomment-2173138295 Draft -- would prefer https://github.com/apache/iceberg/pull/10515 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

  1   2   >