Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1577300578 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc,

Re: [PR] Build: Bump sqlalchemy from 2.0.27 to 2.0.29 [iceberg-python]

Fokko merged PR #645: URL: https://github.com/apache/iceberg-python/pull/645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

pvary commented on PR #10188: URL: https://github.com/apache/iceberg/pull/10188#issuecomment-2074024030 @ludlows: Could you please add a unit test as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

pvary commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1577249432 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc,

Re: [PR] Build: Bump pyparsing from 3.1.1 to 3.1.2 [iceberg-python]

Fokko merged PR #652: URL: https://github.com/apache/iceberg-python/pull/652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows commented on PR #10188: URL: https://github.com/apache/iceberg/pull/10188#issuecomment-2073942683 @pvary could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

danielcweeks commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2073903547 I'm not sure I follow the reasoning here. At the point your creating an issue, you shouldn't try to define subtasks as nothing has been decided. You can always go back and edit

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

ajantha-bhat commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2073868413 ping @danielcweeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add the build from source section [iceberg-go]

git-hulk commented on code in PR #70: URL: https://github.com/apache/iceberg-go/pull/70#discussion_r1577141430 ## README.md: ## @@ -21,6 +21,19 @@ `iceberg` is a Golang implementation of the [Iceberg table spec](https://iceberg.apache.org/spec/). +## Build From Source +

Re: [I] Implement size-based task planning [iceberg-rust]

liurenjie1024 commented on issue #344: URL: https://github.com/apache/iceberg-rust/issues/344#issuecomment-2073860929 > I am looking into this issue. @liurenjie1024, can you please assign it to me? Thanks! Done, thanks! -- This is an automated message from the Apache Git Service.

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

liurenjie1024 commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1577106941 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

liurenjie1024 commented on PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#issuecomment-2073844403 > There are intentionally a lot of `todo!`s in here. The aim is to get this PR merged so that more people can contribute implementations for the different visitor methods that

Re: [PR] Add Pagination To List Apis [iceberg]

rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1577061740 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [I] Improve hidden partition support for Hive CREATE TABLE [iceberg]

github-actions[bot] commented on issue #2681: URL: https://github.com/apache/iceberg/issues/2681#issuecomment-2073690420 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] NPE while processing data in flink job [iceberg]

github-actions[bot] closed issue #2471: NPE while processing data in flink job URL: https://github.com/apache/iceberg/issues/2471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests [iceberg]

github-actions[bot] commented on issue #2474: URL: https://github.com/apache/iceberg/issues/2474#issuecomment-2073690227 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests [iceberg]

github-actions[bot] closed issue #2474: Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests URL: https://github.com/apache/iceberg/issues/2474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] NPE while processing data in flink job [iceberg]

github-actions[bot] commented on issue #2471: URL: https://github.com/apache/iceberg/issues/2471#issuecomment-2073690197 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] NPE while closing OrcFileAppender [iceberg]

github-actions[bot] commented on issue #2470: URL: https://github.com/apache/iceberg/issues/2470#issuecomment-2073690169 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] NPE while closing OrcFileAppender [iceberg]

github-actions[bot] closed issue #2470: NPE while closing OrcFileAppender URL: https://github.com/apache/iceberg/issues/2470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577030981 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/BaseDeltaTaskWriter.java: ## @@ -56,7 +58,8 @@ abstract class BaseDeltaTaskWriter extends

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577029896 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -109,18 +112,30 @@ protected abstract class BaseEqualityDeltaWriter implements Closeable

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577030203 ## core/src/main/java/org/apache/iceberg/deletes/ContinuousFileScopedPositionDeleteWriter.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577029896 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -109,18 +112,30 @@ protected abstract class BaseEqualityDeltaWriter implements Closeable

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577028939 ## core/src/main/java/org/apache/iceberg/deletes/ContinuousFileScopedPositionDeleteWriter.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

chenjunjiedada commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1576141345 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteConf.java: ## @@ -167,6 +168,18 @@ public DistributionMode distributionMode() {

[PR] Build: Bump pyparsing from 3.1.1 to 3.1.2 [iceberg-python]

dependabot[bot] opened a new pull request, #652: URL: https://github.com/apache/iceberg-python/pull/652 Bumps [pyparsing](https://github.com/pyparsing/pyparsing) from 3.1.1 to 3.1.2. Changelog Sourced from https://github.com/pyparsing/pyparsing/blob/master/CHANGES;>pyparsing's

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576706517 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() {

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

geruh commented on PR #650: URL: https://github.com/apache/iceberg-python/pull/650#issuecomment-2073414087 I believe in the Java implementation we have a concept of a

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576805382 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576800710 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,459 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576798073 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

[PR] [WIP]: Add `InclusiveMetricsEvaluator` [iceberg-rust]

sdd opened a new pull request, #347: URL: https://github.com/apache/iceberg-rust/pull/347 `InclusiveMetricsEvaluator` is used inside table scans to filter `DataFile` entries within a Manifest, rejecting any of them if their metrics indicate that they cannot contain any rows that match the

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on PR #59: URL: https://github.com/apache/iceberg-go/pull/59#issuecomment-2073185271 any updates @wolfeidau? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add the build from source section [iceberg-go]

zeroshade commented on code in PR #70: URL: https://github.com/apache/iceberg-go/pull/70#discussion_r1576744685 ## README.md: ## @@ -21,6 +21,19 @@ `iceberg` is a Golang implementation of the [Iceberg table spec](https://iceberg.apache.org/spec/). +## Build From Source +

Re: [PR] Improve CLI Text by Adding Verbose Text for Commands [iceberg-go]

zeroshade commented on PR #68: URL: https://github.com/apache/iceberg-go/pull/68#issuecomment-2073179846 @rtadepalli I like the idea in general, but could you address my question when you have a chance? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576706517 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() {

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576699869 ## core/src/main/java/org/apache/iceberg/DataTaskParser.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

pvary commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1576658083 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -155,6 +155,13 @@ private void validate() {

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

pvary commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1576638673 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -40,24 +44,27 @@ import org.apache.iceberg.FileFormat;

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

stevenzwu commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1576603768 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -155,6 +155,13 @@ private void validate() {

[I] Add support for BigLake Metastore [iceberg-python]

sashkaw opened a new issue, #651: URL: https://github.com/apache/iceberg-python/issues/651 ### Feature Request / Improvement Feature request to add support for loading and interacting with catalogs from BigLake Metastore. -- This is an automated message from the Apache Git

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

stevenzwu commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1576552272 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -201,8 +201,11 @@ private SplitEnumerator createEnumer return

[PR] feat: add `RollingManifestWriter` [iceberg-python]

felixscherz opened a new pull request, #650: URL: https://github.com/apache/iceberg-python/pull/650 Hi, this is in regards to #596 and still WIP. The `RollingManifestWriter` implementation closely follows the java

Re: [I] Implement partition writer [iceberg-rust]

gupteaj commented on issue #342: URL: https://github.com/apache/iceberg-rust/issues/342#issuecomment-2072845233 @liurenjie1024 , I can look into one of this task , can you please assign me ? -- This is an automated message from the Apache Git Service. To respond to the message, please

[I] Tracking:: Writing iceberg tables [iceberg-rust]

ZENOTME opened a new issue, #346: URL: https://github.com/apache/iceberg-rust/issues/346 Inspired by https://github.com/apache/iceberg-rust/pull/275#issuecomment-2041304545, I created this issue to track all our write task. It's based on

Re: [I] Implement the position delete writer [iceberg-rust]

ZENOTME commented on issue #340: URL: https://github.com/apache/iceberg-rust/issues/340#issuecomment-2072825313 > Hi @ZENOTME, can I take on either this task or #341? Sure! Thanks! Before we move to more writers, maybe we can complete #345 first so that we can add the e2e test for

Re: [I] Support to append file on table [iceberg-rust]

ZENOTME commented on issue #345: URL: https://github.com/apache/iceberg-rust/issues/345#issuecomment-2072819533 I can send a PR for this later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1576411506 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [I] Implement size-based task planning [iceberg-rust]

s-akhtar-baig commented on issue #344: URL: https://github.com/apache/iceberg-rust/issues/344#issuecomment-2072785869 I am looking into this issue. @liurenjie1024, can you please assign it to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1576369286 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//, +

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

liurenjie1024 commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2072455753 I've compiled a [doc](https://docs.google.com/document/d/1YncDX-qQ1T9jBGQmJNtRcPU1trRi00cB8eykv5diKw4/edit?usp=sharing) for discussing roadmaps and features for

Re: [I] Implement the equality delta writer [iceberg-rust]

ZENOTME commented on issue #343: URL: https://github.com/apache/iceberg-rust/issues/343#issuecomment-2072374369 > Hi, @ZENOTME Since we have finished the framework, I'm thinking about providing a high level api such as [append in

Re: [I] Implement the equality delta writer [iceberg-rust]

liurenjie1024 commented on issue #343: URL: https://github.com/apache/iceberg-rust/issues/343#issuecomment-2072357168 Hi, @ZENOTME Since we have finished the framework, I'm thinking about providing a high level api such as [append in

Re: [I] Implement the position delete writer [iceberg-rust]

s-akhtar-baig commented on issue #340: URL: https://github.com/apache/iceberg-rust/issues/340#issuecomment-2072353948 Hi @ZENOTME, can I take on either this task or https://github.com/apache/iceberg-rust/issues/341? -- This is an automated message from the Apache Git Service. To respond

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2072339019 Thank you for your suggestion @geruh I'll test this patch manually and update the class. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576036541 ## core/src/main/java/org/apache/iceberg/ScanTaskParser.java: ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576026814 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() {

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576025475 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() {

Re: [PR] Flink: Move ParquetReader to LogicalTypeAnnotationVisitor [iceberg]

pvary commented on PR #9719: URL: https://github.com/apache/iceberg/pull/9719#issuecomment-2071950025 Thanks @Fokko for the PR! Please do not forget the PR for the other Flink versions as well. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576019460 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -40,16 +40,39 @@ public class FileScanTaskParser { private FileScanTaskParser() {} + /**

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576017301 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -40,16 +40,39 @@ public class FileScanTaskParser { private FileScanTaskParser() {} + /**

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary commented on PR #10148: URL: https://github.com/apache/iceberg/pull/10148#issuecomment-2071944795 Thanks for @stargrey102 for the PR and @deniskuzZ for the help during the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary merged PR #10148: URL: https://github.com/apache/iceberg/pull/10148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576014466 ## core/src/main/java/org/apache/iceberg/DataTaskParser.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576011008 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576012292 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576012052 ## core/src/test/java/org/apache/iceberg/TestDataTaskParser.java: ## @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576011200 ## core/src/test/java/org/apache/iceberg/TestDataTaskParser.java: ## @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576007535 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1575985773 ## format/spec.md: ## @@ -128,12 +128,12 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575985174 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1575985072 ## format/spec.md: ## @@ -128,12 +128,12 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575982248 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575983673 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java: ## @@ -84,7 +85,8 @@ public RowDataRewriter( format,

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

lintingbin commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2071811784 > I see, is it because of the dot character? I havent looked deeply at the problem but both solutions make sense, I think the first one may be preferable because then there is

[I] REST Catalog to support custom-catalog like HMS/Glue [iceberg]

osscm opened a new issue, #10205: URL: https://github.com/apache/iceberg/issues/10205 ### Query engine Trino ### Question We are thinking it would be good feature to support custom catalog name in REST Catalog. Before creating an issue, thought will discuss here, as

Re: [I] does iceberg can run on k8s? [iceberg]

JunseoChoJJ commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071683150 @nastra Thanks a lot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] does iceberg can run on k8s? [iceberg]

nastra commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071634712 If Naver Cloud Storage has an S3-compatible API, then you can use it with Iceberg by using `S3FileIO` in your Spark catalog config: `--conf

Re: [PR] Add Pagination To List Apis [iceberg]

nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1575784851 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [I] does iceberg can run on k8s? [iceberg]

JunseoChoJJ commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071620951 @nastra I am using naver cloud storage. it is same like s3, gcs. I want to figure out besides using s3, gcs, or minio. Can I use other storage to configure with iceberg --

Re: [PR] Add Pagination To List Apis [iceberg]

nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1575784851 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Core: FileSystemTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1575783911 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -21,9 +21,12 @@ import static org.apache.iceberg.TableProperties.COMMIT_NUM_RETRIES;

[PR] Fix incorrect metrics calculation for iceberg table due to column name transformation with special characters. [iceberg]

lintingbin opened a new pull request, #10204: URL: https://github.com/apache/iceberg/pull/10204 Here are the main changes to fix [#2623](https://github.com/apache/iceberg/issues/10115) 1. change MetricsConfig.columnModes from map[colName, MetricsMode] to map[fieldId, MetricsMode] 2.

Re: [PR] Build: Bump deptry from 0.14.0 to 0.16.1 [iceberg-python]

Fokko merged PR #649: URL: https://github.com/apache/iceberg-python/pull/649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] does iceberg can run on k8s? [iceberg]

nastra commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071584139 > Can I put this data into our Company storage as well using rest api? I'm not exactly sure I understand. Are you trying to replicate the data that lands in S3/GCS/Minio? Can

Re: [I] does iceberg can run on k8s? [iceberg]

nastra closed issue #10196: does iceberg can run on k8s? URL: https://github.com/apache/iceberg/issues/10196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Core: FileSystemTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1575749329 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -21,9 +21,12 @@ import static org.apache.iceberg.TableProperties.COMMIT_NUM_RETRIES;

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1575736834 ## core/src/main/java/org/apache/iceberg/BaseOverwriteFiles.java: ## @@ -48,6 +48,10 @@ protected OverwriteFiles self() { @Override protected String

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

nastra commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1575735207 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -101,8 +101,8 @@ public class TestOverwrite extends TestBase {

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575706611 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -140,7 +155,9 @@ public void write(T row) throws IOException { PathOffset previous =

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

chenjunjiedada commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575698957 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575700012 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575698414 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException