Re: [PR] Update table metadata [iceberg-python]

2023-11-21 Thread via GitHub
HonahX commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1401641335 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] Update table metadata [iceberg-python]

2023-11-21 Thread via GitHub
HonahX commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1401631738 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [I] Failed to create namespace using spark sql based on iceberg hadoop catalog (rest catalog) [iceberg]

2023-11-21 Thread via GitHub
TCGOGOGO commented on issue #9072: URL: https://github.com/apache/iceberg/issues/9072#issuecomment-1822265704 > Hey @TCGOGOGO Can you provide a bit more context here? The Hadoop Catalog and Rest Catalog are two different catalogs, and they should not be combined. Hi Fokko, thanks for

Re: [PR] Update table metadata [iceberg-python]

2023-11-21 Thread via GitHub
HonahX commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1401631738 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] Python: Add support for Python 3.12 [iceberg-python]

2023-11-21 Thread via GitHub
Fokko commented on PR #35: URL: https://github.com/apache/iceberg-python/pull/35#issuecomment-1822251221 Thanks for picking this up again, it looks like it is now blocked on Poetry: https://github.com/apache/iceberg-python/pull/158 -- This is an automated message from the Apache Git Servi

Re: [PR] Bump Poetry to 1.7.1 [iceberg-python]

2023-11-21 Thread via GitHub
Fokko commented on PR #158: URL: https://github.com/apache/iceberg-python/pull/158#issuecomment-1822248447 Looks like the poetry version is blocking https://github.com/apache/iceberg-python/pull/35 -- This is an automated message from the Apache Git Service. To respond to the message, ple

[PR] Bump Poetry to 1.7.1 [iceberg-python]

2023-11-21 Thread via GitHub
Fokko opened a new pull request, #158: URL: https://github.com/apache/iceberg-python/pull/158 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Replace i64 with DateTime [iceberg-rust]

2023-11-21 Thread via GitHub
liurenjie1024 commented on PR #94: URL: https://github.com/apache/iceberg-rust/pull/94#issuecomment-1822159410 cc @Fokko Any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]

2023-11-21 Thread via GitHub
liurenjie1024 commented on PR #97: URL: https://github.com/apache/iceberg-rust/pull/97#issuecomment-1822159124 cc @Fokko I've fixed comment, PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Spark: Add serialzable isolation test for concurrent MERGE INTOs [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9050: URL: https://github.com/apache/iceberg/pull/9050#discussion_r1401516134 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -390,7 +390,7 @@ private Expression conflictDetectionFilter() { filter

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1822113350 Thanks for your patience, @przemekd! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi merged PR #9029: URL: https://github.com/apache/iceberg/pull/9029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] locationProvider should be marked as transient in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi closed issue #9025: locationProvider should be marked as transient in SerializableTable URL: https://github.com/apache/iceberg/issues/9025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-21 Thread via GitHub
lisirrx commented on code in PR #9122: URL: https://github.com/apache/iceberg/pull/9122#discussion_r1401459825 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/AliyunOSSExtension.java: ## @@ -39,18 +39,13 @@ default String testBucketName() { } @Override - default S

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-21 Thread via GitHub
lisirrx commented on code in PR #9122: URL: https://github.com/apache/iceberg/pull/9122#discussion_r1401456890 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/AliyunOSSExtension.java: ## @@ -39,18 +39,13 @@ default String testBucketName() { } @Override - default S

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-11-21 Thread via GitHub
liurenjie1024 commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1401435068 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
cgpoh commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1401434184 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestCatalogTblLoader.java: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
cgpoh commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1401413672 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/FlinkTestBaseUtils.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Core: Make the default partition name consistent when creating tables and update PartitionSpec [iceberg]

2023-11-21 Thread via GitHub
shidayang closed pull request #9121: Core: Make the default partition name consistent when creating tables and update PartitionSpec URL: https://github.com/apache/iceberg/pull/9121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Parquet: Support reading INT96 column in row group filter [iceberg]

2023-11-21 Thread via GitHub
manuzhang commented on PR #8988: URL: https://github.com/apache/iceberg/pull/8988#issuecomment-1821967428 @RussellSpitzer @aokolnychyi could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[I] The "Status" paragraph in the readme seems very outdated [iceberg]

2023-11-21 Thread via GitHub
ronkorving opened a new issue, #9127: URL: https://github.com/apache/iceberg/issues/9127 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 Looking at the ReadMe in the root of this repository, one would be mistaken for thin

Re: [I] Rewrite with zORDER results in 'Cannot find field 'ICEZVALUE' in struct' Error [iceberg]

2023-11-21 Thread via GitHub
github-actions[bot] commented on issue #7586: URL: https://github.com/apache/iceberg/issues/7586#issuecomment-1821890805 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Rewrite with zORDER results in 'Cannot find field 'ICEZVALUE' in struct' Error [iceberg]

2023-11-21 Thread via GitHub
github-actions[bot] closed issue #7586: Rewrite with zORDER results in 'Cannot find field 'ICEZVALUE' in struct' Error URL: https://github.com/apache/iceberg/issues/7586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Apply spotless for scala code [iceberg]

2023-11-21 Thread via GitHub
github-actions[bot] commented on issue #7695: URL: https://github.com/apache/iceberg/issues/7695#issuecomment-1821890754 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[PR] Build: Bump mkdocstrings-python from 1.7.4 to 1.7.5 [iceberg-python]

2023-11-21 Thread via GitHub
dependabot[bot] opened a new pull request, #157: URL: https://github.com/apache/iceberg-python/pull/157 Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.7.4 to 1.7.5. Release notes Sourced from https://github.com/mkdocstrings/python/releases";>mkdocstrings

Re: [I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
singhpk234 commented on issue #9124: URL: https://github.com/apache/iceberg/issues/9124#issuecomment-1821822583 i my opinion we can re-use what existing impl have for s3 for ex s3A it's not a new problem to be honest : can ref this how s3A handles this : https://github.com/a

Re: [I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
lognoel commented on issue #9124: URL: https://github.com/apache/iceberg/issues/9124#issuecomment-1821816513 Yeah the s3 exception categories are quite broad: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#listBuckets-- -- This is an automate

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
przemekd commented on code in PR #9029: URL: https://github.com/apache/iceberg/pull/9029#discussion_r1401297380 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -237,7 +236,14 @@ public EncryptionManager encryption() { @Override public LocationProvi

Re: [I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
singhpk234 commented on issue #9124: URL: https://github.com/apache/iceberg/issues/9124#issuecomment-1821811296 This is interesting i thought we would be thrown clear exceptions like we get from Glue service model : https://github.com/apache/iceberg/blob/c61c3ca017aaec145f57948f0da9c

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi commented on code in PR #9029: URL: https://github.com/apache/iceberg/pull/9029#discussion_r1401282710 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -237,7 +236,14 @@ public EncryptionManager encryption() { @Override public LocationPr

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1821791818 You are right, the data analyst doesn't have to create new locations in that case. I am convinced now. Let me review the change itself. -- This is an automated message from the A

Re: [PR] Spark: Add serialzable isolation test for concurrent MERGE INTOs [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi commented on code in PR #9050: URL: https://github.com/apache/iceberg/pull/9050#discussion_r1401275467 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -390,7 +390,7 @@ private Expression conflictDetectionFilter() { fi

[PR] API, Core: Optimize CharSequenceMap for file paths [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi opened a new pull request, #9126: URL: https://github.com/apache/iceberg/pull/9126 I am using `CharSequenceMap` in #8755 to build a map of position delete indexes for a delete file. While profiling that change, I noticed we spend quite a bit of time computing hash codes for file

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401243005 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.proj

Re: [I] Question about "Column ID" in Parquet Sepc [iceberg]

2023-11-21 Thread via GitHub
emkornfield commented on issue #9099: URL: https://github.com/apache/iceberg/issues/9099#issuecomment-1821629357 Yes column IDs are meant to be [field ids](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L459) -- This is an automated message from the Ap

Re: [I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
lognoel commented on issue #9124: URL: https://github.com/apache/iceberg/issues/9124#issuecomment-1821623948 > for: > > 1. i think we can allowlist the exception which could be access denied (or the iceberg equivalent exception bubbled up), we need to stop retry on > ht

Re: [PR] Data: Always use delete index for position deletes [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi commented on PR #9117: URL: https://github.com/apache/iceberg/pull/9117#issuecomment-1821621461 Thank you for reviewing, @amogh-jahagirdar @nastra @singhpk234! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Data: Always use delete index for position deletes [iceberg]

2023-11-21 Thread via GitHub
aokolnychyi merged PR #9117: URL: https://github.com/apache/iceberg/pull/9117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[PR] Spec: Clarify partition equality [iceberg]

2023-11-21 Thread via GitHub
emkornfield opened a new pull request, #9125: URL: https://github.com/apache/iceberg/pull/9125 Based on conversation on mailing list: https://lists.apache.org/thread/z5y4qgf1hpc5y5krptb71lx0q1yx9gtx I made an assumption on how floating point partitions should be considered for equali

Re: [I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
singhpk234 commented on issue #9124: URL: https://github.com/apache/iceberg/issues/9124#issuecomment-1821597462 for: 1. i think we can allowlist the exception which could be access denied, we need to stop retry on https://github.com/apache/iceberg/blob/f8d21116b1990ff0f7d7960a0f41f3

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
dchristle commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401083561 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
dchristle commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401083561 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
dchristle commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1401083561 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

[I] `GlueTableOperations` retries on Access Denied exceptions from S3, and does not support configuration of exception retry logic [iceberg]

2023-11-21 Thread via GitHub
lognoel opened a new issue, #9124: URL: https://github.com/apache/iceberg/issues/9124 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 `GlueCatalog.loadTable` can block for 100 seconds if attempting to re

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r140147 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Softw

Re: [I] Multiple Catalog Connection [iceberg-python]

2023-11-21 Thread via GitHub
Fokko commented on issue #156: URL: https://github.com/apache/iceberg-python/issues/156#issuecomment-1821477059 Hey @asheeshgarg You can already do this. The configuration in `~/.pyiceberg.yaml`: ```yaml catalog: hive: uri: thrift://127.0.0.1:9083 s3

[I] Multiple Catalog Connection [iceberg-python]

2023-11-21 Thread via GitHub
asheeshgarg opened a new issue, #156: URL: https://github.com/apache/iceberg-python/issues/156 ### Question From pyiceberg if I want to connect to multiple catalogs we need to create a seperate instance per catalog or there is option to specify multiple catalogs as part of the config

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
stevenzwu commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400990227 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -453,6 +492,18 @@ public IcebergSource build() { contextBuilder.

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400963691 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/MiniFlinkClusterResource.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400963691 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/MiniFlinkClusterResource.java: ## @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400961872 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/FlinkTestBaseUtils.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400960758 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestCatalogTblLoader.java: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-21 Thread via GitHub
nastra commented on code in PR #9122: URL: https://github.com/apache/iceberg/pull/9122#discussion_r1400943843 ## build.gradle: ## @@ -17,7 +17,9 @@ * under the License. */ + Review Comment: unnecessary changes here and below ## aliyun/src/test/java/org/apach

Re: [I] TBLPROPERTIES('history.expire.max-snapshot-age-ms') doesn't work [iceberg]

2023-11-21 Thread via GitHub
nastra commented on issue #9123: URL: https://github.com/apache/iceberg/issues/9123#issuecomment-1821364508 I'm not sure if people that work on the Iceberg Hive integration actually monitor Iceberg issues, because the code for this was introduced as part of https://issues.apache.org/jira/br

Re: [I] TBLPROPERTIES('history.expire.max-snapshot-age-ms') doesn't work [iceberg]

2023-11-21 Thread via GitHub
jkcldr commented on issue #9123: URL: https://github.com/apache/iceberg/issues/9123#issuecomment-1821312908 @nastra I ran the ALTER TABLE query against the hive engine as mentioned in step 9. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] TBLPROPERTIES('history.expire.max-snapshot-age-ms') doesn't work [iceberg]

2023-11-21 Thread via GitHub
nastra commented on issue #9123: URL: https://github.com/apache/iceberg/issues/9123#issuecomment-1821262798 @jkcldr did you run https://iceberg.apache.org/docs/latest/spark-procedures/#expire_snapshots? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400859332 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/FlinkTestBaseUtils.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400851908 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestCatalogTblLoader.java: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9120: URL: https://github.com/apache/iceberg/pull/9120#discussion_r1400850567 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestCatalogTblLoader.java: ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] Create JUnit5-version of SparkCatalogTestBase [iceberg]

2023-11-21 Thread via GitHub
nastra commented on issue #9076: URL: https://github.com/apache/iceberg/issues/9076#issuecomment-1821199246 Nobody is working on this, so feel free to comment on such an issue to signal interest and open a PR. Thanks -- This is an automated message from the Apache Git Service. To respond

[I] TBLPROPERTIES('history.expire.max-snapshot-age-ms') doesn't work [iceberg]

2023-11-21 Thread via GitHub
jkcldr opened a new issue, #9123: URL: https://github.com/apache/iceberg/issues/9123 ### Feature Request / Improvement The 'history.expire.max-snapshot-age-ms' option doesn't have any effect while expiring snapshots. 1) CREATE TABLE IF NOT EXISTS test5d78b6 ( id INT,

Re: [I] Create JUnit5-version of SparkCatalogTestBase [iceberg]

2023-11-21 Thread via GitHub
tomtongue commented on issue #9076: URL: https://github.com/apache/iceberg/issues/9076#issuecomment-1821150653 @nastra If no one is working on this, may I create a PR for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Aliyun: Switch iceberg-aliyun's tests to Junit5 [iceberg]

2023-11-21 Thread via GitHub
lisirrx opened a new pull request, #9122: URL: https://github.com/apache/iceberg/pull/9122 Switch iceberg-aliyun's tests to Junit5 Fix: #9081 @nastra Hi, I have changed all of the junit4 code to junit5 with AssertJ, but I have a question here: To fit the Extension class of Junit5, I c

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-21 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400604336 ## nessie/src/test/java/org/apache/iceberg/nessie/TestBranchVisibilityForView.java: ## @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
cgpoh commented on PR #9120: URL: https://github.com/apache/iceberg/pull/9120#issuecomment-1820796937 @pvary , thanks! Will update the class name and revert the changes for Flink 1.15 and 1.16 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-21 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1400469049 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + publ

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

2023-11-21 Thread via GitHub
przemekd commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1820779641 @aokolnychyi We assume the data analyst uses a completely separate cluster, JAR is not populated on executors and not supplied with the job. With the fix provided she still can do her jo

[PR] Core: Make the default partition name consistent when creating tables and update PartitionSpec [iceberg]

2023-11-21 Thread via GitHub
shidayang opened a new pull request, #9121: URL: https://github.com/apache/iceberg/pull/9121 When I creating Table Using ``` PartitionSpec.builderFor(schema) .bucket("data", 16) .build(); ``` The default partition name is 'data_bucket'

Re: [PR] Hive: Refactor HiveTableOperations with common code for View. [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9011: URL: https://github.com/apache/iceberg/pull/9011#discussion_r1400329609 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -207,11 +195,15 @@ protected void doCommit(TableMetadata base, TableMetadata meta

Re: [PR] Docs: Remove UNIQUE keyword as it is not supported in Flink [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9046: URL: https://github.com/apache/iceberg/pull/9046#discussion_r1400431652 ## docs/flink-writes.md: ## @@ -69,7 +69,7 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab ```sql CREATE TABLE `hive_catalog`.`de

Re: [PR] Create JUnit5 version of FlinkTestBase [iceberg]

2023-11-21 Thread via GitHub
pvary commented on PR #9120: URL: https://github.com/apache/iceberg/pull/9120#issuecomment-1820705097 @cgpoh: We usually only create a PR for a single version of Flink (latest). After that PR is merged, then we create a backport PR to get the changes to the older versions. This helps the re

Re: [PR] Hive: Refactor HiveTableOperations with common code for View. [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #9011: URL: https://github.com/apache/iceberg/pull/9011#discussion_r1400329609 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -207,11 +195,15 @@ protected void doCommit(TableMetadata base, TableMetadata meta

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-11-21 Thread via GitHub
mobley-trent commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1400322448 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400301510 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400299663 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,377 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400292149 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400292444 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400291513 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Update Slack URL [iceberg-docs]

2023-11-21 Thread via GitHub
jbonofre commented on PR #294: URL: https://github.com/apache/iceberg-docs/pull/294#issuecomment-1820539312 Sorry for the late reply, I tested the link and it looks good. Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400220361 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -0,0 +1,359 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400207577 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailoverWithWatermarkExtractor.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apac

Re: [PR] Core: Widen exceptions ignored while deleting files in RollingFileWriter [iceberg]

2023-11-21 Thread via GitHub
glawson0 commented on code in PR #8597: URL: https://github.com/apache/iceberg/pull/8597#discussion_r1400203231 ## core/src/main/java/org/apache/iceberg/io/RollingFileWriter.java: ## @@ -127,9 +130,8 @@ private void closeCurrentWriter() { if (currentFileRows == 0L) {

Re: [PR] Flink: Emit watermarks from the IcebergSource [iceberg]

2023-11-21 Thread via GitHub
pvary commented on code in PR #8553: URL: https://github.com/apache/iceberg/pull/8553#discussion_r1400201726 ## data/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java: ## @@ -120,6 +121,10 @@ private static DataFile appendToLocalFile( appenderFactory.setAll

Re: [I] PARSE_SYNTAX_ERROR was encountered for 'ALTER TABLE ... CREATE BRANCH ... [iceberg]

2023-11-21 Thread via GitHub
jackskmok closed issue #9116: PARSE_SYNTAX_ERROR was encountered for 'ALTER TABLE ... CREATE BRANCH ... URL: https://github.com/apache/iceberg/issues/9116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] PARSE_SYNTAX_ERROR was encountered for 'ALTER TABLE ... CREATE BRANCH ... [iceberg]

2023-11-21 Thread via GitHub
jackskmok commented on issue #9116: URL: https://github.com/apache/iceberg/issues/9116#issuecomment-1820454824 Thanks @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment