Re: [PR] feat: support lower_bound&&upper_bound for parquet writer [iceberg-rust]

2024-06-06 Thread via GitHub
sdd commented on code in PR #383: URL: https://github.com/apache/iceberg-rust/pull/383#discussion_r1630721046 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -78,106 +89,407 @@ impl FileWriterBuilder for ParquetWr type R = ParquetWriter; async fn b

Re: [PR] feat: support lower_bound&&upper_bound for parquet writer [iceberg-rust]

2024-06-06 Thread via GitHub
sdd commented on code in PR #383: URL: https://github.com/apache/iceberg-rust/pull/383#discussion_r1630709857 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -78,106 +89,407 @@ impl FileWriterBuilder for ParquetWr type R = ParquetWriter; async fn b

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
RussellSpitzer commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2154176716 Looking briefly at the code it looks like the error may be in the retry logic for the transaction statements as written. From what I can tell the code does not handle Iceber

Re: [PR] feat: support lower_bound&&upper_bound for parquet writer [iceberg-rust]

2024-06-06 Thread via GitHub
sdd commented on code in PR #383: URL: https://github.com/apache/iceberg-rust/pull/383#discussion_r1630700456 ## crates/iceberg/src/writer/base_writer/data_file_writer.rs: ## @@ -123,195 +126,35 @@ mod test { location_generator::{test::MockLocationGenerator, De

Re: [I] Issue with 'writeTo' [iceberg]

2024-06-06 Thread via GitHub
zero1zero commented on issue #9766: URL: https://github.com/apache/iceberg/issues/9766#issuecomment-2154025618 What was the bug in your code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630345753 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatistics.java: ## @@ -35,28 +35,28 @@ class AggregatedStatistics implements Ser

Re: [PR] Core: Use Failsafe in ClientPoolImpl retry logic [iceberg]

2024-06-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #10458: URL: https://github.com/apache/iceberg/pull/10458#discussion_r1630572426 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -67,30 +71,27 @@ public R run(Action action) throws E, InterruptedException { @Ove

[PR] Core: Use Failsafe in ClientPoolImpl retry logic [iceberg]

2024-06-06 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10458: URL: https://github.com/apache/iceberg/pull/10458 This change is stemming from https://github.com/apache/iceberg/pull/10433#discussion_r1624930096. This PR introduces Failsafe https://failsafe.dev/ as a dependency and integrates tha

Re: [I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
dll02 commented on issue #10453: URL: https://github.com/apache/iceberg/issues/10453#issuecomment-2153700654 @pvary Thank you very much for your reply, it is very helpful to me.You helped me resolve my confusion. -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630469792 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestAggregatedStatisticsTracker.java: ## @@ -246,7 +254,14 @@ public void receiveCompletedS

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630457171 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsCoordinator.java: ## @@ -189,28 +198,75 @@ private void handleDataStatisticRe

Re: [PR] Core: Reword exception message in RewriteManifests [iceberg]

2024-06-06 Thread via GitHub
amogh-jahagirdar commented on PR #10446: URL: https://github.com/apache/iceberg/pull/10446#issuecomment-2153645114 Sorry, and thanks @anuragmantri for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Core: Reword exception message in RewriteManifests [iceberg]

2024-06-06 Thread via GitHub
amogh-jahagirdar merged PR #10446: URL: https://github.com/apache/iceberg/pull/10446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core: Reword exception message in RewriteManifests [iceberg]

2024-06-06 Thread via GitHub
amogh-jahagirdar commented on PR #10446: URL: https://github.com/apache/iceberg/pull/10446#issuecomment-2153644655 Thanks @ajantha-bhat ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] spark3 can't query iceberg: failed to connect to Hive Metastore [iceberg]

2024-06-06 Thread via GitHub
github-actions[bot] commented on issue #2359: URL: https://github.com/apache/iceberg/issues/2359#issuecomment-2153614397 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630345753 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatistics.java: ## @@ -35,28 +35,28 @@ class AggregatedStatistics implements Ser

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630412274 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -103,14 +105,41 @@ public void initializeState(StateInit

Re: [PR] Bump mypy-boto3-glue from 1.34.115 to 1.34.121 [iceberg-python]

2024-06-06 Thread via GitHub
Fokko merged PR #799: URL: https://github.com/apache/iceberg-python/pull/799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Bump mkdocs-material from 9.5.25 to 9.5.26 [iceberg-python]

2024-06-06 Thread via GitHub
Fokko merged PR #798: URL: https://github.com/apache/iceberg-python/pull/798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Open-API: TableRequirements should use union of subclasses [iceberg]

2024-06-06 Thread via GitHub
flyrain commented on PR #10434: URL: https://github.com/apache/iceberg/pull/10434#issuecomment-2153541117 Thanks @anuragmantri for working on this. Thanks @Fokko for the reivew. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Open-API: TableRequirements should use union of subclasses [iceberg]

2024-06-06 Thread via GitHub
flyrain merged PR #10434: URL: https://github.com/apache/iceberg/pull/10434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630376841 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsCoordinator.java: ## @@ -189,28 +198,75 @@ private void handleDataStatisticRe

Re: [PR] Allow parquet writer to use row group from table properties [iceberg-python]

2024-06-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #800: URL: https://github.com/apache/iceberg-python/pull/800#discussion_r1630376663 ## tests/integration/test_writes/test_writes.py: ## @@ -452,7 +452,6 @@ def test_write_parquet_other_properties( @pytest.mark.parametrize( "properti

[PR] Allow parquet writer to use row group from table properties [iceberg-python]

2024-06-06 Thread via GitHub
amogh-jahagirdar opened a new pull request, #800: URL: https://github.com/apache/iceberg-python/pull/800 Right now if the Iceberg parquet row group size table property is set, writes will fail with unsupported. It seems like a bug since the intention of _get_parquet_writer_kwargs is to s

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630358521 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -103,14 +105,41 @@ public void initializeState(StateInit

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630354301 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -139,14 +168,16 @@ public void handleOperatorEvent(Opera

Re: [PR] Bump mypy-boto3-glue from 1.34.115 to 1.34.120 [iceberg-python]

2024-06-06 Thread via GitHub
dependabot[bot] closed pull request #797: Bump mypy-boto3-glue from 1.34.115 to 1.34.120 URL: https://github.com/apache/iceberg-python/pull/797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Bump mypy-boto3-glue from 1.34.115 to 1.34.120 [iceberg-python]

2024-06-06 Thread via GitHub
dependabot[bot] commented on PR #797: URL: https://github.com/apache/iceberg-python/pull/797#issuecomment-2153515650 Superseded by #799. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Bump mypy-boto3-glue from 1.34.115 to 1.34.121 [iceberg-python]

2024-06-06 Thread via GitHub
dependabot[bot] opened a new pull request, #799: URL: https://github.com/apache/iceberg-python/pull/799 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.115 to 1.34.121. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commi

[PR] Bump mkdocs-material from 9.5.25 to 9.5.26 [iceberg-python]

2024-06-06 Thread via GitHub
dependabot[bot] opened a new pull request, #798: URL: https://github.com/apache/iceberg-python/pull/798 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.25 to 9.5.26. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mk

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630345753 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatistics.java: ## @@ -35,28 +35,28 @@ class AggregatedStatistics implements Ser

Re: [I] Upcasting and Downcasting inconsistencies with PyArrow Schema [iceberg-python]

2024-06-06 Thread via GitHub
syun64 commented on issue #791: URL: https://github.com/apache/iceberg-python/issues/791#issuecomment-2153441786 I'm seeing the same restriction when using PolaRs write_parquet, so it looks like a Parquet limitation, instead of an Arrow restriction: ``` ComputeError: parquet: File

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630172641 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestAggregatedStatisticsTracker.java: ## @@ -246,7 +254,14 @@ public void receiveCompletedStati

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630169892 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestAggregatedStatisticsTracker.java: ## @@ -246,7 +254,14 @@ public void receiveCompletedStati

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630164955 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -103,14 +105,41 @@ public void initializeState(StateInitiali

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630153587 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsCoordinator.java: ## @@ -189,28 +198,75 @@ private void handleDataStatisticReques

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630149920 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatistics.java: ## @@ -35,28 +35,28 @@ class AggregatedStatistics implements Seriali

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630124473 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -139,14 +168,16 @@ public void handleOperatorEvent(OperatorE

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630122341 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsOperator.java: ## @@ -139,14 +168,16 @@ public void handleOperatorEvent(OperatorE

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
matthijseikelenboom commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2153255225 SparkSessionProvider.java - line 68: `.config("spark.sql.catalog.iceberghive.cache-enabled", "false")` If this is the property that you mean, then yes -- This

Re: [PR] Flink: handle rescale properly for range bounds in sketch statistics [iceberg]

2024-06-06 Thread via GitHub
stevenzwu commented on code in PR #10457: URL: https://github.com/apache/iceberg/pull/10457#discussion_r1630015642 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsCoordinator.java: ## @@ -71,6 +76,7 @@ class DataStatisticsCoordinator implem

Re: [PR] Kafka-connect: Make required columns optional if missing from record [iceberg]

2024-06-06 Thread via GitHub
tabmatfournier commented on PR #10456: URL: https://github.com/apache/iceberg/pull/10456#issuecomment-2153110975 This should be gated by a config flag, so the user can control behavior. For some users, a missing column may represent a bad message / something going wrong, and they may want

Re: [I] MetricsReporter support close [iceberg]

2024-06-06 Thread via GitHub
allen-abhishekkr commented on issue #9349: URL: https://github.com/apache/iceberg/issues/9349#issuecomment-2153099416 @huyuanfeng2018 I see that the "closeable" trait was added in this commit, however in case of below implementations, the close() method of the MetricsReporter never g

Re: [PR] Kafka-connect: Make required columns optional if missing from record [iceberg]

2024-06-06 Thread via GitHub
fqaiser94 commented on code in PR #10456: URL: https://github.com/apache/iceberg/pull/10456#discussion_r1629982369 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/RecordConverterTest.java: ## @@ -859,6 +859,107 @@ public void testEvolveTypeDetectionS

Re: [PR] Kafka-connect: Make required columns optional if missing from record [iceberg]

2024-06-06 Thread via GitHub
fqaiser94 commented on code in PR #10456: URL: https://github.com/apache/iceberg/pull/10456#discussion_r1629982369 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/RecordConverterTest.java: ## @@ -859,6 +859,107 @@ public void testEvolveTypeDetectionS

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
RussellSpitzer commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2153056909 Is the Catalog Cache in use? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2153003145 Thank you for the quick review @nastra! I believe everything is addressed now. Please have another look when you can. Thanks! -- This is an automated message from the Apache Git Se

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629896432 ## api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java: ## @@ -185,7 +194,28 @@ private static int convertMicros(long micros, ChronoUnit granularity) { }

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629895942 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,162 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629893300 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,162 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629892323 ## api/src/main/java/org/apache/iceberg/transforms/SortOrderVisitor.java: ## @@ -85,21 +85,18 @@ static List visit(SortOrder sortOrder, SortOrderVisitor visitor) {

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629892623 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour.g

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629891562 ## api/src/main/java/org/apache/iceberg/transforms/Transforms.java: ## @@ -188,9 +198,15 @@ public static Transform day(Type type) { @Deprecated @SuppressWarnings(

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629892839 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour.g

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629891910 ## api/src/main/java/org/apache/iceberg/transforms/Days.java: ## @@ -55,7 +57,16 @@ public boolean satisfiesOrderOf(Transform other) { } if (other instanceof

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
findepi commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629836495 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629728440 ## api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java: ## @@ -185,7 +194,28 @@ private static int convertMicros(long micros, ChronoUnit granularity) { }

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629725179 ## api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java: ## @@ -185,7 +194,28 @@ private static int convertMicros(long micros, ChronoUnit granularity) { }

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629722207 ## api/src/test/java/org/apache/iceberg/transforms/TestDays.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629719484 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,162 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
matthijseikelenboom commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2152766019 @nastra The README of the repo describes in detail what the test does. But in short: The test fails because what it expects to be present on disc, isn't there. On the t

Re: [I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
pvary commented on issue #10453: URL: https://github.com/apache/iceberg/issues/10453#issuecomment-2152753909 > So you mean that since my table has a unique key constraint, it is considered a Change Data Capture (CDC) table. If the change contains updates, then it is a CDC usecase.

Re: [I] Prioritize the catalog setting method to polaris-catalog. [iceberg]

2024-06-06 Thread via GitHub
nastra closed issue #10455: Prioritize the catalog setting method to polaris-catalog. URL: https://github.com/apache/iceberg/issues/10455 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Prioritize the catalog setting method to polaris-catalog. [iceberg]

2024-06-06 Thread via GitHub
nastra commented on issue #10455: URL: https://github.com/apache/iceberg/issues/10455#issuecomment-2152720060 Hey @911432, the Polaris Catalog is not managed by the Iceberg Community so we can't provide any answers about how it works internally and what features it supports and such. You wo

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
nastra commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2152494225 @matthijseikelenboom can you please share details of what exactly is failing in which scenario for the people that aren't running the test? This makes it easier to reason about what

Re: [I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-06 Thread via GitHub
nastra commented on issue #10443: URL: https://github.com/apache/iceberg/issues/10443#issuecomment-2152482354 I'll close this one since it's a duplicate of https://github.com/apache/iceberg/issues/10213 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-06 Thread via GitHub
nastra closed issue #10443: Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned URL: https://github.com/apache/iceberg/issues/10443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Core: Reword exception message in RewriteManifests [iceberg]

2024-06-06 Thread via GitHub
ajantha-bhat commented on code in PR #10446: URL: https://github.com/apache/iceberg/pull/10446#discussion_r1629502264 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -275,14 +275,17 @@ private boolean matchesPredicate(ManifestFile manifest) { retur

[I] Iceberg fails ACID verification test [iceberg]

2024-06-06 Thread via GitHub
matthijseikelenboom opened a new issue, #10454: URL: https://github.com/apache/iceberg/issues/10454 ### Apache Iceberg version 1.5.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 ### Problem statement For work we had needed to

Re: [I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-06 Thread via GitHub
jurossiar commented on issue #10443: URL: https://github.com/apache/iceberg/issues/10443#issuecomment-2152265435 Feel free to close some of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Iceberg rest catalog with postgres - List namespaces with parent returns wrong children namespaces [iceberg]

2024-06-06 Thread via GitHub
jurossiar commented on issue #10213: URL: https://github.com/apache/iceberg/issues/10213#issuecomment-2152264490 See: https://github.com/apache/iceberg/issues/10443#issuecomment-2152262679 I duplicated the issue given I forgot that i created this one with a better analysis. I think tha

Re: [I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-06 Thread via GitHub
jurossiar commented on issue #10443: URL: https://github.com/apache/iceberg/issues/10443#issuecomment-2152262679 @nastra Yes, it's true! I forgot I did a better analysis last time and created that issue. It's the same issue and now we have more cases. I found more cases and reported it agai

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629246525 ## api/src/test/java/org/apache/iceberg/transforms/TestYears.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629248863 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,162 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629247791 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,162 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629245294 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629244450 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1629238344 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType = hour.

Re: [PR] Hive: Return new scan after applying column project parameter [iceberg]

2024-06-06 Thread via GitHub
zhangbutao commented on code in PR #10449: URL: https://github.com/apache/iceberg/pull/10449#discussion_r1629227339 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -234,6 +234,8 @@ public void testProjection() throws Exception { Schema projec

Re: [I] unable to show views [iceberg]

2024-06-06 Thread via GitHub
nastra commented on issue #10364: URL: https://github.com/apache/iceberg/issues/10364#issuecomment-2151863884 The root cause of the issue was just fixed by https://github.com/apache/iceberg/pull/10442, but as @jbonofre mentioned, there's no view support available for `HadoopCatalog`. I'll g

Re: [I] unable to show views [iceberg]

2024-06-06 Thread via GitHub
nastra closed issue #10364: unable to show views URL: https://github.com/apache/iceberg/issues/10364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: iss

Re: [I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-06 Thread via GitHub
nastra commented on issue #10443: URL: https://github.com/apache/iceberg/issues/10443#issuecomment-2151858911 Is it possible that this is the same issue that was described in https://github.com/apache/iceberg/issues/10213 (aka it's an issue with `JdbcCatalog`)? The `RESTCatalog` only acts a

Re: [I] Support partitioned writes [iceberg-python]

2024-06-06 Thread via GitHub
deepika094 commented on issue #208: URL: https://github.com/apache/iceberg-python/issues/208#issuecomment-2151843836 hi, do we have any way to write to partitioned table so far? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
dll02 commented on issue #10453: URL: https://github.com/apache/iceberg/issues/10453#issuecomment-2151671859 @pvary Thank you for your response. I have read the paragraph: Streaming read[🔗](https://iceberg.apache.org/docs/nightly/flink-queries/#streaming-read_1) This example will star

Re: [I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
dll02 commented on issue #10453: URL: https://github.com/apache/iceberg/issues/10453#issuecomment-2151652109 The data printed in the first line of the log is historical data, but subsequent monitoring does not print out new data additions. @pvary thanks lingling -

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-06 Thread via GitHub
pvary merged PR #10308: URL: https://github.com/apache/iceberg/pull/10308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-06 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1628951961 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/FlinkSqlExtension.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-06 Thread via GitHub
pvary commented on PR #10308: URL: https://github.com/apache/iceberg/pull/10308#issuecomment-2151632301 Merged to main. Thanks for the review @rodmeneses and @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
pvary commented on issue #10453: URL: https://github.com/apache/iceberg/issues/10453#issuecomment-2151626475 @dll02: I don't understand yet your issue. Could you please elaborate? I see, that your log contains one record, which could be correct, or wrong depending on your use-case:

Re: [PR] Build: Bump Hive 2.3.10 [iceberg]

2024-06-06 Thread via GitHub
pan3793 commented on code in PR #10447: URL: https://github.com/apache/iceberg/pull/10447#discussion_r1628939418 ## gradle/libs.versions.toml: ## @@ -142,8 +142,8 @@ hive3-service = { module = "org.apache.hive:hive-service", version.ref = "hive3" httpcomponents-httpclient5 = {

Re: [PR] Hive: Return new scan after applying column project parameter [iceberg]

2024-06-06 Thread via GitHub
pvary commented on PR #10449: URL: https://github.com/apache/iceberg/pull/10449#issuecomment-2151612132 > I am not sure how to add the test & also think no need add the test. I disagree with you here because of 2 reasons: 1. If there is no tests, then how can we be sure that fut

[I] flink + iceberg can't read realtime data from glue + s3 data with PRIMARY KEY unique table [iceberg]

2024-06-06 Thread via GitHub
dll02 opened a new issue, #10453: URL: https://github.com/apache/iceberg/issues/10453 ### Apache Iceberg version 1.4.2 ### Query engine Flink ### Please describe the bug 🐞 I am using AWS manage Flink service to read data from Iceberg + Glue tables for real-

Re: [PR] Build: Bump Hive 2.3.10 [iceberg]

2024-06-06 Thread via GitHub
nastra commented on code in PR #10447: URL: https://github.com/apache/iceberg/pull/10447#discussion_r1628918232 ## gradle/libs.versions.toml: ## @@ -142,8 +142,8 @@ hive3-service = { module = "org.apache.hive:hive-service", version.ref = "hive3" httpcomponents-httpclient5 = {

Re: [PR] Build: Bump Hive 2.3.10 [iceberg]

2024-06-06 Thread via GitHub
pan3793 commented on code in PR #10447: URL: https://github.com/apache/iceberg/pull/10447#discussion_r1628901207 ## mr/build.gradle: ## @@ -68,8 +67,7 @@ project(':iceberg-mr') { testImplementation libs.avro.avro testImplementation libs.calcite.core testImplementa