Re: [I] Accessing Minio with Pyiceberg [iceberg]

2024-07-29 Thread via GitHub
rggyanav commented on issue #10709: URL: https://github.com/apache/iceberg/issues/10709#issuecomment-2257544318 @muniatl - I think the MinIO endpoints should not use the s3:// prefix for the endpoint configuration. They should instead use the HTTP/HTTPS protocol. e.g: warehouse="s3://i

Re: [PR] Sql catalog [iceberg-rust]

2024-07-29 Thread via GitHub
JanKaul closed pull request #229: Sql catalog URL: https://github.com/apache/iceberg-rust/pull/229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issue

Re: [I] How to query a specified partition data file? [iceberg]

2024-07-29 Thread via GitHub
melin closed issue #10725: How to query a specified partition data file? URL: https://github.com/apache/iceberg/issues/10725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Use Min, Max, and NumOfNulls from Manifest Files for Spark Column Stats [iceberg]

2024-07-29 Thread via GitHub
jeesou commented on issue #10791: URL: https://github.com/apache/iceberg/issues/10791#issuecomment-2257458241 Hi @huaxingao, so the follow up PR will only have min, max, and numOfNulls ? will avgLen and maxLen will be included later? And for min, max, and numOfNulls are we gonna use

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure URL: https://github.com/apache/iceberg/pull/10811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-07-29 Thread via GitHub
moomindani commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2257398666 @laithalzyoud I am super interested in this PR and it will unblock many use cases. Are you working on this now? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure URL: https://github.com/apache/iceberg/pull/10811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696223028 ## api/src/main/java/org/apache/iceberg/RemoveUnusedSpecs.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696192362 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) { dedupF

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure URL: https://github.com/apache/iceberg/pull/10811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang commented on code in PR #10811: URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java: ## @@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IO

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang commented on code in PR #10811: URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java: ## @@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IO

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-07-29 Thread via GitHub
manuzhang commented on code in PR #10811: URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java: ## @@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IO

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696133057 ## core/src/main/java/org/apache/iceberg/BaseRemoveUnusedSpecs.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Docs: Use link addresses instead of descriptions in releases.md [iceberg]

2024-07-29 Thread via GitHub
lurnagao-dahua commented on PR #10815: URL: https://github.com/apache/iceberg/pull/10815#issuecomment-2257278493 @pvary @nastra @Fokko would you mind taking a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) { } private void checkAndAddPart

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) { } private void checkAndAddPart

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) { } private void checkAndAddPart

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696126712 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) { dedupFi

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696124760 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) { } private void checkAndAddPart

Re: [PR] Kafka Connect: Terminate commits on coordinator stop [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on PR #10814: URL: https://github.com/apache/iceberg/pull/10814#issuecomment-2257244778 @fqaiser94 @singhpk234 @ajantha-bhat , it would be great to get your feedback on this if you have a chance. -- This is an automated message from the Apache Git Service. To respond to

[PR] Kafka Connect: Terminate commits on coordinator stop [iceberg]

2024-07-29 Thread via GitHub
bryanck opened a new pull request, #10814: URL: https://github.com/apache/iceberg/pull/10814 This PR aims to improve the handling of coordinator shutdown during sink rebalances, to better ensure only one coordinator is active at a given time. The sink was designed with the assumption that o

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696085669 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -390,7 +391,8 @@ Builder checkConflicts(boolean check) { } private void checkAndAddPar

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on PR #10678: URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257241580 > In light of @rdblue 's comment let's make sure we add a test case which has a partition spec with "both foo and Foo" as different but referenced source columns. > > CREATE TAB

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on PR #10678: URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257241120 @rdblue I am not certain I understand your suggested path forward. Are you suggesting some edit like this at the bottom of PartitionSpec.Builder.checkAndAddPartitionName? ```

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1696120360 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1696120360 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sfc-gh-rspitzer commented on PR #10678: URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257232552 In light of @rdblue 's comment let's make sure we add a test case which has a partition spec with "both foo and Foo" as different but referenced source columns. CREATE

Re: [PR] Use snapshot schema when rollback to snapshot [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on PR #3515: URL: https://github.com/apache/iceberg/pull/3515#issuecomment-2257228096 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Flink streaming read from mix snapshot table [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3486: URL: https://github.com/apache/iceberg/issues/3486#issuecomment-2257228071 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Website: spec page does not show subsections in sidebar [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3475: URL: https://github.com/apache/iceberg/issues/3475#issuecomment-2257228009 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Doc: missing Spark doc for some system tables [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3465: URL: https://github.com/apache/iceberg/issues/3465#issuecomment-2257227939 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: read delete files in parallel [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on PR #3120: URL: https://github.com/apache/iceberg/pull/3120#issuecomment-2257227644 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] java.lang.IllegalArgumentException: schemas must exist in format v2 [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3507: URL: https://github.com/apache/iceberg/issues/3507#issuecomment-2257228078 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Doc: add documentation for newcomers to reference for code contribution [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3485: URL: https://github.com/apache/iceberg/issues/3485#issuecomment-2257228053 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Infra: add Github template for PRs [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3484: URL: https://github.com/apache/iceberg/issues/3484#issuecomment-2257228034 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Website: Hive page does not show sections in sidebar [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] commented on issue #3474: URL: https://github.com/apache/iceberg/issues/3474#issuecomment-2257227961 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: read delete files in parallel [iceberg]

2024-07-29 Thread via GitHub
github-actions[bot] closed pull request #3120: Core: read delete files in parallel URL: https://github.com/apache/iceberg/pull/3120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696102245 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1425,6 +1460,7 @@ private boolean hasChanges() { || (discardChanges && !changes.isEmpty(

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696101726 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this; }

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696100886 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this; }

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696099907 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this; }

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696099533 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -597,6 +597,24 @@ public TableMetadata replaceProperties(Map rawProperties) { .build();

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696097946 ## core/src/main/java/org/apache/iceberg/BaseRemoveUnusedSpecs.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696095743 ## core/src/main/java/org/apache/iceberg/BaseRemoveUnusedSpecs.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696094932 ## api/src/main/java/org/apache/iceberg/RemoveUnusedSpecs.java: ## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more cont

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1696070676 ## core/src/main/java/org/apache/iceberg/BaseRemoveUnusedSpecs.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on PR #10678: URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2257156417 I think that the API proposed here is the right approach. @amogh-jahagirdar makes a good point about `compatibleWith` and possibly other methods that we use to reason about partiti

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1696058526 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on PR #10793: URL: https://github.com/apache/iceberg/pull/10793#issuecomment-2257148519 Yes this pr as is should not require a spec change. > The wording used for apache-datasketches-theta-v1 should have been better and clearly define both: allowed values and their

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1696054978 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1696052319 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void ter

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696038786 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -427,13 +429,21 @@ private void checkForRedundantPartitions(PartitionField field) {

[PR] Bump tenacity from 8.5.0 to 9.0.0 [iceberg-python]

2024-07-29 Thread via GitHub
dependabot[bot] opened a new pull request, #978: URL: https://github.com/apache/iceberg-python/pull/978 Bumps [tenacity](https://github.com/jd/tenacity) from 8.5.0 to 9.0.0. Release notes Sourced from https://github.com/jd/tenacity/releases";>tenacity's releases. tenacity 9.0

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1696048433 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void termi

Re: [PR] Flink: support limit pushdown in FLIP-27 source [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on PR #10748: URL: https://github.com/apache/iceberg/pull/10748#issuecomment-2257125028 thanks @czy006 @nastra @pvary for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Flink: support limit pushdown in FLIP-27 source [iceberg]

2024-07-29 Thread via GitHub
stevenzwu merged PR #10748: URL: https://github.com/apache/iceberg/pull/10748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Flink: improve snapshot compatibility check by comparing projected sort schema in SortKeySerializer [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on PR #10794: URL: https://github.com/apache/iceberg/pull/10794#issuecomment-2257123912 thanks @pvary for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Flink: improve snapshot compatibility check by comparing projected sort schema in SortKeySerializer [iceberg]

2024-07-29 Thread via GitHub
stevenzwu merged PR #10794: URL: https://github.com/apache/iceberg/pull/10794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] feat(table/scanner): Initial pass for planing a scan and returning the files to use [iceberg-go]

2024-07-29 Thread via GitHub
zeroshade commented on PR #118: URL: https://github.com/apache/iceberg-go/pull/118#issuecomment-2257123512 @Fokko @nastra This should be ready for review now, though there's a weirdness in the number of data files being created for one of the integration testing tables on the CI here vs whe

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1695810956 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java: ## @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1696035086 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void ter

[PR] Fence zombie coordinators [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 opened a new pull request, #10812: URL: https://github.com/apache/iceberg/pull/10812 The last commit demonstrates how we could easily avoid duplicate-file-appends if iceberg were to offer a conditional-commit API. This is just a draft to demonstrate the idea but I'm happy t

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1696032626 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void ter

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
fqaiser94 commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1696029218 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void ter

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696029266 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -95,6 +95,19 @@ public void testSpecInfoPartitionedTable() { .doesNotContainKey(I

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-07-29 Thread via GitHub
stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1651100591 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software F

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-29 Thread via GitHub
dramaticlly commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1696005290 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -95,6 +95,19 @@ public void testSpecInfoPartitionedTable() { .doesNotContainKe

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695985484 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
emkornfield commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695984822 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an is

Re: [I] [Feature] Support Metadata Log Update For Non-Rest Catalogs [iceberg-python]

2024-07-29 Thread via GitHub
soumya-ghosh commented on issue #849: URL: https://github.com/apache/iceberg-python/issues/849#issuecomment-2257006408 PR for review - https://github.com/apache/iceberg-python/pull/977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Update metadata-log for non-rest catalogs [iceberg-python]

2024-07-29 Thread via GitHub
soumya-ghosh opened a new pull request, #977: URL: https://github.com/apache/iceberg-python/pull/977 Implements: https://github.com/apache/iceberg-python/issues/849 Also added support to control size of `metadata-log` using `write.metadata.previous-versions-max` property. -- This i

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
findepi commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1695941907 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob ma

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
findepi commented on PR #10793: URL: https://github.com/apache/iceberg/pull/10793#issuecomment-2256968863 > misinterpreted this pr to support double as per [#10288 (comment)](https://github.com/apache/iceberg/pull/10288#discussion_r1691077522) . sorry for the confusion! in that PR

Re: [PR] Docs: Clarify wording on releases [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10806: URL: https://github.com/apache/iceberg/pull/10806#discussion_r1695832403 ## site/docs/how-to-release.md: ## @@ -76,17 +70,33 @@ For more information, see the Gradle [signing documentation](https://docs.gradle The release should be execute

Re: [PR] Docs: Clarify wording on releases [iceberg]

2024-07-29 Thread via GitHub
rdblue commented on code in PR #10806: URL: https://github.com/apache/iceberg/pull/10806#discussion_r1695832403 ## site/docs/how-to-release.md: ## @@ -76,17 +70,33 @@ For more information, see the Gradle [signing documentation](https://docs.gradle The release should be execute

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-29 Thread via GitHub
venkata91 commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-2256836243 > > > This basically resulted in "inconsistencies" between the 1.17-1.18/1.19 tests. > > > > > > @pvary Can you clarify what do you mean by the "inconsistencies" here? >

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695801167 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [PR] DOC: Strawman proposal for PR merging [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10780: URL: https://github.com/apache/iceberg/pull/10780#discussion_r1695800041 ## site/docs/contribute.md: ## @@ -45,6 +45,16 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is related to an issu

Re: [PR] AddPartitionSpec: A new way to set new partition specs [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on code in PR #10737: URL: https://github.com/apache/iceberg/pull/10737#discussion_r1695794990 ## core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java: ## @@ -66,20 +66,36 @@ class BaseUpdatePartitionSpec implements UpdatePartitionSpec {

Re: [PR] AddPartitionSpec: A new way to set new partition specs [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on code in PR #10737: URL: https://github.com/apache/iceberg/pull/10737#discussion_r1695790388 ## core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java: ## @@ -66,20 +66,36 @@ class BaseUpdatePartitionSpec implements UpdatePartitionSpec {

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695665021 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void termi

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on PR #10792: URL: https://github.com/apache/iceberg/pull/10792#issuecomment-2256789099 Thanks for the reviews everyone, I'm going to close this and open a new one after giving it some more thought. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck closed pull request #10792: Kafka Connect: Wait for coordinator shutdown URL: https://github.com/apache/iceberg/pull/10792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on PR #10736: URL: https://github.com/apache/iceberg/pull/10736#issuecomment-2256785593 > Overall LGTM, as @RussellSpitzer mentioned you'll need to fix RevAPI. Also @RussellSpitzer not sure if you saw the naming change to make the API called `addNonDefaultSpec` , ju

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695784467 ## crates/test_utils/src/docker.rs: ## @@ -40,15 +41,28 @@ impl DockerCompose { self.project_name.as_str() } +// docker/podman do not consistent

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695784200 ## crates/test_utils/src/cmd.rs: ## @@ -28,6 +28,17 @@ pub fn run_command(mut cmd: Command, desc: impl ToString) { } } +pub fn get_cmd_output_dont_panic(mut

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on code in PR #10793: URL: https://github.com/apache/iceberg/pull/10793#discussion_r1695779192 ## format/puffin-spec.md: ## @@ -121,7 +121,9 @@ distinct values converted to bytes using Iceberg's single-value serialization. The blob metadata for this blob

Re: [PR] Document stats `ndv` value representation [iceberg]

2024-07-29 Thread via GitHub
szehon-ho commented on PR #10793: URL: https://github.com/apache/iceberg/pull/10793#issuecomment-2256768734 @findepi Got it sorry i misinterpreted this pr to support double as per https://github.com/apache/iceberg/pull/10288#discussion_r1691077522 . This pr makes more sense then.

Re: [PR] Kafka Connect: Wait for coordinator shutdown [iceberg]

2024-07-29 Thread via GitHub
bryanck commented on code in PR #10792: URL: https://github.com/apache/iceberg/pull/10792#discussion_r1695768314 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CoordinatorThread.java: ## @@ -65,5 +66,15 @@ boolean isTerminated() { void termi

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695747997 ## crates/test_utils/src/cmd.rs: ## @@ -28,6 +28,17 @@ pub fn run_command(mut cmd: Command, desc: impl ToString) { } } +pub fn get_cmd_output_dont_panic(mut

Re: [I] Max number of columns [iceberg]

2024-07-29 Thread via GitHub
nlarusstone commented on issue #9220: URL: https://github.com/apache/iceberg/issues/9220#issuecomment-2256703993 Is there any workaround here? As a sample usecase, some biological data may have 20K columns (# of genes) by millions of rows (# of cells) -- This is an automated messa

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695719265 ## crates/catalog/hms/testdata/hms_catalog/Dockerfile: ## @@ -15,8 +15,6 @@ FROM --platform=$BUILDPLATFORM openjdk:8-jre-slim AS build -ARG BUILDPLATFORM Revi

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695718458 ## crates/catalog/glue/tests/glue_catalog_test.rs: ## @@ -83,15 +81,15 @@ async fn get_catalog() -> GlueCatalog { (AWS_REGION_NAME.to_string(), "us-east-1"

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695717804 ## crates/catalog/glue/tests/glue_catalog_test.rs: ## @@ -83,15 +81,15 @@ async fn get_catalog() -> GlueCatalog { (AWS_REGION_NAME.to_string(), "us-east-1"

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695714416 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695710521 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695706639 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695706639 ## website/src/reference/podman.md: ## @@ -0,0 +1,87 @@ + + +# Using Podman instead of Docker + +The majority of iceberg-rust is agnostic as to which containerizati

Re: [PR] feat: podman support [iceberg-rust]

2024-07-29 Thread via GitHub
alexyin1 commented on code in PR #489: URL: https://github.com/apache/iceberg-rust/pull/489#discussion_r1695526821 ## crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml: ## @@ -15,6 +15,9 @@ # specific language governing permissions and limitations # under the Licen

  1   2   3   >