Re: [I] Iceberg Materialized View Spec [iceberg]

2023-10-20 Thread via GitHub
JanKaul commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1773687907 I think we almost have a consensus about the Materialized View Spec. I brought up one issue with the **pointer to the storage table** that is holding up the progress at the moment. T

Re: [I] Apache hive 3 with Tez engine select table no empty [iceberg]

2023-10-20 Thread via GitHub
pvary commented on issue #8891: URL: https://github.com/apache/iceberg/issues/8891#issuecomment-1773670762 @anvanna: Please read the relevant parts of the documentation about the supported Hive versions/features on the following page: https://iceberg.apache.org/docs/latest/hive/ I wo

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-20 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1365732924 ## api/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [I] Iceberg Materialized View Spec [iceberg]

2023-10-20 Thread via GitHub
liurenjie1024 commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1773609261 Any update on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] OPTIMIZE operation prunes data not visible to the table [iceberg]

2023-10-20 Thread via GitHub
github-actions[bot] commented on issue #7192: URL: https://github.com/apache/iceberg/issues/7192#issuecomment-1773544050 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] OPTIMIZE operation prunes data not visible to the table [iceberg]

2023-10-20 Thread via GitHub
github-actions[bot] closed issue #7192: OPTIMIZE operation prunes data not visible to the table URL: https://github.com/apache/iceberg/issues/7192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] TableLoader tableLoader = TableLoader.fromHadoopTable(String location, Configuration hadoopConf) NullPointerException [iceberg]

2023-10-20 Thread via GitHub
github-actions[bot] commented on issue #7394: URL: https://github.com/apache/iceberg/issues/7394#issuecomment-1773543988 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367567052 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -374,4 +386,128 @@ static TableMetadata commit(TableOperations ops, UpdateTableRequest request

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r136759 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -374,4 +385,107 @@ static TableMetadata commit(TableOperations ops, UpdateTableRequest request

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367565894 ## core/src/main/java/org/apache/iceberg/rest/RESTCatalog.java: ## @@ -35,17 +35,22 @@ import org.apache.iceberg.catalog.SupportsNamespaces; import org.apache.iceberg.

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367565571 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -943,11 +968,11 @@ private static Cache newSessionCache(Map pr .build(); } -

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367565262 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -971,4 +996,261 @@ public void commitTransaction(SessionContext context, List commits)

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367564841 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -971,4 +996,261 @@ public void commitTransaction(SessionContext context, List commits)

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367563075 ## core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367562363 ## core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367562181 ## core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java: ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[I] Apache hive 3 with Tez engine select table no empty [iceberg]

2023-10-20 Thread via GitHub
anvanna opened a new issue, #8891: URL: https://github.com/apache/iceberg/issues/8891 I am testing iceberg table as below with hive version 3.x and Tez version 0.9.x - Hive engine running on Tez ++ |set | +--

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367561171 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that def

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367560695 ## core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java: ## @@ -89,11 +94,13 @@ public class RESTCatalogAdapter implements RESTClient { private final

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367560477 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that def

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on PR #8865: URL: https://github.com/apache/iceberg/pull/8865#issuecomment-1773525223 Thanks, @nastra! Looks like this is also included in #7913 so it's good to get this in. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-20 Thread via GitHub
rdblue merged PR #8865: URL: https://github.com/apache/iceberg/pull/8865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Improvements around View catalog tests [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8865: URL: https://github.com/apache/iceberg/pull/8865#discussion_r1367559524 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -132,15 +144,21 @@ public void completeCreateView() { .withQuery("trino", "select

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367559085 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that def

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367558698 ## open-api/rest-catalog-open-api.yaml: ## @@ -2033,6 +2537,68 @@ components: metadata-location: type: string +CreateViewRequest: + type:

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367558132 ## open-api/rest-catalog-open-api.yaml: ## @@ -2033,6 +2537,68 @@ components: metadata-location: type: string +CreateViewRequest: + type:

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367558418 ## open-api/rest-catalog-open-api.yaml: ## @@ -2033,6 +2537,68 @@ components: metadata-location: type: string +CreateViewRequest: + type:

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367558272 ## open-api/rest-catalog-open-api.yaml: ## @@ -2033,6 +2537,68 @@ components: metadata-location: type: string +CreateViewRequest: + type:

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367557579 ## open-api/rest-catalog-open-api.yaml: ## @@ -1860,6 +2320,26 @@ components: items: type: string +AddViewVersionUpdate: + all

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367556869 ## open-api/rest-catalog-open-api.yaml: ## @@ -1630,6 +1990,102 @@ components: metadata-log: $ref: '#/components/schemas/MetadataLog' +SQLViewR

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367556869 ## open-api/rest-catalog-open-api.yaml: ## @@ -1630,6 +1990,102 @@ components: metadata-log: $ref: '#/components/schemas/MetadataLog' +SQLViewR

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367555467 ## open-api/rest-catalog-open-api.yaml: ## @@ -1056,6 +1056,357 @@ paths: } } + /v1/{prefix}/namespaces/{namespace}/views: +parame

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367555195 ## open-api/rest-catalog-open-api.yaml: ## @@ -1056,6 +1056,357 @@ paths: } } + /v1/{prefix}/namespaces/{namespace}/views: +parame

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367554935 ## open-api/rest-catalog-open-api.yaml: ## @@ -1056,6 +1056,357 @@ paths: } } + /v1/{prefix}/namespaces/{namespace}/views: +parame

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1367546614 ## core/src/main/java/org/apache/iceberg/catalog/BaseSessionCatalog.java: ## @@ -30,8 +30,10 @@ import org.apache.iceberg.exceptions.NamespaceNotEmptyException; import

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on PR #8723: URL: https://github.com/apache/iceberg/pull/8723#issuecomment-1773504162 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-20 Thread via GitHub
rdblue merged PR #8723: URL: https://github.com/apache/iceberg/pull/8723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Derive 'operation' from view version [iceberg]

2023-10-20 Thread via GitHub
rdblue merged PR #8678: URL: https://github.com/apache/iceberg/pull/8678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Derive 'operation' from view version [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on PR #8678: URL: https://github.com/apache/iceberg/pull/8678#issuecomment-1773497347 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Core: Derive 'operation' from view version [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8678: URL: https://github.com/apache/iceberg/pull/8678#discussion_r1367544626 ## core/src/test/java/org/apache/iceberg/view/TestViewMetadataParser.java: ## @@ -335,7 +335,7 @@ public void metadataCompression(String fileName) throws IOException {

Re: [PR] API, Core: Add UUID API to Table [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8800: URL: https://github.com/apache/iceberg/pull/8800#discussion_r1367540014 ## core/src/main/java/org/apache/iceberg/BaseMetadataTable.java: ## @@ -199,6 +199,11 @@ public Map refs() { return table().refs(); } + @Override + public St

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367530477 ## site/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,57 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift ;;

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367530603 ## site/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,57 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift ;;

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367530281 ## site/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,57 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift ;;

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367527689 ## site/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,57 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift ;;

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367527441 ## site/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,57 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift ;;

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367527094 ## site/.github/bin/deploy_docs.sh.orig: ## @@ -0,0 +1,67 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shif

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367526487 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that define the

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367526033 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that define the

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1367525768 ## site/README.md: ## @@ -0,0 +1,182 @@ + + +# Iceberg site and documentation + +This subproject contains the [MkDocs projects](https://www.mkdocs.org/) that define the

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1367434248 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,133 +185,223 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1367436893 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -106,7 +106,7 @@ public void testCreatingAndDroppingNamespaceWithContent() throws NessieNotF

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1367436336 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -323,28 +417,13 @@ public void renameTable(TableIdentifier from, TableIdentifier to) {

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-20 Thread via GitHub
RussellSpitzer commented on PR #8844: URL: https://github.com/apache/iceberg/pull/8844#issuecomment-1773316624 I'm also pretty cautious about changing the behavior here. I'm pretty happy with the current default and using "REFRESH TABLE" if the user is in doubt about cache state (or disabli

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1367434248 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,133 +185,223 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1773313071 Again ready for review, all namespace operations were re-implemented. \cc @ajantha-bhat @dimas-b -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Infra: Update slack invite link [iceberg]

2023-10-20 Thread via GitHub
Fokko merged PR #8889: URL: https://github.com/apache/iceberg/pull/8889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Add Refurb to ruff [iceberg-python]

2023-10-20 Thread via GitHub
jayceslesar commented on PR #87: URL: https://github.com/apache/iceberg-python/pull/87#issuecomment-1773176325 @Fokko running the latest version of refurb: ``` refurb pyiceberg pyiceberg/avro/encoder.py:40:31 [FURB123]: Replace `bool(x)` with `x` pyiceberg/avro/reader.py:354:16 [F

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
dramaticlly commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1367320490 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -134,8 +134,13 @@ final class JdbcUtil { + " WHERE " + CATALOG_NAME

Re: [I] Add View Support to Spark [iceberg]

2023-10-20 Thread via GitHub
singhpk234 commented on issue #7938: URL: https://github.com/apache/iceberg/issues/7938#issuecomment-1773085005 @nastra @amogh-jahagirdar any plans for this, i think spark view interfaces are in it's just that complete wiring in spark is still pending via [1] , do you plan to pickup impleme

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-20 Thread via GitHub
zhangminglei commented on PR #8844: URL: https://github.com/apache/iceberg/pull/8844#issuecomment-1773078235 Allow me to provide some context. The reason I initially opened this PR was because we observed that when a user attempted to create a table, they got an exception indicating that th

Re: [PR] GCP: Add Iceberg Catalog for GCP BigLake Metastore [iceberg]

2023-10-20 Thread via GitHub
emkornfield commented on PR #7412: URL: https://github.com/apache/iceberg/pull/7412#issuecomment-1773029673 @rdblue just wanted to check if you had any remaining concerns here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[PR] Infra: Update slack invite link [iceberg]

2023-10-20 Thread via GitHub
ajantha-bhat opened a new pull request, #8889: URL: https://github.com/apache/iceberg/pull/8889 follow up for https://github.com/apache/iceberg-docs/pull/286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Update Slack URL [iceberg-docs]

2023-10-20 Thread via GitHub
ajantha-bhat commented on code in PR #286: URL: https://github.com/apache/iceberg-docs/pull/286#discussion_r1367204786 ## landing-page/config.toml: ## @@ -22,7 +22,7 @@ sectionPagesMenu = "main" [[params.social]] title = "slack" icon = "slack" - url = "https://join.slac

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1367148848 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -345,19 +354,27 @@ public static String deletePropertiesStatement(Set properties) { static boole

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on PR #8340: URL: https://github.com/apache/iceberg/pull/8340#issuecomment-1772957166 Thanks, @ismailsimsek! Looks a lot better now. I like the new fix using ESCAPE. I think we just need to have more test cases now. -- This is an automated message from the Apache Git Serv

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1367147296 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -604,6 +604,27 @@ public void testCreateNamespace() { // Test with no metadata catalo

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
rdblue commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1367146172 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -604,6 +604,27 @@ public void testCreateNamespace() { // Test with no metadata catalo

Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-20 Thread via GitHub
liurenjie1024 commented on PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#issuecomment-1772894697 cc @Fokko Any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Docs: PostgreSql integration [iceberg-python]

2023-10-20 Thread via GitHub
mobley-trent commented on issue #78: URL: https://github.com/apache/iceberg-python/issues/78#issuecomment-1772868143 @Fokko What should we place in the location parameter of `catalog.create_table(...)` ? Looking at the code, we are supposed to specify a warehouse location but I'm not quite

Re: [PR] Support timestamp type in partition string when importing files [iceberg]

2023-10-20 Thread via GitHub
puchengy commented on PR #7291: URL: https://github.com/apache/iceberg/pull/7291#issuecomment-1772732327 @camper42 Hi, sorry, I am not actively working on the PR at the moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] feat: add builder to TableMetadata interface [iceberg-rust]

2023-10-20 Thread via GitHub
y0psolo commented on code in PR #62: URL: https://github.com/apache/iceberg-rust/pull/62#discussion_r1366870135 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -93,22 +116,278 @@ pub struct TableMetadata { /// previous metadata file location should be added to the list.

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
liurenjie1024 commented on code in PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#discussion_r1366853709 ## docs/content/catalog.md: ## @@ -1,7 +1,10 @@ --- Review Comment: Cool, I'll fix that -- This is an automated message from the Apache Git Service. T

Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-20 Thread via GitHub
nastra commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1366852703 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -435,13 +437,11 @@ public void testConcurrentFastAppends(@TempDir File dir) throws Excepti

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
nastra commented on code in PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#discussion_r1366846991 ## docs/content/catalog.md: ## @@ -1,7 +1,10 @@ --- Review Comment: it should stay under `common` IMO. We just need to make sure we properly link to that page

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-20 Thread via GitHub
nastra commented on PR #8844: URL: https://github.com/apache/iceberg/pull/8844#issuecomment-1772561757 @RussellSpitzer any thoughts on changing the caching behavior here? I'm a little bit hesitant with this change. On one hand I can see a reasoning about wanting a table to expire from the c

Re: [PR] Nessie: reimplement create and drop namespace operations [iceberg]

2023-10-20 Thread via GitHub
adutra commented on PR #8857: URL: https://github.com/apache/iceberg/pull/8857#issuecomment-1772558933 Finally got the conflict handling logic right. I am going to switch this to draft until I finish migrating the remaning namespace ops (list, setProperties, removeProperties). --

Re: [I] iceberg HiveCatalog insert exception of GSS initiate failed [iceberg]

2023-10-20 Thread via GitHub
hzxiongyinke commented on issue #3127: URL: https://github.com/apache/iceberg/issues/3127#issuecomment-1772550326 The root cause is that the Hive Client CliDriver does not write the HMS DelegationToken into the UGI, while HiveServer2 has done. -- This is an automated message from the Apa

Re: [I] Alter table to generic types [iceberg]

2023-10-20 Thread via GitHub
paulboocock commented on issue #2791: URL: https://github.com/apache/iceberg/issues/2791#issuecomment-1772547132 We have two common type evolutions that we'd like to see: int -> float int -> string Actually, for the latter it's more like: numeric -> string This is f

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
liurenjie1024 commented on code in PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#discussion_r1366834493 ## docs/content/catalog.md: ## @@ -1,7 +1,10 @@ --- Review Comment: I moved it to doc on purpose since it seems that currently there is only once concept

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
ismailsimsek commented on PR #8340: URL: https://github.com/apache/iceberg/pull/8340#issuecomment-1772439329 @rdblue @dramaticlly resolved the issue with pure sql, and rolled back previous changes. could you please review it again? -- This is an automated message from the Apache Git Servi

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-20 Thread via GitHub
ismailsimsek commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1366755414 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -345,38 +345,44 @@ public static String deletePropertiesStatement(Set properties) { static

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
nastra commented on PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#issuecomment-1772340731 I think the initial goal was to have a `Concepts` section on the website and that catalog page was the first page to be shown here. So I think what we want here is ``` diff --git

Re: [PR] Update Slack URL [iceberg-docs]

2023-10-20 Thread via GitHub
nastra merged PR #286: URL: https://github.com/apache/iceberg-docs/pull/286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
nastra commented on PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#issuecomment-1772320478 I think the problem is rather that this page isn't linked in `docs/config.toml` under the main menu section: https://github.com/apache/iceberg-docs/blob/3211e6ec736b82c719e1b960cf839b3d

[PR] Update Slack URL [iceberg-docs]

2023-10-20 Thread via GitHub
Fokko opened a new pull request, #286: URL: https://github.com/apache/iceberg-docs/pull/286 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] fix: Add catalog pages to docs. [iceberg-docs]

2023-10-20 Thread via GitHub
nastra commented on code in PR #284: URL: https://github.com/apache/iceberg-docs/pull/284#discussion_r1366657036 ## docs/content/catalog.md: ## @@ -1,7 +1,10 @@ --- Review Comment: I see that the file got moved out of the `common` directory. It should actually stay in that

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-20 Thread via GitHub
zhangminglei commented on PR #8844: URL: https://github.com/apache/iceberg/pull/8844#issuecomment-1772312138 @nastra Hi, if has no problem, please merge this in ? šŸ˜„ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add sort_order_id to SCAN_COLUMNS to address null sort order ID in pā€¦ [iceberg]

2023-10-20 Thread via GitHub
nastra merged PR #8873: URL: https://github.com/apache/iceberg/pull/8873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] SortOrderId is null after planFile operation [iceberg]

2023-10-20 Thread via GitHub
nastra closed issue #8864: SortOrderId is null after planFile operation URL: https://github.com/apache/iceberg/issues/8864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Add sort_order_id to SCAN_COLUMNS to address null sort order ID in pā€¦ [iceberg]

2023-10-20 Thread via GitHub
nastra commented on PR #8873: URL: https://github.com/apache/iceberg/pull/8873#issuecomment-1772307552 thanks for fixing this @zhangminglei -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[PR] 1.4.1 release notes [iceberg-docs]

2023-10-20 Thread via GitHub
nastra opened a new pull request, #285: URL: https://github.com/apache/iceberg-docs/pull/285 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Add section on Github releases [iceberg-docs]

2023-10-20 Thread via GitHub
nastra merged PR #280: URL: https://github.com/apache/iceberg-docs/pull/280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[I] Document that distribution modes do not work with Create statements [Documentation] [iceberg]

2023-10-20 Thread via GitHub
a-agmon opened a new issue, #8887: URL: https://github.com/apache/iceberg/issues/8887 ### Apache Iceberg version 1.4.0 (latest release) ### Query engine Spark ### Please describe the bug šŸž Spark writes are currently using hash as default distribution modes,

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.13.30 to 3.14.2 [iceberg]

2023-10-20 Thread via GitHub
nastra merged PR #8790: URL: https://github.com/apache/iceberg/pull/8790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

2023-10-20 Thread via GitHub
nastra commented on code in PR #8805: URL: https://github.com/apache/iceberg/pull/8805#discussion_r1366551865 ## core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java: ## @@ -282,6 +286,16 @@ public void testPositionSetRowFilter() { @Test public void test

Re: [PR] Add sort_order_id to SCAN_COLUMNS to address null sort order ID in pā€¦ [iceberg]

2023-10-20 Thread via GitHub
zhangminglei commented on PR #8873: URL: https://github.com/apache/iceberg/pull/8873#issuecomment-1772195781 @Fokko @nastra Just a friendly ping, Can we merge it in ? šŸ˜ƒ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use