gh-yzou commented on code in PR #1889: URL: https://github.com/apache/polaris/pull/1889#discussion_r2152869098
########## site/content/in-dev/unreleased/generic-table.md: ########## @@ -0,0 +1,168 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Generic Table (Beta) +type: docs +weight: 435 +--- + +The Generic Table in Apache Polaris provides basic management support for non-Iceberg tables. + +With the Generic Table API, you can: +- Create generic tables under a namespace +- Load a generic table +- Drop a generic table +- List all generic tables under a namespace + +**NOTE** The current generic table is in beta release, there can still be incomplete features and bugs. Please use it +with caution and report any issue if encountered. + +## What is a Generic Table? + +A generic table in Polaris is a structured entity that defines the necessary information. Each +generic table entity contains the following properties: + +- **name** (required): A unique identifier for the table within a namespace +- **format** (required): The format for the generic table, i.e. "delta", "csv" +- **base-location** (optional): Table base location in URI format. For example: s3://<my-bucket>/path/to/table + - The table base location is a location that includes all files for the table + - A table with multiple disjoint locations (i.e. containing files that are outside the configured base location) is not compliant with the current generic table support in Polaris. + - If no location is provided, clients or users are responsible for managing the location. +- **properties** (optional): Properties for the generic table passed on creation +- **doc** (optional): Comment or description for the table + +## Generic Table API Vs. Iceberg Table API + +Generic Table provides a different set of APIs to operate on the Generic Table entities while Iceberg APIs operates on +the Iceberg Table entities. + +| Operations | **Generic Table API** | **Iceberg Table API** | +|--------------|-------------------------------------------------------------|-------------------------------------------------------------| +| Create Table | create an Iceberg table | create a generic table | +| Load Table | load an Iceberg table, load a generic table throws an error | load a generic table, load an Iceberg table throws an error | +| Drop Table | drop an Iceberg table, drop a generic table throws an error | drop a generic table, drop an Iceberg table throws an error | +| List Table | list all Iceberg tables | list all generic tables | + +Note that generic table shares the same namespace with Iceberg tables, the table name has to be unique under the same namespace. + +## Working with Generic Table + +There are two ways to work with Polaris Generic Tables today: +1) Directly communicate with Polaris through REST API calls using curl. Details will be described in the later section. +2) Use the Spark Client provided if you are working with Spark. Please refer to [Polaris Spark Client]({{% ref "polaris-spark-client" %}}) for detailed instructions. + +### Create a Generic Table + +To create a generic table, you need to provide the corresponding fields as described in [What is a Generic Table](#what-is-a-generic-table). +Following is the REST API for creating a generic Table: + +```json +POST /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables Review Comment: oh, sorry. i moved it as plain text with ``, and I also added an actual example with curl call. ########## site/content/in-dev/unreleased/generic-table.md: ########## @@ -0,0 +1,178 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Generic Table (Beta) +type: docs +weight: 435 +--- + +The Generic Table in Apache Polaris provides the following capabilities for non-Iceberg tables: +- Create a generic table under a namespace +- Load a generic table +- Drop a generic table +- List all generic tables under a namespace + +**NOTE** The current generic table is in beta release. Please use it with caution and report any issue if encountered. + +## What is a Generic Table? + +A generic table in Polaris is an entity that defines the following fields: + +- **name** (required): A unique identifier for the table within a namespace +- **format** (required): The format for the generic table, i.e. "delta", "csv" +- **base-location** (optional): Table base location in URI format. For example: s3://<my-bucket>/path/to/table + - The table base location is a location that includes all files for the table + - A table with multiple disjoint locations (i.e. containing files that are outside the configured base location) is not compliant with the current generic table support in Polaris. + - If no location is provided, clients or users are responsible for managing the location. +- **properties** (optional): Properties for the generic table passed on creation +- **doc** (optional): Comment or description for the table + +## Generic Table API Vs. Iceberg Table API + +Generic Table provides a different set of APIs to operate on the generic table entities while Iceberg APIs operates on +the Iceberg table entities. + +| Operations | **Iceberg Table API** | **Generic Table API** | +|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| +| Create Table | Create an Iceberg table | Create a generic table | +| Load Table | Load an Iceberg table. If the table to load is a generic table, you need to call the Generic Table loadTable API, otherwise a TableNotFoundException will be thrown | Load a generic table. Similarly, try to load an Iceberg table through Generic Table API will thrown a TableNotFoundException. | +| Drop Table | Drop an Iceberg table. Similar as load table, if the table to drop is a Generic table, a tableNotFoundException will be thrown. | Drop a generic table. Drop an Iceberg table through Generic table endpoint will thrown an TableNotFound Exception | +| List Table | List all Iceberg tables | List all generic tables | + +Note that generic table shares the same namespace with Iceberg tables, the table name has to be unique under the same namespace. + +## Working with Generic Table + +There are two ways to work with Polaris Generic Tables today: +1) Directly communicate with Polaris through REST API calls using tools such as `curl`. Details will be described in the later section. +2) Use the Spark client provided if you are working with Spark. Please refer to [Polaris Spark Client]({{% ref "polaris-spark-client" %}}) for detailed instructions. + +### Create a Generic Table + +To create a generic table, you need to provide the corresponding fields as described in [What is a Generic Table](#what-is-a-generic-table). +Following is the REST API for creating a generic Table: + +```shell +POST /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables +``` +```json +{ + "name": "<table_name>", + "format": "<table_format>", + "base-location": "<table_base_location>", + "doc": "<comment or description for table>", + "properties": { + "<property-key>": "<property-value>" + } +} +``` + +Here is an example to create a generic table with name `delta_table` and format as `delta` under a namespace `delta_ns` Review Comment: sorry for the late reply! as we don't have update capability today, rename will require a re-creation, I added some description at section Generic Table API Vs. Iceberg Table API ########## site/content/in-dev/unreleased/generic-table.md: ########## @@ -0,0 +1,178 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Generic Table (Beta) +type: docs +weight: 435 +--- + +The Generic Table in Apache Polaris provides the following capabilities for non-Iceberg tables: +- Create a generic table under a namespace +- Load a generic table +- Drop a generic table +- List all generic tables under a namespace + +**NOTE** The current generic table is in beta release. Please use it with caution and report any issue if encountered. + +## What is a Generic Table? + +A generic table in Polaris is an entity that defines the following fields: + +- **name** (required): A unique identifier for the table within a namespace +- **format** (required): The format for the generic table, i.e. "delta", "csv" +- **base-location** (optional): Table base location in URI format. For example: s3://<my-bucket>/path/to/table + - The table base location is a location that includes all files for the table + - A table with multiple disjoint locations (i.e. containing files that are outside the configured base location) is not compliant with the current generic table support in Polaris. + - If no location is provided, clients or users are responsible for managing the location. +- **properties** (optional): Properties for the generic table passed on creation +- **doc** (optional): Comment or description for the table + +## Generic Table API Vs. Iceberg Table API + +Generic Table provides a different set of APIs to operate on the generic table entities while Iceberg APIs operates on +the Iceberg table entities. + +| Operations | **Iceberg Table API** | **Generic Table API** | +|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| +| Create Table | Create an Iceberg table | Create a generic table | +| Load Table | Load an Iceberg table. If the table to load is a generic table, you need to call the Generic Table loadTable API, otherwise a TableNotFoundException will be thrown | Load a generic table. Similarly, try to load an Iceberg table through Generic Table API will thrown a TableNotFoundException. | +| Drop Table | Drop an Iceberg table. Similar as load table, if the table to drop is a Generic table, a tableNotFoundException will be thrown. | Drop a generic table. Drop an Iceberg table through Generic table endpoint will thrown an TableNotFound Exception | +| List Table | List all Iceberg tables | List all generic tables | + +Note that generic table shares the same namespace with Iceberg tables, the table name has to be unique under the same namespace. + +## Working with Generic Table + +There are two ways to work with Polaris Generic Tables today: +1) Directly communicate with Polaris through REST API calls using tools such as `curl`. Details will be described in the later section. +2) Use the Spark client provided if you are working with Spark. Please refer to [Polaris Spark Client]({{% ref "polaris-spark-client" %}}) for detailed instructions. + +### Create a Generic Table + +To create a generic table, you need to provide the corresponding fields as described in [What is a Generic Table](#what-is-a-generic-table). +Following is the REST API for creating a generic Table: + +```shell +POST /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables +``` +```json +{ + "name": "<table_name>", + "format": "<table_format>", + "base-location": "<table_base_location>", + "doc": "<comment or description for table>", + "properties": { + "<property-key>": "<property-value>" + } +} +``` + +Here is an example to create a generic table with name `delta_table` and format as `delta` under a namespace `delta_ns` +for catalog `delta_catalog`: + +```shell +POST /polaris/v1/delta_catalog/namespaces/delta_ns/generic-tables +``` +```json +{ + "name": "delta_table", + "format": "delta", + "base-location": "s3://<my-bucket>/path/to/table", + "doc": "delta table example", + "properties": { + "key1": "value1" + } +} +``` + +### Load a Generic Table +The REST API for load a generic table is the following: + +```shell +GET /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table} +``` + +Here is an example to load the table `delta_table`: +```shell +GET /polaris/v1/delta_catalog/namespaces/delta_ns/generic-tables/{generic-table} +``` +And the response looks like the following: +```json +{ + "table": { + "name": "delta_table", + "format": "delta", + "base-location": "s3://<my-bucket>/path/to/table", + "doc": "delta table example", + "properties": { + "key1": "value1" + } + } +} +``` + +### List a Generic Table +Here is the REST API for listing the generic tables under a given namespace: +```shell +GET /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/ +``` + +Following rest call lists all tables under namespace delta_namespace: +```shell +GET /polaris/v1/delta_catalog/namespaces/delta_ns/generic-tables/ +``` +Example Response: +```json +{ + "identifiers": [ + { + "namespace": ["delta_ns"], + "name": "delta_table" + } + ], + "next-page-token": null +} +``` + +### Drop a Generic Table +The drop generic table REST API is the following: +```shell +DELETE /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table} +``` + +To drop the table `delat_table`, use the following: +```shell +DELETE /polaris/v1/delta_catalog/namespaces/delta_ns/generic-tables/{generic-table} +``` + +### API Reference + +For the complete and up-to-date API specification, see the [Catalog API Spec](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/apache/polaris/refs/heads/main/spec/generated/bundled-polaris-catalog-service.yaml). + + +## Limitations + +The Generic Table support today is very limited: Review Comment: updated ########## site/content/in-dev/unreleased/generic-table.md: ########## @@ -0,0 +1,168 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Generic Table (Beta) +type: docs +weight: 435 +--- + +The Generic Table in Apache Polaris provides basic management support for non-Iceberg tables. + +With the Generic Table API, you can: +- Create generic tables under a namespace +- Load a generic table +- Drop a generic table +- List all generic tables under a namespace + +**NOTE** The current generic table is in beta release, there can still be incomplete features and bugs. Please use it +with caution and report any issue if encountered. + +## What is a Generic Table? + +A generic table in Polaris is a structured entity that defines the necessary information. Each +generic table entity contains the following properties: + +- **name** (required): A unique identifier for the table within a namespace +- **format** (required): The format for the generic table, i.e. "delta", "csv" +- **base-location** (optional): Table base location in URI format. For example: s3://<my-bucket>/path/to/table + - The table base location is a location that includes all files for the table + - A table with multiple disjoint locations (i.e. containing files that are outside the configured base location) is not compliant with the current generic table support in Polaris. + - If no location is provided, clients or users are responsible for managing the location. +- **properties** (optional): Properties for the generic table passed on creation Review Comment: sounds good! i added the following description ``` - Currently, there is no reserved property key defined. - The definition and interpretation is delegated to client or engine implementations. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
