dimas-b commented on code in PR #3000: URL: https://github.com/apache/polaris/pull/3000#discussion_r2515253144
########## site/content/in-dev/unreleased/polaris-spark-client.md: ########## @@ -119,11 +119,16 @@ If you would like to use a version of the Spark client that is currently not yet build a Spark client jar locally from source. Please check out the Polaris repo and refer to the Spark plugin [README](https://github.com/apache/polaris/blob/main/plugins/spark/README.md) for detailed instructions. -## Limitations -The Polaris Spark client has the following functionality limitations: -1) Create table as select (CTAS) is not supported for Delta tables. As a result, the `saveAsTable` method of `Dataframe` +## Current Limitations +The following describes the current limitations of the Polaris Spark client: + +### General Limitations +1. The Polaris Spark client only supports Iceberg and Delta tables. It does not support other table formats like CSV, JSON, etc. +2. Generic tables (non-Iceberg tables) do not currently support credential vending. Review Comment: nit: `Generic Tables API ...` ########## site/content/in-dev/unreleased/generic-table.md: ########## @@ -157,13 +158,10 @@ curl -X DELETE http://localhost:8181/api/catalog/polaris/v1/delta_catalog/namesp For the complete and up-to-date API specification, see the [Catalog API Spec](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/apache/polaris/refs/heads/main/spec/generated/bundled-polaris-catalog-service.yaml). -## Limitations +## Current Limitations -Current limitations of Generic Table support: -1) Limited spec information. Currently, there is no spec for information like Schema, Partition etc. -2) No commit coordination or update capability provided at the catalog service level. - -Therefore, the catalog itself is unaware of anything about the underlying table except some of the loosely defined metadata. -It is the responsibility of the engine (and plugins used by the engine) to determine exactly how loading or committing data -should look like based on the metadata. For example, with the delta support, th delta log serialization, deserialization -and update all happens at client side. +There are some known limitations for the generic table support: +1. Generic tables provide limited spec information. For example, there is no spec for Schema or Partition. +2. There is no commit coordination provided by Polaris. It is the responsibility of the engine to coordinate loading and committing data. The catalog is only aware of the generic table fields above. +3. There is no update capability provided by Polaris. Any update to a generic table must be done through a drop and create. +4. Generic tables do not support credential vending. Review Comment: nit: `Generic Tables API does not support credential vending.`? I think we need to be clear on what is relevant to the API provided by Polaris and what belongs to table formats. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
