andreaschat-db commented on PR #56039: URL: https://github.com/apache/spark/pull/56039#issuecomment-4613112745
> @andreaschat-db, what do you think? Can we support SQL on path on the Delta side? If we do that, we can start transactions pretty easily as I assume `DeltaCatalog` would implement the new transactional API. > > We already have `SupportsCatalogOptions` for non-SQL transactions. We only care about SQL cases here. My understanding is that yes we can do this on the connector (Delta) side. The requirement is that for path based tables the current catalog needs to be a transactional catalog that can also understand the path. IIUC this is consistent today with how delta works. If the current catalog is set to a non-delta catalog, delta paths won't work. The idea in this PRs was to have a more consistent handling of paths in Spark since we are already doing this in Dataframes. There are advantages of doing that (uniform handling across connectors, consistency with Dataframe handling, catalog extraction that independent of the current catalog etc) but I am ok of pushing this to the connector. Regarding the concerns above: 1) We could narrow down the scope by creating `CatalogAndIdentifierWithPathBasedResolution`. This does whatever `CatalogAndIdentifier` plus the path based resolution. Then can cherry pick which call sites use the former and which the latter. 2) To guard against `SupportsNamespaces` being optional, we can change the resolution order so that if the current catalog does not implement `SupportsNamespaces`, never tries to resolve via the data source code path. Therefore, the behaviour remains the same for connectors that do not adapt their implementation. Connectors that want to support path based resolution need to also implement `SupportsNamespaces`. 3) We could introduce a new trait on top of `SupportsCatalogOptions`. Only connectors that implement the new trait get the path resolution in SQL. This also separates the intent between supporting SCO in Dataframes and SQL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
