RussellSpitzer commented on code in PR #1857: URL: https://github.com/apache/polaris/pull/1857#discussion_r2145881707
########## site/content/in-dev/unreleased/polaris-spark-client.md: ########## @@ -128,3 +128,21 @@ The Polaris Spark client has the following functionality limitations: 3) Rename a Delta table is not supported. 4) ALTER TABLE ... SET LOCATION is not supported for DELTA table. 5) For other non-Iceberg tables like csv, it is not supported. + +## Iceberg Spark Client compatibility with Polaris Spark Client +The Polaris Spark client today is not designed to be used with Iceberg Spark client together. In other words, Review Comment: I think we really like the idea that we can ship a single jar that a user can use to enable this functionality. A lot of folks do not have access to maven and the ability to share a single jar and begin working is a great help. This comes up a lot. It also makes sense to me to provide the non-bundled resource just in case users do what to do just-in-time substitutions and possibly shade it other resources. I can say from my own previous experience the single Jar versions are much more popular with Spark plugins. We could essentially have two projects here (as in the Iceberg repo), the first being an unshaded version of the build which doesn't include any dependencies by default. End users would then download all the dependencies using --spark.packages or what not. The second would be a shadow jar project (polaris-spark-catalog-plugin-bundle) or something like that. Which just uses the first project as a dependency and shades everything in it. --- Now both of these things I don't think are necessarily related or blocking 1.0. If users start using the bundle and like it, then we should keep shipping it. If they also need the non-bundled version we can add that later. Even if we restructure the build it doesn't really matter as long as we keep the artifact names the same. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
