adutra commented on code in PR #1857: URL: https://github.com/apache/polaris/pull/1857#discussion_r2145569033
########## site/content/in-dev/unreleased/polaris-spark-client.md: ########## @@ -128,3 +128,21 @@ The Polaris Spark client has the following functionality limitations: 3) Rename a Delta table is not supported. 4) ALTER TABLE ... SET LOCATION is not supported for DELTA table. 5) For other non-Iceberg tables like csv, it is not supported. + +## Iceberg Spark Client compatibility with Polaris Spark Client +The Polaris Spark client today is not designed to be used with Iceberg Spark client together. In other words, Review Comment: @gh-yzou can you clarify item 1 above? I.e. "some classes will be missing due to reflection (construct based on class name) used in the Iceberg class". For item 2, I'd say this is already the case for anybody. I could mix the wrong Spark runtime jar with an incompatible AWS SDK package and things would break at runtime. Also, not sure if asking users to _replace_ the Spark runtime jar with the Polaris/Spark runtime jar isn't actually worse than asking them to _add_ an extra jar while keeping the Spark runtime jar they are already using. This would obviously require us to publish and maintain a detailed matrix compatibility table. But we'll have to do it anyways imo. How de we consider this issue wrt to the upcoming 1.0 Polaris release? Changing this after the 1.0 release will be impossible, I guess. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
