gh-yzou commented on code in PR #1857: URL: https://github.com/apache/polaris/pull/1857#discussion_r2145559397
########## site/content/in-dev/unreleased/polaris-spark-client.md: ########## @@ -128,3 +128,21 @@ The Polaris Spark client has the following functionality limitations: 3) Rename a Delta table is not supported. 4) ALTER TABLE ... SET LOCATION is not supported for DELTA table. 5) For other non-Iceberg tables like csv, it is not supported. + +## Iceberg Spark Client compatibility with Polaris Spark Client +The Polaris Spark client today is not designed to be used with Iceberg Spark client together. In other words, Review Comment: @adutra Yes, there are certain drawbacks. However, as we mentioned before, there are couple of challenges: 1) some classes will be missing due to reflection (construct based on class name) used in the Iceberg class. 2) compatibility with other iceberg version needs to be tested, instead of letting user try to figure out what is the version they need to use to make sure thing are working. It would be easier to just include it to start with, so that user don't have to worry about that too much to begin with. Again, this is just the first release, future improvement can always be made once we add improvement and compatibility testing to provide better guarantee to the user. ########## site/content/in-dev/unreleased/polaris-spark-client.md: ########## @@ -128,3 +128,21 @@ The Polaris Spark client has the following functionality limitations: 3) Rename a Delta table is not supported. 4) ALTER TABLE ... SET LOCATION is not supported for DELTA table. 5) For other non-Iceberg tables like csv, it is not supported. + +## Iceberg Spark Client compatibility with Polaris Spark Client +The Polaris Spark client today is not designed to be used with Iceberg Spark client together. In other words, +there is no guarantee provided if both `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:<iceberg_version>` and +`org.apache.polaris:polaris-spark-3.5_2.12:<polaris_version>` are provided for the Spark `package` configuration. + +In order to help the usage of IcebergExtension use the same client, Polaris Spark client also ships a version of +Iceberg Spark client with it, the version information is described in the following table: + +| Spark Client Version | Iceberg Spark Client Version | +|----------------------|------------------------------| +| 1.0.0 | 1.9.0 | + +## Trouble Shooting +1. When starting spark, sometimes it complains it failed to download a package, for example, Review Comment: updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
