sfc-gh-mrojas opened a new issue, #12469:
URL: https://github.com/apache/iceberg/issues/12469
### Apache Iceberg version
1.8.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When using polaris catalog to write to Iceberg from Spark, it stopped
working after 1.7.1.
It does not work for 1.8.0 and 1.8.1
This setup fails
```
import findspark
findspark.init()
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when
EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.8.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.8.1.jar"
spark = SparkSession.builder.appName('iceberg_lab')\
.config('spark.driver.extraClassPath', EXTRA_CLASS_PATH) \
.config('spark.sql.extensions',
'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
.config('spark.sql.defaultCatalog', 'polaris') \
.config('spark.sql.catalog.polaris',
'org.apache.iceberg.spark.SparkCatalog') \
.config('spark.sql.catalog.polaris.type', 'rest') \
.config('spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation',
'true') \
.config('spark.sql.catalog.polaris.uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog')
\
.config('spark.sql.catalog.polaris.oauth2-server-uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens')
\
.config('spark.sql.catalog.polaris.credential',f'{CLIENT_ID}:{CLIENT_SECRET}') \
.config('spark.sql.catalog.polaris.warehouse','AZURE_CATALOG') \
.config('spark.sql.catalog.polaris.scope',f'PRINCIPAL_ROLE:ALL')
.getOrCreate()
```
Using old jars it works
EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.6.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.7.1.jar"
It retuns an error related to authentation:
```
com.azure.identity.CredentialUnavailableException: EnvironmentCredential
authentication unavailable. Environment variables are not fully configured.To
mitigate this issue, please refer to the troubleshooting guidelines here at
https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot
WorkloadIdentityCredential authentication unavailable. The workload options
are not fully configured. See the troubleshooting guide for more information.
https://aka.ms/azsdk/java/identity/workloadidentitycredential/troubleshoot
Managed Identity authentication is not available.
SharedTokenCacheCredential authentication unavailable. No accounts were
found in the cache.
IntelliJ Authentication not available. Please log in with Azure Tools for
IntelliJ plugin in the IDE. Fore more details refer to the troubleshooting
guidelines here at
https://aka.ms/azsdk/java/identity/intellijcredential/troubleshoot
AzureCliCredential authentication unavailable. Azure CLI not installed.To
mitigate this issue, please refer to the troubleshooting guidelines here at
https://aka.ms/azsdk/java/identity/azclicredential/troubleshoot
Az.Account module with version >= 2.2.0 is not installed. It needs to be
installed to use Azure PowerShell Credential.
AzureDeveloperCliCredential authentication unavailable. Azure Developer CLI
not installed.To mitigate this issue, please refer to the troubleshooting
guidelines here at
https://aka.ms/azsdk/java/identity/azdevclicredential/troubleshootTo mitigate
this issue, please refer to the troubleshooting guidelines here at
https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
