sebbegg opened a new issue, #1619: URL: https://github.com/apache/sedona/issues/1619
Hi, we're currently using sedona mostly in on Azure Databricks. With databricks strongly promoting it's Unity Catalog we've tested/investigated this and found some issues with running sedona on Databricks Clusters using the shared access mode. In this context spark (apparently) runs via spark-connect. ## Actual behavior Calling `SedonaContext.create(spark)` on such a cluster produces the following output: <img width="1334" alt="image" src="https://github.com/user-attachments/assets/cae6c518-8b6e-4d93-a847-402c4a349e74";> After doing a bit of research, it seems that the python API in general heavily relies on the `spark._jvm` attribute, which doesn't exist in when running via spark-connect. Are there any plans/possibilities to make the python apis spark-connect compatible? ## Steps to reproduce the problem Setup databricks cluster: * Databricks Runtime 14.3 LTS * `%pip install apache-sedona==1.6.0` (with appropriate java deps installed) ## Settings Sedona version = 1.6.0 Apache Spark version = 3.5.0 API type = Python Scala version = 2.12 JRE version = 1.8 Python version = 3.10 Environment = Azure Databricks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
