Weichen Xu created SPARK-50281:
----------------------------------
Summary: pyspark local session `spark.jars` configuration does not
work
Key: SPARK-50281
URL: https://issues.apache.org/jira/browse/SPARK-50281
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.0.0
Reporter: Weichen Xu
pyspark local session `spark.jars` configuration does not work
Reproducing code:
if I do the following in a python shell:
{code:java}
from pyspark.sql import SparkSession
spark=SparkSession.builder.config("spark.jars",
<my_jar_file_path>).master("local[*]").getOrCreate() {code}
then do:
{code:java}
spark.sparkContext._jvm.<my-class>
{code}
it can't access the Java class correctly.
this issue only happens in spark master, and it causes our CI failure:
https://github.com/mlflow-automation/mlflow/actions/runs/11765501211/job/32775243482#step:12:291
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
