[
https://issues.apache.org/jira/browse/SPARK-50281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Xu updated SPARK-50281:
-------------------------------
Description:
pyspark local session `spark.jars` configuration does not work
Reproducing code:
if I do the following in a python shell:
{code:java}
from pyspark.sql import SparkSession
spark=SparkSession.builder.config("spark.jars",
<my_jar_file_path>).master("local[*]").getOrCreate() {code}
then do:
{code:java}
spark.sparkContext._jvm.aaa.bbb.ccc # aaa.bbb.ccc is the class in the jar file
{code}
it can't access the Java class correctly.
but if I do:
{code:java}
bin/pyspark --jars=<my_jar_file_path>{code}
then in the pyspark shell, the issue vanished.
this issue only happens in spark master, and it causes our CI failure:
[https://github.com/mlflow-automation/mlflow/actions/runs/11765501211/job/32775243482#step:12:291]
was:
pyspark local session `spark.jars` configuration does not work
Reproducing code:
if I do the following in a python shell:
{code:java}
from pyspark.sql import SparkSession
spark=SparkSession.builder.config("spark.jars",
<my_jar_file_path>).master("local[*]").getOrCreate() {code}
then do:
{code:java}
spark.sparkContext._jvm.aaa.bbb.ccc # aaa.bbb.ccc is the class in the jar file
{code}
it can't access the Java class correctly.
this issue only happens in spark master, and it causes our CI failure:
[https://github.com/mlflow-automation/mlflow/actions/runs/11765501211/job/32775243482#step:12:291]
> pyspark local session `spark.jars` configuration does not work
> ---------------------------------------------------------------
>
> Key: SPARK-50281
> URL: https://issues.apache.org/jira/browse/SPARK-50281
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.0.0
> Reporter: Weichen Xu
> Priority: Major
>
> pyspark local session `spark.jars` configuration does not work
>
> Reproducing code:
>
> if I do the following in a python shell:
> {code:java}
> from pyspark.sql import SparkSession
> spark=SparkSession.builder.config("spark.jars",
> <my_jar_file_path>).master("local[*]").getOrCreate() {code}
> then do:
> {code:java}
> spark.sparkContext._jvm.aaa.bbb.ccc # aaa.bbb.ccc is the class in the jar
> file
> {code}
> it can't access the Java class correctly.
>
> but if I do:
> {code:java}
> bin/pyspark --jars=<my_jar_file_path>{code}
> then in the pyspark shell, the issue vanished.
>
> this issue only happens in spark master, and it causes our CI failure:
> [https://github.com/mlflow-automation/mlflow/actions/runs/11765501211/job/32775243482#step:12:291]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
