[
https://issues.apache.org/jira/browse/SPARK-35072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Mahadik updated SPARK-35072:
------------------------------------
Description:
During reviews of SPARK-34472, there was a desire to support local:// and
hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See
https://github.com/apache/spark/pull/31591#discussion_r598850998 and
https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently
this fails with the following error:
{code:java}
./bin/spark-shell --conf
spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf
spark.jars.ivySettings="local:///Use
rs/test/temp/ivysettings.xml"
Exception in thread "main" java.lang.IllegalArgumentException: requirement
failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
at scala.Predef$.require(Predef.scala:281)
at
org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
at
org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
We should make sure that this also works with {{SparkContext#addJar}} when
running in cluster mode. At this point, only YARN resource manager supports
managing {{ivysettings}} file in cluster mode. This can change based on
SPARK-35073.
was:
During reviews of SPARK-34472, there was a desire to support local:// and
hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See
https://github.com/apache/spark/pull/31591#discussion_r598850998 and
https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently
this fails with the following error:
{code:java}
./bin/spark-shell --conf
spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf
spark.jars.ivySettings="local:///Use
rs/test/temp/ivysettings.xml"
Exception in thread "main" java.lang.IllegalArgumentException: requirement
failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
at scala.Predef$.require(Predef.scala:281)
at
org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
at
org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
We should make sure that this also works with {{SparkContext#addJar}} when
running in cluster mode. At this point, only YARN resource manager supports
managing {{ivysettings}} file in cluster mode.
> spark.jars.ivysettings should support local:// and hdfs:// schemes
> ------------------------------------------------------------------
>
> Key: SPARK-35072
> URL: https://issues.apache.org/jira/browse/SPARK-35072
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Shardul Mahadik
> Priority: Minor
>
> During reviews of SPARK-34472, there was a desire to support local:// and
> hdfs:// schemes and potentially other schemes with spark.jars.ivysettings.
> See https://github.com/apache/spark/pull/31591#discussion_r598850998 and
> https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently
> this fails with the following error:
> {code:java}
> ./bin/spark-shell --conf
> spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf
> spark.jars.ivySettings="local:///Use
> rs/test/temp/ivysettings.xml"
> Exception in thread "main" java.lang.IllegalArgumentException: requirement
> failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not
> exist
> at scala.Predef$.require(Predef.scala:281)
> at
> org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
> at
> org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
> at
> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> We should make sure that this also works with {{SparkContext#addJar}} when
> running in cluster mode. At this point, only YARN resource manager supports
> managing {{ivysettings}} file in cluster mode. This can change based on
> SPARK-35073.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
