Shardul Mahadik created SPARK-35072: ---------------------------------------
Summary: spark.jars.ivysettings should support local:// and hdfs:// schemes Key: SPARK-35072 URL: https://issues.apache.org/jira/browse/SPARK-35072 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.2.0 Reporter: Shardul Mahadik During reviews of SPARK-34472, there was a desire to support local:// and hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See https://github.com/apache/spark/pull/31591#discussion_r598850998 and https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently this fails with the following error: {code:java} ./bin/spark-shell --conf spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf spark.jars.ivySettings="local:///Use rs/test/temp/ivysettings.xml" Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist at scala.Predef$.require(Predef.scala:281) at org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288) at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code} We should make sure that this also works with {{SparkContext#addJar}} when running in cluster mode. At this point, only YARN resource manager supports managing {{ivysettings}} file in cluster mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org