Hey,

Spark Submit adds maven central & spark bintray to the ChainResolver before it 
adds any external resolvers. 
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821

When running on a cluster without internet access, this means the spark shell 
takes forever to launch as it tries these two remote repos before the ones 
specified in the --repositories list. In our case we have a proxy which the 
cluster can access it and supply it via —repositories. This is also a problem 
for users who maintain a proxy for maven/ivy repos with something like 
Nexus/Artifactory.

I see two options for a fix;

  *   Change the order repos are added to the ChainResolver, making the 
--repositories supplied repos come before anything else. 
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843
  *   Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default 
true) which when false wont add the maven central & bintry to the ChainResolver.

Happy to do a PR now for this if someone can give me a recommendation on which 
option would be better.

JIRA here; https://issues.apache.org/jira/browse/SPARK-8475

Cheers,
Nathan

Reply via email to