[ https://issues.apache.org/jira/browse/SPARK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Burak Yavuz updated SPARK-8475: ------------------------------- Issue Type: Improvement (was: Bug) > SparkSubmit with Ivy jars is very slow to load with no internet access > ---------------------------------------------------------------------- > > Key: SPARK-8475 > URL: https://issues.apache.org/jira/browse/SPARK-8475 > Project: Spark > Issue Type: Improvement > Components: Spark Submit > Affects Versions: 1.4.0 > Reporter: Nathan McCarthy > Priority: Minor > > Spark Submit adds maven central & spark bintray to the ChainResolver before > it adds any external resolvers. > https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821 > When running on a cluster without internet access, this means the spark shell > takes forever to launch as it tries these two remote repos before the ones > specified in the --repositories list. In our case we have a proxy which the > cluster can access it and supply it via --repositories. > This is also a problem for users who maintain a proxy for maven/ivy repos > with something like Nexus/Artifactory. Having a repo proxy is popular at many > organisations so I'd say this would be a useful change for these users as > well. In the current state even if a maven central proxy is supplied, it will > still try and hit central. > I see two options for a fix; > * Change the order repos are added to the ChainResolver, making the > --repositories supplied repos come before anything else. > https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843 > > * Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default > true) which when false wont add the maven central & bintry to the > ChainResolver. > Happy to do a PR for this fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org