[
https://issues.apache.org/jira/browse/SPARK-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955001#comment-14955001
]
Dustin Cote commented on SPARK-11085:
-------------------------------------
[~sowen] The problem here is that the dependencies to be downloaded with
--packages can't be reached because those settings do not get forwarded into
the respective Spark client. I'll note this was being tried with Spark on YARN
and the JAVA_OPTS was being set through spark.driver.extraJavaOptions. The
ivysettings change was being done through ~/.m2/ivysettings.xml. It's more of
a forwarding the settings to the Spark client issue.
At least on CDH, the relevant ivysettings.xml is bundled in the assembly jar
and apparently not modified by the two methods:
:: loading settings :: url =
jar:file:/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p819.487/jars/spark-assembly-1.3.0-cdh5.4.2-hadoop2.6.0-cdh5.4.2.jar!/org/apache/ivy/core/settings/ivysettings.xml
This JIRA would be to come up with a way to modify or override this
ivysettings.xml so that it can be used with proxy settings.
> Add support for HTTP proxy
> ---------------------------
>
> Key: SPARK-11085
> URL: https://issues.apache.org/jira/browse/SPARK-11085
> Project: Spark
> Issue Type: Improvement
> Components: Spark Shell, Spark Submit
> Reporter: Dustin Cote
> Priority: Minor
>
> Add a way to update ivysettings.xml for the spark-shell and spark-submit to
> support proxy settings for clusters that need to access a remote repository
> through an http proxy. Typically this would be done like:
> JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxy.host -Dhttp.proxyPort=8080
> -Dhttps.proxyHost=proxy.host.secure -Dhttps.proxyPort=8080"
> Directly in the ivysettings.xml would look like:
> <ivysettings>
> <setproxy proxyhost="proxy.host"
> proxyport="8080"
> nonproxyhosts="nonproxy.host"/>
> </ivysettings>
> Even better would be a way to customize the ivysettings.xml with command
> options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
