[
https://issues.apache.org/jira/browse/SPARK-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311906#comment-15311906
]
Ion Alberdi commented on SPARK-11085:
-------------------------------------
Hello to all,
I reproduce the error using the docker in
https://github.com/Yannael/kafka-sparkstreaming-cassandra.
What I tried to communicate the -Dhttp.proxyHost, -Dhttp.proxyPort parameters:
- the javaopts worarkound mentionned above
- setting, spark.executor.extraJavaOptions. The launched command becomes
% java org.apache.spark.deploy.SparkSubmit ... --conf
spark.driver.extraJavaOptions=-Dhttp.proxyHost=<proxyHost>
-Dhttp.proxyPort=<proxyPort>
I wonder whether the shell is able to parse that line and thus transfer the two
(-Dhttp.proxyHost=<proxyHost> and -Dhttp.proxyPort=<proxyPort>) parameters to
org.apache.spark.deploy.SparkSubmit
- setting "--driver-java-options" parameters that ends with
% java org.apache.spark.deploy.SparkSubmit ... -Dhttp.proxyHost=<proxyHost>
-Dhttp.proxyPort=<proxyPort>
even if the shell seems more likely to parse the two informations, the packages
are not downloaded as the http request do not go through the proxy
> Add support for HTTP proxy
> ---------------------------
>
> Key: SPARK-11085
> URL: https://issues.apache.org/jira/browse/SPARK-11085
> Project: Spark
> Issue Type: Improvement
> Components: Spark Shell, Spark Submit
> Reporter: Dustin Cote
> Priority: Minor
>
> Add a way to update ivysettings.xml for the spark-shell and spark-submit to
> support proxy settings for clusters that need to access a remote repository
> through an http proxy. Typically this would be done like:
> JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxy.host -Dhttp.proxyPort=8080
> -Dhttps.proxyHost=proxy.host.secure -Dhttps.proxyPort=8080"
> Directly in the ivysettings.xml would look like:
> <ivysettings>
> <setproxy proxyhost="proxy.host"
> proxyport="8080"
> nonproxyhosts="nonproxy.host"/>
> </ivysettings>
> Even better would be a way to customize the ivysettings.xml with command
> options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
