Did you try ssh tunneling instead of SOCKS?
Thanks
Best Regards
On Wed, Mar 18, 2015 at 5:45 AM, Kelly, Jonathan jonat...@amazon.com
wrote:
I'm trying to figure out how I might be able to use Spark with a SOCKS
proxy. That is, my dream is to be able to write code in my IDE then run it
without much trouble on a remote cluster, accessible only via a SOCKS proxy
between the local development machine and the master node of the
cluster (ignoring, for now, any dependencies that would need to be
transferred--assume it's a very simple app with no dependencies that aren't
part of the Spark classpath on the cluster). This is possible with Hadoop
by setting hadoop.rpc.socket.factory.class.default to
org.apache.hadoop.net.SocksSocketFactory and hadoop.socks.server to
localhost:port on which a SOCKS proxy has been opened via ssh -D to the
master node. However, I can't seem to find anything like this for Spark,
and I only see very few mentions of it on the user list and on
stackoverflow, with no real answers. (See links below.)
I thought I might be able to use the JVM's -DsocksProxyHost and
-DsocksProxyPort system properties, but it still does not seem to work.
That is, if I start a SOCKS proxy to my master node using something like
ssh -D 2600 master node public name then run a simple Spark app that
calls SparkConf.setMaster(spark://master node private IP:7077), passing
in JVM args of -DsocksProxyHost=locahost -DsocksProxyPort=2600, the
driver hangs for a while before finally giving up (Application has been
killed. Reason: All masters are unresponsive! Giving up.). It seems like
it is not even attempting to use the SOCKS proxy. Do
-DsocksProxyHost/-DsocksProxyPort not even work for Spark?
http://stackoverflow.com/questions/28047000/connect-to-spark-through-a-socks-proxy
(unanswered
similar question from somebody else about a month ago)
https://issues.apache.org/jira/browse/SPARK-5004 (unresolved, somewhat
related JIRA from a few months ago)
Thanks,
Jonathan
