Dear All,

I am having problems trying to access a hadoop cluster through a SOCKS proxy. I 
did the following:

1. Started a SOCKS proxy on my local machine using:
ssh -D 10000 node-outside-firewall

2. Started the Hadoop cluster, node1 is the master and is behind the firewall.

3. Set proxy on a browser to verify that I could see the admin interface for 
the JobTracker and the DFS. It worked by visiting http://node1:50030.

4. Using java code, changed my config to use the proxy using:
config.set("hadoop.socks.server", "localhost:10000");
config.set("hadoop.rpc.socket.factory.class.default", 
"org.apache.hadoop.net.SocksSocketFactory");
and used node1 as the master
config.set("mapred.job.tracker", "node1:9001");
config.set("fs.default.name", "node1:9000");

5. I get an exception:
java.net.UnknownHostException: unknown host: node1
        at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:850)
        at org.apache.hadoop.ipc.Client.call(Client.java:720)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
        at 
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
        at 
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
        at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)

Can it be that, while the browser forwards DNS requests through the SOCKS 
proxy, the default behavior of Hadoop is to do the DNS resolution locally? Or 
is there a mistake somewhere in my settings?

Thanks in advance,
Spyros

Reply via email to