Dear All, I am having problems trying to access a hadoop cluster through a SOCKS proxy. I did the following:
1. Started a SOCKS proxy on my local machine using: ssh -D 10000 node-outside-firewall 2. Started the Hadoop cluster, node1 is the master and is behind the firewall. 3. Set proxy on a browser to verify that I could see the admin interface for the JobTracker and the DFS. It worked by visiting http://node1:50030. 4. Using java code, changed my config to use the proxy using: config.set("hadoop.socks.server", "localhost:10000"); config.set("hadoop.rpc.socket.factory.class.default", "org.apache.hadoop.net.SocksSocketFactory"); and used node1 as the master config.set("mapred.job.tracker", "node1:9001"); config.set("fs.default.name", "node1:9000"); 5. I get an exception: java.net.UnknownHostException: unknown host: node1 at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195) at org.apache.hadoop.ipc.Client.getConnection(Client.java:850) at org.apache.hadoop.ipc.Client.call(Client.java:720) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) Can it be that, while the browser forwards DNS requests through the SOCKS proxy, the default behavior of Hadoop is to do the DNS resolution locally? Or is there a mistake somewhere in my settings? Thanks in advance, Spyros
