You need connectivity from edge node to the entire cluster, not just
namenode. Your topology, unfortunately, probably won’t work too well. A
proper VPN / IPSec tunnel might be a better idea.
On Thu, Sep 12, 2019 at 12:04 AM saurabh pratap singh <
saurabh.cs...@gmail.com> wrote:
> Hadoop version :
Thank you so much for your reply .
I have further question there are some blogs which talks about some similar
setup like this one
https://github.com/vkovalchuk/hadoop-2.6.0-windows/wiki/How-to-access-HDFS-behind-firewall-using-SOCKS-proxy
I am just curious how does that works.
On Thu, Sep 12,
Hi
Hadoop is designed to avoid proxy as it will act as a bottleneck. Namenodes
are used to obtain a direct socket client / datanodes that is specific to
each job.
Le ven. 13 sept. 2019 à 14:21, Tony S. Wu a écrit :
> You need connectivity from edge node to the entire cluster, not just
> namenode
You will have to use a socks proxy (-D option in ssh tunnel). In addition,
when invoking hadoop fs command, you will have to add -Dsocks.proxyHost and
- Dsocks.proxyPort.
Thanks,
Hariharan
On Thu, 12 Sep 2019, 23:26 saurabh pratap singh,
wrote:
> Thank you so much for your reply .
> I have furt
Thank you all for your help .
Solution that worked for me is as follows:
I opened ssh tunnel for namenode which ensure that hadoop fs -ls works
In order for hadoop fs -put to work (as it was timing out because namenode
was returning private ip addresses of datanode which cant be resolved by
edge ma
Hi all
So I was not satisfied with the above mentioned approach and tried hadoop
socks server config at client end and used ssh with -D option as mentioned
by Hariharan Iyer (Thank you for that) and it worked as expected without
the need of opening separate ssh tunnels for data nodes.
Thanks.
On