Hi Aneela,
1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node.
For this node datanode connection is successfull as mentioned in below logs.
2016-06-29 10:01:35,700 INFO
SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
nn/hadoop-master@platalyticsrealm (auth:KERBEROS)
2016-06-29 10:01:35,744 INFO
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)
for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol
2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology:
Adding a new node: /default-rack/192.168.23.206:1004
2. For the other node, 'hadoop-slave' kerberos athentication is successfull,
but ServiceAuthorizationManager check failed.
2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)
2016-06-29 10:01:37,512 WARN
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for
protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol,
expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm
2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from
192.168.23.207:32807 for protocol
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for
user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)
reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both
nodes differ. I can see that this configuration in hadoop-master's
hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to
'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.
Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm'
and restart all NNs and DNs, and check again.
If this does not help, then please share the hdfs-site.xml of hadoop-slave node
too.
-Vinay
From: Aneela Saleem [mailto:ane...@platalytics.com]
Sent: 29 June 2016 21:35
To: user@hadoop.apache.org
Subject: Fwd: datanode is unable to connect to namenode
Sent from my iPhone
Begin forwarded message:
From: Aneela Saleem <ane...@platalytics.com<mailto:ane...@platalytics.com>>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>>
Subject: Re: datanode is unable to connect to namenode
Attached are the log files for datanode and namenode. Also i have attached
hdfs-site.xml for namenode please check if there are any issues in
configuration file.
I have following two Kerberos Principals:
nn/hadoop-master
dn/hadoop-slave
i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file
on datanode. And i have starting services with principal nn/hadoop-master.
On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni .
<sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>> wrote:
Probably sharing both Name node and datanode logs may help.
On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem
<ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote:
Following is the result of telnet
Trying 192.168.23.206...
Connected to hadoop-master.
Escape character is '^]'.
On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem
<ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote:
Thanks Sreebalineni for the response.
This is the result of the netstat -a | grep 8020 command
tcp 0 0 hadoop-master:8020 *:* LISTEN
tcp 0 0 hadoop-master:33356 hadoop-master:8020 ESTABLISHED
tcp 0 0 hadoop-master:8020 hadoop-master:33356 ESTABLISHED
tcp 0 0 hadoop-master:55135 hadoop-master:8020 TIME_WAIT
And this is my /etc/hosts file
#127.0.0.1 localhost
#127.0.1.1 vm6-VirtualBox
192.168.23.206 hadoop-master platalytics.com<http://platalytics.com>
vm6-VirtualBox
192.168.23.207 hadoop-slave
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Can you please tell me what's wrong with above configuration and how can i
check whether it is firewall issue?
Thanks
On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni .
<sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>> wrote:
Are you able to telnet ping. Check the firewalls as well
On Jun 29, 2016 12:39 AM, "Aneela Saleem"
<ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote:
Hi all,
I have setup two nodes cluster with security enabled. I have everything running
successful like namenode, datanode, resourcemanager, nodemanager,
jobhistoryserver etc. But datanode is unable to connect to namenode, as i can
see only one node on the web UI. checking logs of datanode gives following
warning:
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
server: hadoop-master/192.168.23.206:8020<http://192.168.23.206:8020>
Rest of the things look fine. Please help me in this regard, what could be the
issue?
