Thanks Vinayakumar Yes you got it right i was using different principal names i.e., *nn/_HOST* for namenode and *dn/_HOST* for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode can connect to namenode successfully.
So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many tutorials that the convention is to have different principals for all services like dn/_HOST for datanode nn/_HOST for namenode sn/_HOST for secondarynamenode etc Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml Thanks On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <vinayakumar...@huawei.com> wrote: > Hi Aneela, > > > > 1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' > node. For this node datanode connection is successfull as mentioned in > below logs. > > > > 2016-06-29 10:01:35,700 INFO > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for > nn/hadoop-master@platalyticsrealm (auth:KERBEROS) > > 2016-06-29 10:01:35,744 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for nn/hadoop-master@platalyticsrealm > (auth:KERBEROS) for protocol=interface > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol > > > > 2016-06-29 10:01:36,845 INFO > org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/ > 192.168.23.206:1004 > > > > > > 2. For the other node, 'hadoop-slave' kerberos athentication is > successfull, but ServiceAuthorizationManager check failed. > > > > 2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > > 2016-06-29 10:01:37,512 WARN > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > for protocol=interface > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client > Kerberos principal is nn/hadoop-slave@platalyticsrealm > > 2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from > 192.168.23.207:32807 for protocol > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for > user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > > > > reason could be mostly, "dfs.datanode.kerberos.principal" configuration in > both nodes differ. I can see that this configuration in hadoop-master's > hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been > set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations. > > > > Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' > and restart all NNs and DNs, and check again. > > > > If this does not help, then please share the hdfs-site.xml of hadoop-slave > node too. > > > > -Vinay > > > > *From:* Aneela Saleem [mailto:ane...@platalytics.com] > *Sent:* 29 June 2016 21:35 > *To:* user@hadoop.apache.org > *Subject:* Fwd: datanode is unable to connect to namenode > > > > > > Sent from my iPhone > > > Begin forwarded message: > > *From:* Aneela Saleem <ane...@platalytics.com> > *Date:* 29 June 2016 at 10:16:36 GMT+5 > *To:* "sreebalineni ." <sreebalin...@gmail.com> > *Subject:* *Re: datanode is unable to connect to namenode* > > Attached are the log files for datanode and namenode. Also i have attached > hdfs-site.xml for namenode please check if there are any issues in > configuration file. > > > > I have following two Kerberos Principals: > > > > nn/hadoop-master > > dn/hadoop-slave > > > > i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab > file on datanode. And i have starting services with principal > nn/hadoop-master. > > > > On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <sreebalin...@gmail.com> > wrote: > > Probably sharing both Name node and datanode logs may help. > > > > On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <ane...@platalytics.com> > wrote: > > Following is the result of telnet > > > > Trying 192.168.23.206... > > Connected to hadoop-master. > > Escape character is '^]'. > > > > On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <ane...@platalytics.com> > wrote: > > Thanks Sreebalineni for the response. > > This is the result of the *netstat -a | grep 8020* command > > tcp 0 0 hadoop-master:8020 *:* LISTEN > > tcp 0 0 hadoop-master:33356 hadoop-master:8020 > ESTABLISHED > > tcp 0 0 hadoop-master:8020 hadoop-master:33356 > ESTABLISHED > > tcp 0 0 hadoop-master:55135 hadoop-master:8020 > TIME_WAIT > > And this is my */etc/hosts* file > > #127.0.0.1 localhost > > #127.0.1.1 vm6-VirtualBox > > 192.168.23.206 hadoop-master platalytics.com vm6-VirtualBox > > 192.168.23.207 hadoop-slave > > # The following lines are desirable for IPv6 capable hosts > > ::1 ip6-localhost ip6-loopback > > fe00::0 ip6-localnet > > ff00::0 ip6-mcastprefix > > ff02::1 ip6-allnodes > > ff02::2 ip6-allrouters > > > > Can you please tell me what's wrong with above configuration and how can i > check whether it is firewall issue? > > Thanks > > > > On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <sreebalin...@gmail.com> > wrote: > > Are you able to telnet ping. Check the firewalls as well > > On Jun 29, 2016 12:39 AM, "Aneela Saleem" <ane...@platalytics.com> wrote: > > Hi all, > > > > I have setup two nodes cluster with security enabled. I have everything > running successful like namenode, datanode, resourcemanager, nodemanager, > jobhistoryserver etc. But datanode is unable to connect to namenode, as i > can see only one node on the web UI. checking logs of datanode gives > following warning: > > > > *WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting > to server: hadoop-master/192.168.23.206:8020 <http://192.168.23.206:8020>* > > > > Rest of the things look fine. Please help me in this regard, what could be > the issue? > > > > > > > > > >
