Please note, there are two different configs. “dfs.datanode.kerberos.principal” and “dfs.namenode.kerberos.principal”
Following configs can be set, as required. dfs.datanode.kerberos.principal --> dn/_HOST dfs.namenode.kerberos.principal --> nn/_HOST “nn/_HOST” will be used only in namenode side. -Vinay From: Aneela Saleem [mailto:ane...@platalytics.com] Sent: 30 June 2016 13:24 To: Vinayakumar B <vinayakumar...@huawei.com> Cc: user@hadoop.apache.org Subject: Re: datanode is unable to connect to namenode Thanks Vinayakumar Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode can connect to namenode successfully. So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many tutorials that the convention is to have different principals for all services like dn/_HOST for datanode nn/_HOST for namenode sn/_HOST for secondarynamenode etc Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml Thanks On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <vinayakumar...@huawei.com<mailto:vinayakumar...@huawei.com>> wrote: Hi Aneela, 1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs. 2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) 2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol 2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004<http://192.168.23.206:1004> 2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed. 2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) 2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm 2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807<http://192.168.23.207:32807> for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations. Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again. If this does not help, then please share the hdfs-site.xml of hadoop-slave node too. -Vinay From: Aneela Saleem [mailto:ane...@platalytics.com<mailto:ane...@platalytics.com>] Sent: 29 June 2016 21:35 To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: Fwd: datanode is unable to connect to namenode Sent from my iPhone Begin forwarded message: From: Aneela Saleem <ane...@platalytics.com<mailto:ane...@platalytics.com>> Date: 29 June 2016 at 10:16:36 GMT+5 To: "sreebalineni ." <sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>> Subject: Re: datanode is unable to connect to namenode Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file. I have following two Kerberos Principals: nn/hadoop-master dn/hadoop-slave i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master. On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>> wrote: Probably sharing both Name node and datanode logs may help. On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote: Following is the result of telnet Trying 192.168.23.206... Connected to hadoop-master. Escape character is '^]'. On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote: Thanks Sreebalineni for the response. This is the result of the netstat -a | grep 8020 command tcp 0 0 hadoop-master:8020 *:* LISTEN tcp 0 0 hadoop-master:33356 hadoop-master:8020 ESTABLISHED tcp 0 0 hadoop-master:8020 hadoop-master:33356 ESTABLISHED tcp 0 0 hadoop-master:55135 hadoop-master:8020 TIME_WAIT And this is my /etc/hosts file #127.0.0.1 localhost #127.0.1.1 vm6-VirtualBox 192.168.23.206 hadoop-master platalytics.com<http://platalytics.com> vm6-VirtualBox 192.168.23.207 hadoop-slave # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue? Thanks On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <sreebalin...@gmail.com<mailto:sreebalin...@gmail.com>> wrote: Are you able to telnet ping. Check the firewalls as well On Jun 29, 2016 12:39 AM, "Aneela Saleem" <ane...@platalytics.com<mailto:ane...@platalytics.com>> wrote: Hi all, I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning: WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020<http://192.168.23.206:8020> Rest of the things look fine. Please help me in this regard, what could be the issue?
