Thanks Vinaykumar and Gurmukh, I have made it working successfully through auth_to_local configs. But i faced much issues.
Actually I have two nodes cluster, one being namenode and datanode, and second being datanode only. I faced some authentication from keytab related issues. for example: I added nn/hadoop-master to both nn.keytab and dn.keytab and did the same with dn/hadoop-slave (following you github dn.keytab file). but when i start cluster i got following error: *Login failure for nn/hadoop-master@platalyticsrealm from keytab /etc/hadoop/conf/hdfs.keytab: javax.security.auth.login.LoginException: Checksum failed* i verified the authentication of nn/hadoop-master with keytab nn/hadoop-master through kinit, but couldn't do so. because of error like *could not verify credentials* i then removed nn/hadoop-master from dn.keytab then it authenticated successfully. And i removed all the hadoop-master principals from dn.keytab and hadoop-slave principals from nn.keytab. So does it mean that a principal can't belong to more than one keytabs? And please make some time to review the attached hdfs-site.xml for both namenode and datanode, and keytab files. And point me out if something wrong. Thanks On Thu, Jun 30, 2016 at 1:21 PM, Vinayakumar B <vinayakumar...@huawei.com> wrote: > Please note, there are two different configs. > > > > “dfs.datanode.kerberos.principal” and “dfs.namenode.kerberos.principal” > > > > Following configs can be set, as required. > > > > dfs.datanode.kerberos.principal à dn/_HOST > > dfs.namenode.kerberos.principal à nn/_HOST > > > > “nn/_HOST” will be used only in namenode side. > > > > -Vinay > > *From:* Aneela Saleem [mailto:ane...@platalytics.com] > *Sent:* 30 June 2016 13:24 > *To:* Vinayakumar B <vinayakumar...@huawei.com> > *Cc:* user@hadoop.apache.org > > *Subject:* Re: datanode is unable to connect to namenode > > > > Thanks Vinayakumar > > > > Yes you got it right i was using different principal names i.e., > *nn/_HOST* for namenode and *dn/_HOST* for datanode. Setting the same > principal name for both datanode and namenode i.e., > hdfs/_HOST@platalyticsrealm solved the issue. Now datanode > > can connect to namenode successfully. > > > > So my question is, is it mandatory to have same principal name on all > hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many > > tutorials that the convention is to have different principals for all > services like > > dn/_HOST for datanode > > nn/_HOST for namenode > > sn/_HOST for secondarynamenode etc > > > > Secondly for map reduce and yarn, would that mapred-site.xml and > yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml > > > > Thanks > > > > On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <vinayakumar...@huawei.com> > wrote: > > Hi Aneela, > > > > 1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' > node. For this node datanode connection is successfull as mentioned in > below logs. > > > > 2016-06-29 10:01:35,700 INFO > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for > nn/hadoop-master@platalyticsrealm (auth:KERBEROS) > > 2016-06-29 10:01:35,744 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for nn/hadoop-master@platalyticsrealm > (auth:KERBEROS) for protocol=interface > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol > > > > 2016-06-29 10:01:36,845 INFO > org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/ > 192.168.23.206:1004 > > > > > > 2. For the other node, 'hadoop-slave' kerberos athentication is > successfull, but ServiceAuthorizationManager check failed. > > > > 2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > > 2016-06-29 10:01:37,512 WARN > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > for protocol=interface > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client > Kerberos principal is nn/hadoop-slave@platalyticsrealm > > 2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from > 192.168.23.207:32807 for protocol > org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for > user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) > > > > reason could be mostly, "dfs.datanode.kerberos.principal" configuration in > both nodes differ. I can see that this configuration in hadoop-master's > hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been > set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations. > > > > Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' > and restart all NNs and DNs, and check again. > > > > If this does not help, then please share the hdfs-site.xml of hadoop-slave > node too. > > > > -Vinay > > > > *From:* Aneela Saleem [mailto:ane...@platalytics.com] > *Sent:* 29 June 2016 21:35 > *To:* user@hadoop.apache.org > *Subject:* Fwd: datanode is unable to connect to namenode > > > > > > Sent from my iPhone > > > Begin forwarded message: > > *From:* Aneela Saleem <ane...@platalytics.com> > *Date:* 29 June 2016 at 10:16:36 GMT+5 > *To:* "sreebalineni ." <sreebalin...@gmail.com> > *Subject:* *Re: datanode is unable to connect to namenode* > > Attached are the log files for datanode and namenode. Also i have attached > hdfs-site.xml for namenode please check if there are any issues in > configuration file. > > > > I have following two Kerberos Principals: > > > > nn/hadoop-master > > dn/hadoop-slave > > > > i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab > file on datanode. And i have starting services with principal > nn/hadoop-master. > > > > On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <sreebalin...@gmail.com> > wrote: > > Probably sharing both Name node and datanode logs may help. > > > > On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <ane...@platalytics.com> > wrote: > > Following is the result of telnet > > > > Trying 192.168.23.206... > > Connected to hadoop-master. > > Escape character is '^]'. > > > > On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <ane...@platalytics.com> > wrote: > > Thanks Sreebalineni for the response. > > This is the result of the *netstat -a | grep 8020* command > > tcp 0 0 hadoop-master:8020 *:* LISTEN > > tcp 0 0 hadoop-master:33356 hadoop-master:8020 > ESTABLISHED > > tcp 0 0 hadoop-master:8020 hadoop-master:33356 > ESTABLISHED > > tcp 0 0 hadoop-master:55135 hadoop-master:8020 > TIME_WAIT > > And this is my */etc/hosts* file > > #127.0.0.1 localhost > > #127.0.1.1 vm6-VirtualBox > > 192.168.23.206 hadoop-master platalytics.com vm6-VirtualBox > > 192.168.23.207 hadoop-slave > > # The following lines are desirable for IPv6 capable hosts > > ::1 ip6-localhost ip6-loopback > > fe00::0 ip6-localnet > > ff00::0 ip6-mcastprefix > > ff02::1 ip6-allnodes > > ff02::2 ip6-allrouters > > > > Can you please tell me what's wrong with above configuration and how can i > check whether it is firewall issue? > > Thanks > > > > On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <sreebalin...@gmail.com> > wrote: > > Are you able to telnet ping. Check the firewalls as well > > On Jun 29, 2016 12:39 AM, "Aneela Saleem" <ane...@platalytics.com> wrote: > > Hi all, > > > > I have setup two nodes cluster with security enabled. I have everything > running successful like namenode, datanode, resourcemanager, nodemanager, > jobhistoryserver etc. But datanode is unable to connect to namenode, as i > can see only one node on the web UI. checking logs of datanode gives > following warning: > > > > *WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting > to server: hadoop-master/**192.168.23.206:8020* > <http://192.168.23.206:8020> > > > > Rest of the things look fine. Please help me in this regard, what could be > the issue? > > > > > > > > > > >
dn.keytab
Description: Binary data
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>fs.permissions.umask-mode</name>
<value>000</value>
</property>
<!-- General HDFS security config -->
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<!-- NameNode security config -->
<property>
<name>dfs.https.address</name>
<value>192.168.23.206:8020</value>
</property>
<property>
<name>dfs.https.port</name>
<value>8020</value>
</property>
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/hadoop/conf/nn.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>nn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.namenode.kerberos.https.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<!-- Secondary NameNode security config -->
<property>
<name>dfs.secondary.http.address</name>
<value>192.168.23.206:50495</value>
</property>
<property>
<name>dfs.secondary.http.port</name>
<value>50495</value>
</property>
<property>
<name>dfs.secondary.namenode.keytab.file</name>
<value>/etc/hadoop/conf/nn.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.secondary.namenode.kerberos.principal</name>
<value>nn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>HTTP/_HOST@palatlyticsrealm</value>
</property>
<!-- DataNode security config -->
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1004</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1006</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/etc/hadoop/conf/nn.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.datanode.kerberos.https.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.namenode.acls.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
</configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>fs.permissions.umask-mode</name>
<value>000</value>
</property>
<!-- General HDFS security config -->
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<!-- NameNode security config -->
<property>
<name>dfs.https.address</name>
<value>192.168.23.206:8020</value>
</property>
<property>
<name>dfs.https.port</name>
<value>8020</value>
</property>
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>nn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.namenode.kerberos.https.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<!-- Secondary NameNode security config -->
<property>
<name>dfs.secondary.http.address</name>
<value>192.168.23.206:50495</value>
</property>
<property>
<name>dfs.secondary.http.port</name>
<value>50495</value>
</property>
<property>
<name>dfs.secondary.namenode.keytab.file</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.secondary.namenode.kerberos.principal</name>
<value>nn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>HTTP/_HOST@palatlyticsrealm</value>
</property>
<!-- DataNode security config -->
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1004</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1006</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>dn/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.datanode.kerberos.https.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@platalyticsrealm</value>
</property>
<property>
<name>dfs.namenode.acls.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
</configuration>
nn.keytab
Description: Binary data
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
