RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Vishwakarma, Chhaya]

Thanks to all I got it working ☺

From: Vinayakumar B [mailto:vinayakumarb.apa...@gmail.com]
Sent: 02 November 2015 20:12
To: user@hadoop.apache.org
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java


For simplicity You just can copy HADOOP_CONF_DIR from one of the cluster's 
machine. And place it in class path of the client program.

Principal you are using to login is the client principal. It can be different 
from server principal.

-Vinay
On Nov 2, 2015 22:37, "Vishwakarma, Chhaya" 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:
Thanks Subroto

dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
 this is principal specified here 
“UserGroupInformation.loginUserFromKeytab(String, String) “ or it is something 
different ?

From: Subroto Sanyal [mailto:ssan...@datameer.com<mailto:ssan...@datameer.com>]
Sent: 02 November 2015 19:54
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

Few configuration you need to set:
hadoop.security.authentication=kerberos
hadoop.security.authorization=true
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
fs.defaultFS=hdfs://host:port
Further you need to use 
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String, 
String) as suggested in one of the trailing mail.

Cheers,
Subroto Sanyal

On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:
Code is successfully authenticating to Kerberos but when I try to run any hdfs 
command I get error as "Failed to specify server's kerberos principal name"

Can somebody please assist me on this?

Sent from my android device.


-Original Message-
From: andreina j <andrein...@huawei.com<mailto:andrein...@huawei.com>>
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" 
<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,

Please find below sample code .

  System.setProperty("java.security.krb5.conf", 
"D:\\data\\Desktop\\cluster-test\\krb5.conf");

// Login using keytab if have access to keytab. else

UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
   "  D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");

Note: Above 2 lines should be at the beginning  in your application.

Regards
Andreina J

From: Vishwakarma, Chhaya 
[mailto:chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Niranjan It would be great if you can share a sample code if any?

From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

You can use the UserGroupInformation class from org.apache.hadoop.security 
package.

Specifically following 2 methods of that class

UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:


I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Vishwakarma, Chhaya]

Code is successfully authenticating to Kerberos but when I try to run any hdfs 
command I get error as "Failed to specify server's kerberos principal name"

Can somebody please assist me on this?

Sent from my android device.

-Original Message-
From: andreina j <andrein...@huawei.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

Please find below sample code .

  System.setProperty("java.security.krb5.conf", 
"D:\\data\\Desktop\\cluster-test\\krb5.conf");

// Login using keytab if have access to keytab. else

UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
   "  
D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");

Note: Above 2 lines should be at the beginning  in your application.

Regards
Andreina J

From: Vishwakarma, Chhaya [mailto:chhaya.vishwaka...@thinkbiganalytics.com]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Niranjan It would be great if you can share a sample code if any?

From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

You can use the UserGroupInformation class from org.apache.hadoop.security 
package.

Specifically following 2 methods of that class

UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:


I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya

Re: Authenticating to Kerberos enabled Hadoop cluster using Java

[Subroto Sanyal]

Hi Chhaya,

Few configuration you need to set:

hadoop.security.authentication=kerberos

hadoop.security.authorization=true

dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com

fs.defaultFS=hdfs://host:port

Further you need to
use org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String,
String) as suggested in one of the trailing mail.


Cheers,

Subroto Sanyal

On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya <
chhaya.vishwaka...@thinkbiganalytics.com> wrote:

> Code is successfully authenticating to Kerberos but when I try to run any
> hdfs command I get error as "Failed to specify server's kerberos principal
> name"
>
> Can somebody please assist me on this?
>
> Sent from my android device.
>
>
> -Original Message-
> From: andreina j <andrein...@huawei.com>
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Sent: Mon, 02 Nov 2015 4:57 pm
> Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
>
> Hi Chhaya,
>
>
>
> Please find below sample code .
>
>
>
>   System.setProperty("java.security.krb5.conf", "D:
> \\data\\Desktop\\cluster-test\\krb5.conf");
>
> // Login using keytab if have access to keytab. else
> UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com",
>"  D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");
>
>
>
> Note: Above 2 lines should be at the beginning  in your application.
>
>
>
> Regards
>
> Andreina J
>
>
>
> *From:* Vishwakarma, Chhaya [mailto:
> chhaya.vishwaka...@thinkbiganalytics.com]
> *Sent:* 02 November 2015 PM 04:20
> *To:* user@hadoop.apache.org
> *Subject:* RE: Authenticating to Kerberos enabled Hadoop cluster using
> Java
>
>
>
> Thanks Niranjan It would be great if you can share a sample code if any?
>
>
>
> *From:* Niranjan Subramanian [mailto:niran...@webaction.com
> <niran...@webaction.com>]
> *Sent:* 02 November 2015 16:18
> *To:* user@hadoop.apache.org
> *Subject:* Re: Authenticating to Kerberos enabled Hadoop cluster using
> Java
>
>
>
> Hi Chhaya,
>
>
>
> You can use the UserGroupInformation class from org.apache.hadoop.security
> package.
>
>
>
> Specifically following 2 methods of that class
>
>
>
> UserGroupInformation.setConfiguration(hdfsConfiguration);
>
> UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
>
>
>
> Regards,
>
> Niranjan
>
>
>
> On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya <
> chhaya.vishwaka...@thinkbiganalytics.com> wrote:
>
>
>
> I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations
> using JAVA code.
>
> I have keytab file and username can someone please suggest how can I
> autheticate to Kerberos using JAVA code?
>
> Regards,
>
> Chhaya
>
>
>

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Vishwakarma, Chhaya]

Thanks Subroto

dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
 this is principal specified here 
“UserGroupInformation.loginUserFromKeytab(String, String) “ or it is something 
different ?

From: Subroto Sanyal [mailto:ssan...@datameer.com]
Sent: 02 November 2015 19:54
To: user@hadoop.apache.org
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

Few configuration you need to set:
hadoop.security.authentication=kerberos
hadoop.security.authorization=true
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
fs.defaultFS=hdfs://host:port
Further you need to use 
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String, 
String) as suggested in one of the trailing mail.

Cheers,
Subroto Sanyal

On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:
Code is successfully authenticating to Kerberos but when I try to run any hdfs 
command I get error as "Failed to specify server's kerberos principal name"

Can somebody please assist me on this?

Sent from my android device.


-Original Message-
From: andreina j <andrein...@huawei.com<mailto:andrein...@huawei.com>>
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" 
<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,

Please find below sample code .

  System.setProperty("java.security.krb5.conf", 
"D:\\data\\Desktop\\cluster-test\\krb5.conf");

// Login using keytab if have access to keytab. else

UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
   "  D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");

Note: Above 2 lines should be at the beginning  in your application.

Regards
Andreina J

From: Vishwakarma, Chhaya 
[mailto:chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Niranjan It would be great if you can share a sample code if any?

From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

You can use the UserGroupInformation class from org.apache.hadoop.security 
package.

Specifically following 2 methods of that class

UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:


I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Vinayakumar B]

For simplicity You just can copy HADOOP_CONF_DIR from one of the cluster's
machine. And place it in class path of the client program.

Principal you are using to login is the client principal. It can be
different from server principal.

-Vinay
On Nov 2, 2015 22:37, "Vishwakarma, Chhaya" <
chhaya.vishwaka...@thinkbiganalytics.com> wrote:

> Thanks Subroto
>
>
>
> dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com this is principal
> specified here “UserGroupInformation.loginUserFromKeytab(String, String) “
> or it is something different ?
>
>
>
> *From:* Subroto Sanyal [mailto:ssan...@datameer.com]
> *Sent:* 02 November 2015 19:54
> *To:* user@hadoop.apache.org
> *Subject:* Re: Authenticating to Kerberos enabled Hadoop cluster using
> Java
>
>
>
> Hi Chhaya,
>
>
>
> Few configuration you need to set:
>
> hadoop.security.authentication=kerberos
>
> hadoop.security.authorization=true
>
> dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com
>
> fs.defaultFS=hdfs://host:port
>
> Further you need to
> use 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String,
> String) as suggested in one of the trailing mail.
>
>
>
> Cheers,
>
> Subroto Sanyal
>
>
>
> On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya <
> chhaya.vishwaka...@thinkbiganalytics.com> wrote:
>
> Code is successfully authenticating to Kerberos but when I try to run any
> hdfs command I get error as "Failed to specify server's kerberos principal
> name"
>
> Can somebody please assist me on this?
>
> Sent from my android device.
>
>
>
> -Original Message-
> From: andreina j <andrein...@huawei.com>
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Sent: Mon, 02 Nov 2015 4:57 pm
> Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
>
> Hi Chhaya,
>
>
>
> Please find below sample code .
>
>
>
>   System.setProperty("java.security.krb5.conf",
> "D:\\data\\Desktop\\cluster-test\\krb5.conf");
>
> // Login using keytab if have access to keytab. else
> UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com",
>"  D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");
>
>
>
> Note: Above 2 lines should be at the beginning  in your application.
>
>
>
> Regards
>
> Andreina J
>
>
>
> *From:* Vishwakarma, Chhaya [mailto:
> chhaya.vishwaka...@thinkbiganalytics.com]
> *Sent:* 02 November 2015 PM 04:20
> *To:* user@hadoop.apache.org
> *Subject:* RE: Authenticating to Kerberos enabled Hadoop cluster using
> Java
>
>
>
> Thanks Niranjan It would be great if you can share a sample code if any?
>
>
>
> *From:* Niranjan Subramanian [mailto:niran...@webaction.com
> <niran...@webaction.com>]
> *Sent:* 02 November 2015 16:18
> *To:* user@hadoop.apache.org
> *Subject:* Re: Authenticating to Kerberos enabled Hadoop cluster using
> Java
>
>
>
> Hi Chhaya,
>
>
>
> You can use the UserGroupInformation class from org.apache.hadoop.security
> package.
>
>
>
> Specifically following 2 methods of that class
>
>
>
> UserGroupInformation.setConfiguration(hdfsConfiguration);
>
> UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
>
>
>
> Regards,
>
> Niranjan
>
>
>
> On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya <
> chhaya.vishwaka...@thinkbiganalytics.com> wrote:
>
>
>
> I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations
> using JAVA code.
>
> I have keytab file and username can someone please suggest how can I
> autheticate to Kerberos using JAVA code?
>
> Regards,
>
> Chhaya
>
>
>
>
>

Re: Authenticating to Kerberos enabled Hadoop cluster using Java

[Niranjan Subramanian]

Hi Chhaya, 

You can use the UserGroupInformation class from org.apache.hadoop.security 
package. 

Specifically following 2 methods of that class 

UserGroupInformation.setConfiguration(hdfsConfiguration); 
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
 wrote:

> I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations 
> using JAVA code.
> 
> I have keytab file and username can someone please suggest how can I 
> autheticate to Kerberos using JAVA code?
> 
> Regards,
> Chhaya

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Daniel Schulz]

Hi Chhaya,
Thank you for asking straight away. At my latest project (HDP with Kerberos) we 
used authenticated users; as they submitted Java jobs MR or Spark reached 
through their user name and looked for a valid Kerberos Principal. Upon file 
access, HDFS does so. So your user needs to have a valid Kerberos Ticket from 
`kinit` or a keytab file.
Of course: you could issue `kinit` via JNI; but this looks rather like a hack. 
If a service needs credentials, I'd rather use Keystores for that. On Hadoop 
with Kerberos, your user's Kerberos Principal is the default way to go.
Alternatively, you may use this parameter chain before start up:

SET KINIT=c:\Program Files\Java\jdk1.6.0_20\bin\kinit
SET KEYTAB=d:\webapps\app1\WEB-INF\serverhostname.keytab
SET SPN=HTTP/app1.intranet.company@realm002.company.com
SET KRB5INI=d:\webapps\app1\WEB-INF\krb5.ini
"%KINIT%" -k -t "%KEYTAB%" %SPN% -J-Dsun.security.krb5.debug=true 
"-J-Djava.security.krb5.conf=%KRB5INI%"source: 
michael-behrendt.net/blog/2011/01/kerberos-keytab-uberprufen-mit-java-boardmitteln
 (German)
Kind regards, Daniel.

From: Chhaya.Vishwakarma@Thinkbiganalytics.comTo: user@hadoop.apache.org
Subject: Authenticating to Kerberos enabled Hadoop cluster using Java
Date: Mon, 2 Nov 2015 10:45:37 +










I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya
  

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[Vishwakarma, Chhaya]

Thanks Niranjan It would be great if you can share a sample code if any?

From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

You can use the UserGroupInformation class from org.apache.hadoop.security 
package.

Specifically following 2 methods of that class

UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:



I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

[andreina j]

Hi Chhaya,

Please find below sample code .

  System.setProperty("java.security.krb5.conf", 
"D:\\data\\Desktop\\cluster-test\\krb5.conf");

// Login using keytab if have access to keytab. else

UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
   "  
D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");

Note: Above 2 lines should be at the beginning  in your application.

Regards
Andreina J

From: Vishwakarma, Chhaya [mailto:chhaya.vishwaka...@thinkbiganalytics.com]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Niranjan It would be great if you can share a sample code if any?

From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya,

You can use the UserGroupInformation class from org.apache.hadoop.security 
package.

Specifically following 2 methods of that class

UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

Regards,
Niranjan

On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya 
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
 wrote:


I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using 
JAVA code.

I have keytab file and username can someone please suggest how can I 
autheticate to Kerberos using JAVA code?
Regards,
Chhaya