RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks to all I got it working ☺
From: Vinayakumar B [mailto:vinayakumarb.apa...@gmail.com]
Sent: 02 November 2015 20:12
To: user@hadoop.apache.org
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
For simplicity You just can copy HADOOP_CONF_DIR from one of the cluster's
machine. And place it in class path of the client program.
Principal you are using to login is the client principal. It can be different
from server principal.
-Vinay
On Nov 2, 2015 22:37, "Vishwakarma, Chhaya"
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
Thanks Subroto
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
this is principal specified here
“UserGroupInformation.loginUserFromKeytab(String, String) “ or it is something
different ?
From: Subroto Sanyal [mailto:ssan...@datameer.com<mailto:ssan...@datameer.com>]
Sent: 02 November 2015 19:54
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
Few configuration you need to set:
hadoop.security.authentication=kerberos
hadoop.security.authorization=true
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
fs.defaultFS=hdfs://host:port
Further you need to use
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String,
String) as suggested in one of the trailing mail.
Cheers,
Subroto Sanyal
On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
Code is successfully authenticating to Kerberos but when I try to run any hdfs
command I get error as "Failed to specify server's kerberos principal name"
Can somebody please assist me on this?
Sent from my android device.
-Original Message-
From: andreina j <andrein...@huawei.com<mailto:andrein...@huawei.com>>
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
Please find below sample code .
System.setProperty("java.security.krb5.conf",
"D:\\data\\Desktop\\cluster-test\\krb5.conf");
// Login using keytab if have access to keytab. else
UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
" D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");
Note: Above 2 lines should be at the beginning in your application.
Regards
Andreina J
From: Vishwakarma, Chhaya
[mailto:chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks Niranjan It would be great if you can share a sample code if any?
From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
You can use the UserGroupInformation class from org.apache.hadoop.security
package.
Specifically following 2 methods of that class
UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
Regards,
Niranjan
On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using
JAVA code.
I have keytab file and username can someone please suggest how can I
autheticate to Kerberos using JAVA code?
Regards,
Chhaya
RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Code is successfully authenticating to Kerberos but when I try to run any hdfs
command I get error as "Failed to specify server's kerberos principal name"
Can somebody please assist me on this?
Sent from my android device.
-Original Message-
From: andreina j <andrein...@huawei.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
Please find below sample code .
System.setProperty("java.security.krb5.conf",
"D:\\data\\Desktop\\cluster-test\\krb5.conf");
// Login using keytab if have access to keytab. else
UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
"
D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");
Note: Above 2 lines should be at the beginning in your application.
Regards
Andreina J
From: Vishwakarma, Chhaya [mailto:chhaya.vishwaka...@thinkbiganalytics.com]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks Niranjan It would be great if you can share a sample code if any?
From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
You can use the UserGroupInformation class from org.apache.hadoop.security
package.
Specifically following 2 methods of that class
UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
Regards,
Niranjan
On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using
JAVA code.
I have keytab file and username can someone please suggest how can I
autheticate to Kerberos using JAVA code?
Regards,
Chhaya
RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks Subroto
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
this is principal specified here
“UserGroupInformation.loginUserFromKeytab(String, String) “ or it is something
different ?
From: Subroto Sanyal [mailto:ssan...@datameer.com]
Sent: 02 November 2015 19:54
To: user@hadoop.apache.org
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
Few configuration you need to set:
hadoop.security.authentication=kerberos
hadoop.security.authorization=true
dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>
fs.defaultFS=hdfs://host:port
Further you need to use
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(String,
String) as suggested in one of the trailing mail.
Cheers,
Subroto Sanyal
On Mon, Nov 2, 2015 at 3:13 PM, Vishwakarma, Chhaya
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
Code is successfully authenticating to Kerberos but when I try to run any hdfs
command I get error as "Failed to specify server's kerberos principal name"
Can somebody please assist me on this?
Sent from my android device.
-Original Message-
From: andreina j <andrein...@huawei.com<mailto:andrein...@huawei.com>>
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Sent: Mon, 02 Nov 2015 4:57 pm
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
Please find below sample code .
System.setProperty("java.security.krb5.conf",
"D:\\data\\Desktop\\cluster-test\\krb5.conf");
// Login using keytab if have access to keytab. else
UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com<mailto:hdfs/had...@hadoop.com>",
" D:\\data\\Desktop\\cluster-test\\conf\\hdfs.keytab");
Note: Above 2 lines should be at the beginning in your application.
Regards
Andreina J
From: Vishwakarma, Chhaya
[mailto:chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>]
Sent: 02 November 2015 PM 04:20
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks Niranjan It would be great if you can share a sample code if any?
From: Niranjan Subramanian [mailto:niran...@webaction.com]
Sent: 02 November 2015 16:18
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java
Hi Chhaya,
You can use the UserGroupInformation class from org.apache.hadoop.security
package.
Specifically following 2 methods of that class
UserGroupInformation.setConfiguration(hdfsConfiguration);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);
Regards,
Niranjan
On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya
<chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>>
wrote:
I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using
JAVA code.
I have keytab file and username can someone please suggest how can I
autheticate to Kerberos using JAVA code?
Regards,
Chhaya
Authenticating to Kerberos enabled Hadoop cluster using Java
I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using JAVA code. I have keytab file and username can someone please suggest how can I autheticate to Kerberos using JAVA code? Regards, Chhaya
RE: Authenticating to Kerberos enabled Hadoop cluster using Java
Thanks Niranjan It would be great if you can share a sample code if any? From: Niranjan Subramanian [mailto:niran...@webaction.com] Sent: 02 November 2015 16:18 To: user@hadoop.apache.org Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java Hi Chhaya, You can use the UserGroupInformation class from org.apache.hadoop.security package. Specifically following 2 methods of that class UserGroupInformation.setConfiguration(hdfsConfiguration); UserGroupInformation.loginUserFromKeytab(principal, keytabPath); Regards, Niranjan On 02-Nov-2015, at 4:15 pm, Vishwakarma, Chhaya <chhaya.vishwaka...@thinkbiganalytics.com<mailto:chhaya.vishwaka...@thinkbiganalytics.com>> wrote: I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using JAVA code. I have keytab file and username can someone please suggest how can I autheticate to Kerberos using JAVA code? Regards, Chhaya
Reading/Writing files to HDFS from Windwos server
I want to write files to HDFS from windows server. Hadoop cluster is on Linux. I tried researching everywhere I got a java code that can be run using "hadoop jar" Can somebody help me to understand how can I run HDFS file write java code from windows? What is required on Windows box? Even a proper link will do. Regards, Chhaya
Hdfs put VS webhdfs
Hi, I'm loading 28 GB file in hadoop hdfs using webhdfs and it takes ~25 mins to load. I tried loading same file using hdfs put and It took ~6 mins. Why there is so much difference in performance? What is recommended to use? Can somebody explain or direct me to some good link it will be really helpful. Below us the command I'm using curl -i --negotiate -u: -X PUT http://$hostname:$port/webhdfs/v1/$destination_file_location/$source_filename.temp?op=CREATEoverwrite=true; this will redirect to a datanode address which I use in next step to write the data Regards, Chhaya
