unsubscribe

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Code is successfully authenticating to Kerberos but when I try to run any hdfs command I get error as "Failed to specify server's kerberos principal name" Can somebody please assist me on this? Sent from my android device. -Original Message- From: andreina j To:

Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya, Few configuration you need to set: hadoop.security.authentication=kerberos hadoop.security.authorization=true dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com fs.defaultFS=hdfs://host:port Further you need to use

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Subroto dfs.namenode.kerberos.principal=hdfs/had...@hadoop.com this is principal specified here “UserGroupInformation.loginUserFromKeytab(String, String) “ or it is something different ? From: Subroto Sanyal [mailto:ssan...@datameer.com] Sent: 02 November

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

For simplicity You just can copy HADOOP_CONF_DIR from one of the cluster's machine. And place it in class path of the client program. Principal you are using to login is the client principal. It can be different from server principal. -Vinay On Nov 2, 2015 22:37, "Vishwakarma, Chhaya" <

[no subject]

Unsubscribe

Re: Exception while appending to an existing file in HDFS

Hey Abhishek, I tried both tweaks, still I ended up with same exception. Maybe I should try upgrading to 2.6.0 and see? Regards, Niranjan On 01-Nov-2015, at 3:49 pm, ABHISHEK SINGH <23singhabhis...@gmail.com> wrote: > ("dfs.datanode.socket.write.timeout", "2");

Re: Exception while appending to an existing file in HDFS

Hi Niranjan, Glad to hear your issue was resolved. Actually, upgrading was a wise decision. Thanks, Abhishek On Monday 02 November 2015 03:59 PM, Niranjan Subramanian wrote: Hi Abhishek, Thanks for your pointer on letting me know there is an issue from 2.2.0 to 2.5.1, I just upgraded to

Re: unsubscribe

Unsubscribe. On Mon, Nov 2, 2015 at 2:47 PM, Joao Alves wrote: > > -- ==Thanks Shatrughan Thakur +44(0)7545924982

How to set user in ContainerLaunchContext?

Hi, i have an ApplicationMaster which accepts requests and launches spark jobs. In request i have a field "username" - the user i want to laucnh a job from. How can i set a user which will be run conmmand on a container? Currently they all running as yarn user even though AM is running as a

Max Parallel task executors

Hi Team, I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31 parallel tasks are executed whereas my machines have more CPU available but still YARN/AM does not create more task. Is there any configuration which I can change to start more MAP/REDUCER task in parallel? Each

Re: Exception while appending to an existing file in HDFS

Hi Abhishek, Thanks for your pointer on letting me know there is an issue from 2.2.0 to 2.5.1, I just upgraded to 2.7.1 and without adding any of these properties, it is working fine. Regards, Niranjan On 02-Nov-2015, at 1:38 pm, Niranjan Subramanian wrote: > Hey

Authenticating to Kerberos enabled Hadoop cluster using Java

I have Kerberos enabled Hadoop cluster, I need to perform HDFS operations using JAVA code. I have keytab file and username can someone please suggest how can I autheticate to Kerberos using JAVA code? Regards, Chhaya

Re: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya, You can use the UserGroupInformation class from org.apache.hadoop.security package. Specifically following 2 methods of that class UserGroupInformation.setConfiguration(hdfsConfiguration); UserGroupInformation.loginUserFromKeytab(principal, keytabPath); Regards, Niranjan On

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya, Thank you for asking straight away. At my latest project (HDP with Kerberos) we used authenticated users; as they submitted Java jobs MR or Spark reached through their user name and looked for a valid Kerberos Principal. Upon file access, HDFS does so. So your user needs to have a

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Thanks Niranjan It would be great if you can share a sample code if any? From: Niranjan Subramanian [mailto:niran...@webaction.com] Sent: 02 November 2015 16:18 To: user@hadoop.apache.org Subject: Re: Authenticating to Kerberos enabled Hadoop cluster using Java Hi Chhaya, You can use the

RE: Authenticating to Kerberos enabled Hadoop cluster using Java

Hi Chhaya, Please find below sample code . System.setProperty("java.security.krb5.conf", "D:\\data\\Desktop\\cluster-test\\krb5.conf"); // Login using keytab if have access to keytab. else UserGroupInformation.loginUserFromKeytab("hdfs/had...@hadoop.com",

RE: Utility to push data into HDFS

Hi Shashi Vishwakarma , You can follow below steps to perform HDFS operation using java code on a secure cluster 1. Copy krb5.conf, hdfs.keytab and conf directory from installed cluster 2. Create a maven project with dependeny hadoop-client org.apache.hadoop

RE: Utility to push data into HDFS

Hi Shashi, Did you copy conf directory (ex: /etc/hadoop by default) from any of the cluster machine’s Hadoop installation as mentioned in #1 in Andreina’s reply below? I hope, if cluster is running successfully with Kerberos enabled, it should have a configuration

Re:

To unsubscribe please send a mail to user-unsubscr...@hadoop.apache.org On Mon, Nov 2, 2015 at 6:32 PM, Joe Doherty wrote: > Unsubscribe >

Yarn related queries

Do people only answers HDFS related queries on this mailing list? I was wondering where to ask my YARN related queries. On the mailing list page (i.e. https://hadoop.apache.org/mailing_lists.html) this is the only mail id provided for users otherwise rest are for development, issues and releases

Hdfs service failed to start

Hi Folks, My first run fails while installing CDH-5 (parcels) using cloudera manager. Below is the error message . Service did not start successfully; not all of the required roles started: Service has only 0 NameNode roles running instead of minimum required 1. Any help?

Decommission datanode

Hi, I don't have much data, but it took around 40 minutes to decommission. How long will it take to decommission a datanode? Is there any way to optimize the process? Thanks.