Reg : Hadoop Administration Material
Hi all I am changing my project role as Hadoop Adminstrator please share me useful material for admin role. Thanks Sajid
Re: Reg : Hadoop Administration Material
Hi sajid, I would like to let you know my notes on that, See https://github.com/youngwookim/awesome-hadoop#resources As always, Google and StackOverflow are your friends. HTH, Youngwoo On Fri, Jul 24, 2015 at 4:13 PM, sajid mohammed wrote: > Hi all > > I am changing my project role as Hadoop Adminstrator > > please share me useful material for admin role. > > > Thanks > Sajid > > >
Hdfs FSShell getmerge feature idea when using -nl
Hello hadoop users, I have an idea about a small feature for the getmerge tool. I recently was in the need of using the new line option -nl because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. I just wrote down 2 change one could try at the end. Do you guys consider this as a good improvement to the command line tools? Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied Or one would check the file size before. Please let me know If you would consider this useful and is worth a feature ticket in Jira. Thank you Jan
dfs.permissions.superusergroup not working
Hi, I have add two groups under dfs.permissions.superusergroup property as below dfs.cluster.administrators hdfs,hdpdadmngrp Restarted the HDFS after the change. But I am not able to run the below command hdfs dfsadmin -refreshNodes Error says I don't have privilege to run the above command My cluster is kerberized. I have principal for my user id. Hdfs user: uid=504(hdfs) gid=501(hadoop) groups=501(hadoop),503(hdfs),506(hdpdadmngrp) New admin user: uid=1007(gv07680) gid=506(hdpdadmngrp) groups=506(hdpdadmngrp) I am using hadoop-2.6.0 Please help me with this. Thanks, Venkat
Re: dfs.permissions.superusergroup not working
Is there a typo in your email, or did you set dfs.cluster.administrators instead of intending to set dfs.permissions.superusergroup? Also, are your id outputs from the NameNode machines? Cause by default the group lookups happen local to your NameNode machine. On Sat, Jul 25, 2015 at 1:31 AM Gangavarapu, Venkata < venkata.gangavar...@bcbsa.com> wrote: > Hi, > > > > I have add two groups under dfs.permissions.superusergroup property as > below > > > > dfs.cluster.administrators > >hdfs,hdpdadmngrp > > > > > > Restarted the HDFS after the change. > > > > But I am not able to run the below command > > hdfs dfsadmin –refreshNodes > > > > > > Error says I don’t have privilege to run the above command > > > > My cluster is kerberized. I have principal for my user id. > > > > Hdfs user: > > uid=504(hdfs) gid=501(hadoop) groups=501(hadoop),503(hdfs),506(hdpdadmngrp) > > > > New admin user: > > uid=1007(gv07680) gid=506(hdpdadmngrp) groups=506(hdpdadmngrp) > > > > I am using hadoop-2.6.0 > > > > Please help me with this. > > > > Thanks, > > Venkat >
Re: Reg : Hadoop Administration Material
Can refer my Github, very specific to Hadoop Adminstration. https://github.com/netxillon/hadoop On 24/07/15 1:59 pm, 김영우 wrote: Hi sajid, I would like to let you know my notes on that, See https://github.com/youngwookim/awesome-hadoop#resources As always, Google and StackOverflow are your friends. HTH, Youngwoo On Fri, Jul 24, 2015 at 4:13 PM, sajid mohammed mailto:sajid.had...@gmail.com>> wrote: Hi all I am changing my project role as Hadoop Adminstrator please share me useful material for admin role. Thanks Sajid -- -- Thanks and Regards Gurmukh Singh
