Best option for mounting HDFS

Hi all, I was looking at various options available to mount HDFS on unix boxes. I found following option on wiki page. * contrib/fuse-dfs is built on fuse, some C glue, libhdfs and the hadoop-dev.jar * fuse-j-hdfs is built on fuse, fuse for java, and the hadoop-dev.jar * hdfs-fuse - a google

Hadoop Security

Hi , I wanted to know about the security in Hadoop. I have read few articles but not very sure about this so I wanted to discuss this topic in forum. As we know that Hadoop provide its security using Filesystem permissions like chown, chmod etc. 1. Is Kerberos or any security algo is

Re: Hadoop Security

HI, 1. yes: https://ccp.cloudera.com/display/CDHDOC/Configuring+Hadoop+Security+in+CDH3 http://hortonworks.com/the-role-of-delegation-tokens-in-apache-hadoop-security/ 2. yes http://mapredit.blogspot.com/2011/10/secure-your-hadoop-cluster-part-i.html 3. at the moment hdfs has no encryption

RE: Hadoop Security

Thanks Alexander for this info. Currently I am using Apache Hadoop version 0.20.2 and not cloudera’s Hadoop version. I read that Apache Hadoop 0.20.205 supports Security. Any thoughts on that. Since currently I am using Apache Hadoop and quite familiar with it so I would like to use it some

Re: Hadoop Security

Hi Stuti, here a overview about the classes: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/security/package-summary.html hortonworks has written a really good tutorial: http://hortonworks.com/fine-tune-your-apache-hadoop-security-settings/ If I right hadoop supports

Re: Hadoop Security

Apache Hadoop 0.20.2 did not have security features in it. You'd need 0.20.203 at least, if not one of the current CDH3/0.20.205 (Both of which also carry 0.20-append along with 0.20-security). On Mon, Nov 28, 2011 at 3:56 PM, Stuti Awasthi stutiawas...@hcl.com wrote: Thanks Alexander for this

Re: What's the equivalent of tail -f

Thanks Harsh, that's what i was looking for. - inder On Mon, Nov 28, 2011 at 4:48 PM, Harsh J ha...@cloudera.com wrote: If you ask just from a shell POV, then yes, you may use hadoop fs -tail -f file. On Mon, Nov 28, 2011 at 4:17 PM, Inder Pall inder.p...@gmail.com wrote: Is there

hdfs behavior

People, i am seeing the following - 1. writing to a large file on HDFS 2. tail -f on the same file shows data is streaming. 3. hadoop dfs -ls on the same file shows size as 0. Has anyone experienced this? -- Inder

RE: hdfs behavior

I think, you might not completed even single block write completely. Length will be updated in NN after completing the block. Currently partial block lengths will not be included in length calculation. Regards, Uma From: Inder Pall [inder.p...@gmail.com]

Re: Hadoop Security

Hi, 3. Is any kind of encryption is handled in hadoop at the time of storing the files in HDFS. You could define a compression codec that does the encryption. Check the below thread for more details. http://www.mail-archive.com/common-user@hadoop.apache.org/msg06229.html Thanks, Praveen On

RE: how to find data nodes on which a file is distributed to?

From Java API, FileSystem#getFileBlockLocations should give you the blocklocations. Regards, Uma From: Praveen Sripati [praveensrip...@gmail.com] Sent: Monday, November 28, 2011 10:01 PM To: hdfs-user@hadoop.apache.org Subject: Re: how to find data nodes on

Error in namenode log file

Hello, Although the namenode is running properly I can see the below shown warning in the log file.Could anyone tell me its meaning 2011-11-29 01:42:30,582 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user webuser

RE: Symbolic Links in HDFS

HI all, Any thoughts on this ?? -Original Message- From: Stuti Awasthi Sent: Monday, November 28, 2011 2:45 PM To: hdfs-user@hadoop.apache.org Subject: Symbolic Links in HDFS Hi Friends, I wanted to use symbolic links in HDFS for my application. I found

Re: Best option for mounting HDFS

Hi Stuti, I have good experience with FUSE (http://fuse.sourceforge.net/), but thats not a recommandation. - alex On Tue, Nov 29, 2011 at 5:38 AM, Stuti Awasthi stutiawas...@hcl.com wrote: Hi Friends , Any thoughts on this ?? -Original Message- From: Stuti Awasthi Sent: Monday,