RE: Is FileSystem thread-safe?

2013-05-17 Thread John Lilley
...@gmail.com] Subject: Re: Is FileSystem thread-safe? FileSystem is an abstract class, what concrete class are you using (DistributedFileSystem, etc) ? Good point. I am calling FileSystem.get(URI uri, Configuration conf) with an URI like hdfs://server:port/... on a remote server, so I assume

Re: Is FileSystem thread-safe?

2013-05-17 Thread Vinod Kumar Vavilapalli
: Re: Is FileSystem thread-safe? Hi John, DistributedFileSystem is intended to be thread-safe, true to its name. Metadata operations are handled by the NameNode server which synchronizes concurrent client requests via locks (you can look at the FSNameSystem class). Some discussion

RE: Is FileSystem thread-safe?

2013-05-17 Thread John Lilley
...@hortonworks.com] Sent: Friday, May 17, 2013 11:14 AM To: user@hadoop.apache.org Subject: Re: Is FileSystem thread-safe? As of today, there is no atomic append, so no, what you say isn't possible. FWIU, it is one appender at a time - achieved through a lease per file, and multiple concurrent leases

Re: Is FileSystem thread-safe?

2013-05-17 Thread Vinod Kumar Vavilapalli
, users opt to output a folder full of file parts into HDFS and then read them directly or unify them later. John From: Vinod Kumar Vavilapalli [mailto:vino...@hortonworks.com] Sent: Friday, May 17, 2013 11:14 AM To: user@hadoop.apache.org Subject: Re: Is FileSystem thread-safe

Re: Is FileSystem thread-safe?

2013-04-02 Thread Matthew Farrellee
...@redpoint.net mailto:john.lil...@redpoint.net wrote: *From:*Ted Yu [mailto:yuzhih...@gmail.com mailto:yuzhih...@gmail.com] *Subject:* Re: Is FileSystem thread-safe? FileSystem is an abstract class, what concrete class are you using (DistributedFileSystem

Re: Is FileSystem thread-safe?

2013-04-01 Thread Arpit Agarwal
through ClientProtocol. An hdfs committer would be able to give you affirmative answer. On Sun, Mar 31, 2013 at 11:27 AM, John Lilley john.lil...@redpoint.netwrote: *From:* Ted Yu [mailto:yuzhih...@gmail.com] *Subject:* Re: Is FileSystem thread-safe? FileSystem is an abstract class

Re: Is FileSystem thread-safe?

2013-03-31 Thread Ted Yu
FileSystem is an abstract class, what concrete class are you using (DistributedFileSystem, etc) ? For FileSystem, I find the following for create() method: * but the implementation is thread-safe. The other option is to change the * value of umask in configuration to be 0, but it is not

RE: Is FileSystem thread-safe?

2013-03-31 Thread John Lilley
From: Ted Yu [mailto:yuzhih...@gmail.com] Subject: Re: Is FileSystem thread-safe? FileSystem is an abstract class, what concrete class are you using (DistributedFileSystem, etc) ? Good point. I am calling FileSystem.get(URI uri, Configuration conf) with an URI like hdfs://server:port