On Fri, Mar 23, 2012 at 7:44 PM, Scott Carey <sc...@richrelevance.com> wrote: > > > On 3/22/12 10:25 AM, "Eli Collins" <e...@cloudera.com> wrote: > >>On Thu, Mar 22, 2012 at 1:26 AM, Konstantin Shvachko >><shv.had...@gmail.com> wrote: >>> Eli, >>> >>> I went over the entire discussion on the topic, and did not get it. Is >>> there a problem with append? We know it does not work in hadoop-1, >>> only flush() does. Is there anything wrong with the new append >>> (HDFS-265)? If so please file a bug. >>> I tested it in Hadoop-0.22 branch it works fine. >>> >>> I agree with people who were involved with the implementation of the >>> new append that the complexity is mainly in >>> 1. pipeline recovery >>> 2. consistent client reading while writing, and >>> 3. hflush() >>> Once it is done the append itself, which is reopening of previously >>> closed files for adding data, is not complex. >>> >> >>I agree that much of the complexity is in #1-3 above, which is why >>HDFS-265 is leveraged. >>The primary simplicity of not having append (and truncate) comes from >>not leveraging the invariant that finalized blocks are immutable, that >>blocks once written won't eg shrink in size (which we assume today). > > That invariant can co-exist with append via copy-on-write. The new state > and old state would co-exist until the old state was not needed, a file's > block map would have to use a persistent data structure. Copy on write > semantics with blocks in file systems is all the rage these days. Free > snapshots, atomic transactions for operations on multiple blocks, etc.
Hi Scott, If a client accesses a file, and then the client becomes unresponsive, how long should you wait before declaring the blocks he was looking at unused? No matter how long or how short a period you choose, someone will argue with it. And having to track this kind of state in the NameNode introduces a huge amount of complexity, not to mention extra memory consumption. Basically, we would have to track the ID of every block that any client looked at, at all times. Colin > >> >>> You mentioned it and I agree you indeed should be more involved with >>> your customer base. As for eBay, append was of the motivations to work >>> on stabilizing 0.22 branch. And there is a lot of use cases which >>> require append for our customers. >>> Some of them were mentioned in this discussion. >>> >> > >From what I've seen 0.22 isn't ready for production use. Aside from >>not supporting critical features like security, it doesn't have a >>size-able user-base behind it testing and fixing bugs, etc. All things >>I'd imagine an org like eBay would want. I've never gotten a request >>to support 0.22 from a customer. >> >>Thanks, >>Eli >
