Harsh, idea is to call sync for a configured batch. Still under implementation as other parts of the system's aren't complete.
recovery/resume-from-errors-at-DN code around general tail-like >>This sounds promising, can you please shed some more light on this. - inder On Tue, Apr 10, 2012 at 1:07 AM, Harsh J <ha...@cloudera.com> wrote: > Your approach looks fine to me. I'd throw in some > recovery/resume-from-errors-at-DN code around general tail-like > consumption but I think you may have already done that :) > > But just for my curiosity - do you call sync for every record/unit or > batch it by a few, for your problem? > > On Mon, Apr 9, 2012 at 10:34 PM, Inder Pall <inder.p...@gmail.com> wrote: > > Yes makes sense. My use-case is more like a producer/consumer and > consumer > > trying to stream data as it arrives. > > Has anyone hit this before and if so resolved it in a better way. > > > > Apologies, if i am digressing from the subject of this thread however > seems > > to land in the bucket of append support in HDFS. > > > > - Inder > > > > > > On Mon, Apr 9, 2012 at 6:27 PM, Harsh J <ha...@cloudera.com> wrote: > >> > >> Inder, > >> > >> Yes, that is a requirement for readers of sync-ing data. The new meta > >> entries can only be read by new readers. The read code would end up > >> being exactly like the implementation for method "fs -tail" at > >> > >> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/fs/FsShell.java?view=markup > >> (Line 1101) > >> > >> HBase does not read the WAL (HLog) continuously/vigorously as it > >> syncs, by the way. It only reads the them when a specific request is > >> made (for splitting, replaying and debug-printing). > >> > >> On Mon, Apr 9, 2012 at 6:05 PM, Inder Pall <inder.p...@gmail.com> > wrote: > >> > Based on what i have tried, after a sync you need to open a new > Reader. > >> > Please correct if that's not the write semantics. > >> > > >> > Thanks, > >> > - Inder > >> > > >> > > >> > On Mon, Apr 9, 2012 at 4:23 PM, Harsh J <ha...@cloudera.com> wrote: > >> >> > >> >> I'd also like to note that there are some unresolved issues with the > >> >> append version in the 1.x (stable) line. > >> >> > >> >> Note that HBase's use of the 0.20-append branch features are limited > >> >> to using "sync" calls alone (Described in p68 "Coherency Model", > >> >> Chapter 3 (The Hadoop Distributed File System) in Hadoop: The > >> >> Definitive Guide, 2nd Edition (O'Reilly)). Not the file-reopening > >> >> "append" calls. The latter is what is still with issues in the 1.x > >> >> releases today. Using the former is alright if its done in the way > >> >> similar to HBase's WAL (HLog) (or for similar needs). > >> >> > >> >> On Mon, Apr 9, 2012 at 3:45 PM, Ioan Eugen Stan < > stan.ieu...@gmail.com> > >> >> wrote: > >> >> > 2012/4/7 Florin P <florinp...@yahoo.com>: > >> >> >> Hello! > >> >> >> Just google it for supporting of append into HDFS files and the > >> >> >> result: > >> >> >> I'm puzzled. Can someone say: YES you can append in TextFile or > >> >> >> SequenceFile > >> >> >> or whatever format. If yes, in which version this feature is > >> >> >> supported > >> >> >> ? > >> >> >> Also where can I find a good example of using the API? I know that > >> >> >> is a > >> >> >> long > >> >> >> debate about this subject, but really it is challenge to find on > the > >> >> >> google > >> >> >> the current status of this feature. > >> >> >> I look forward for a trust source answer. > >> >> >> Thank you, > >> >> >> Regards, > >> >> >> Florin > >> >> > > >> >> > Hi Florian, > >> >> > > >> >> > HDFS supports append in Hadoop 1.0.x branch and also 0.22 (a.k.a > >> >> > hadoop 2.x branch) and 0.23 (a.k.a hadoop 3.x branch). > >> >> > > >> >> > [1] http://hbase.apache.org/book/hadoop.html > >> >> > [2] http://hbase.apache.org/book/hadoop.html -- search for append > in > >> >> > release notes > >> >> > > >> >> > Cheers, > >> >> > -- > >> >> > Ioan Eugen Stan > >> >> > http://ieugen.blogspot.com/ > >> >> > >> >> > >> >> > >> >> -- > >> >> Harsh J > >> > > >> > > >> > > >> > > >> > -- > >> > Thanks, > >> > - Inder > >> > Tech Platforms @Inmobi > >> > Linkedin - http://goo.gl/eR4Ub > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > > > -- > > Thanks, > > - Inder > > Tech Platforms @Inmobi > > Linkedin - http://goo.gl/eR4Ub > > > > -- > Harsh J > -- Thanks, - Inder Tech Platforms @Inmobi Linkedin - http://goo.gl/eR4Ub
