----- Original Message ----- From: kartheek muthyala <kartheek0...@gmail.com> Date: Tuesday, October 18, 2011 11:54 am Subject: Re: Does hadoop support append option? To: common-user@hadoop.apache.org
> I am just concerned about the use case of appends in Hadoop. I > know that > they have provided support for appends in hadoop. But how > frequently are the > files getting appended? . In normal case file block details will not be persisted in edit log before closing the file. As part of close only, this will happen. If NN restart happens before closing the file, we loose this data. Consider a case, we have a very big file and data also very important, in this case, we should have an option to persist the block details frequently into editlog file rite, inorder to avoid the dataloss in case of NN restarts. To do this, DFS exposed the API called sync. This will basically persist the editlog entries to disk. To reopen the stream back again we will use append api. In trunk, this support has been refactored cleanly and handled many corner cases. APIs also provided as hflush. There is this version concept too that is > maintained in the block report, according to my guess this version > number is > maintained to make sure that if a datanode gets disconnected once > and comes > back if it has a old copy of the data , then discard read requests > to this > data node. But if the files are not getting appended frequently > does the > version number remain the same?. Any typical use case can you guys > point to? > I am not sure, what is your exact question here. Can you please clarify more on this? > ~Kartheek > > On Mon, Oct 17, 2011 at 12:53 PM, Uma Maheswara Rao G 72686 < > mahesw...@huawei.com> wrote: > > > AFAIK, append option is there in 20Append branch. Mainly > supports sync. But > > there are some issues with that. > > > > Same has been merged to 20.205 branch and will be released soon (rc2 > > available). And also fixed many bugs in this branch. As per our > basic> testing it is pretty good as of now.Need to wait for > official release. > > > > Regards, > > Uma > > > > ----- Original Message ----- > > From: bourne1900 <bourne1...@yahoo.cn> > > Date: Monday, October 17, 2011 12:37 pm > > Subject: Does hadoop support append option? > > To: common-user <common-user@hadoop.apache.org> > > > > > I know that hadoop0.19.0 supports append option, but not stable. > > > Does the latest version support append option? Is it stable? > > > Thanks for help. > > > > > > > > > > > > > > > bourne > > > Regards, Uma
