----- Original Message ----- From: kartheek muthyala <kartheek0...@gmail.com> Date: Tuesday, October 18, 2011 1:31 pm Subject: Re: Does hadoop support append option? To: common-user@hadoop.apache.org
> Thanks Uma for the clarification of the append functionality. > > My second question is about the version number concept used in the > blockmap. Why does it maintain this version number? sorry Karthik, As i know, there is no version number in blocks map. Are you talking about generationTimeStamp or something? can you paste the snippet where you have seen that version number, so, that i can get your question clearly. > > ~Kartheek > > On Tue, Oct 18, 2011 at 12:14 PM, Uma Maheswara Rao G 72686 < > mahesw...@huawei.com> wrote: > > > ----- Original Message ----- > > From: kartheek muthyala <kartheek0...@gmail.com> > > Date: Tuesday, October 18, 2011 11:54 am > > Subject: Re: Does hadoop support append option? > > To: common-user@hadoop.apache.org > > > > > I am just concerned about the use case of appends in Hadoop. I > > > know that > > > they have provided support for appends in hadoop. But how > > > frequently are the > > > files getting appended? . > > In normal case file block details will not be persisted in edit > log before > > closing the file. As part of close only, this will happen. If NN > restart> happens before closing the file, we loose this data. > > > > Consider a case, we have a very big file and data also very > important, in > > this case, we should have an option to persist the block details > frequently> into editlog file rite, inorder to avoid the dataloss > in case of NN > > restarts. To do this, DFS exposed the API called sync. This will > basically> persist the editlog entries to disk. To reopen the > stream back again we will > > use append api. > > > > In trunk, this support has been refactored cleanly and handled > many corner > > cases. APIs also provided as hflush. > > > > There is this version concept too that is > > > maintained in the block report, according to my guess this version > > > number is > > > maintained to make sure that if a datanode gets disconnected once > > > and comes > > > back if it has a old copy of the data , then discard read requests > > > to this > > > data node. But if the files are not getting appended frequently > > > does the > > > version number remain the same?. Any typical use case can you guys > > > point to? > > > > > I am not sure, what is your exact question here. Can you please > clarify> more on this? > > > > > ~Kartheek > > > > > > On Mon, Oct 17, 2011 at 12:53 PM, Uma Maheswara Rao G 72686 < > > > mahesw...@huawei.com> wrote: > > > > > > > AFAIK, append option is there in 20Append branch. Mainly > > > supports sync. But > > > > there are some issues with that. > > > > > > > > Same has been merged to 20.205 branch and will be released > soon (rc2 > > > > available). And also fixed many bugs in this branch. As per our > > > basic> testing it is pretty good as of now.Need to wait for > > > official release. > > > > > > > > Regards, > > > > Uma > > > > > > > > ----- Original Message ----- > > > > From: bourne1900 <bourne1...@yahoo.cn> > > > > Date: Monday, October 17, 2011 12:37 pm > > > > Subject: Does hadoop support append option? > > > > To: common-user <common-user@hadoop.apache.org> > > > > > > > > > I know that hadoop0.19.0 supports append option, but not > stable.> > > > Does the latest version support append option? Is > it stable? > > > > > Thanks for help. > > > > > > > > > > > > > > > > > > > > > > > > > bourne > > > > > > > > > > > Regards, > > Uma > > >
