I'd like to second that. I think it would be good to have the database elevated to a first class use case for HDFS. Getting fsync() working properly is critical for HBase, Hypertable, or any database built on top of HDFS.
- Doug On Fri, Feb 27, 2009 at 9:08 AM, Jim Kellerman (POWERSET) < [email protected]> wrote: > I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible. > We are really hurting without it. > > --- > Jim Kellerman, Powerset (Live Search, Microsoft Corporation) > > > -----Original Message----- > > From: Nigel Daley [mailto:[email protected]] > > Sent: Thursday, February 26, 2009 9:13 PM > > To: [email protected] > > Subject: Re: Hadoop 0.20.0 > > > > Thanks Jim. > > > > Dhruba, can we move > > https://issues.apache.org/jira/browse/HADOOP-4379 > > to 0.21.0? > > > > Nige > > > > On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote: > > > > > With the availability of HADOOP-5332 I remove my objection. > > > > > >> -----Original Message----- > > >> From: Dhruba Borthakur [mailto:[email protected]] > > >> Sent: Wednesday, February 25, 2009 9:32 PM > > >> To: [email protected] > > >> Subject: Re: Hadoop 0.20.0 > > >> > > >> I posted a patch for HADOOP-5332. I am suggesting that this patch be > > >> applied > > >> into the 0.19, 0.20 and trunk. This patch switches off "append" by > > >> default, > > >> but it can be switched on by setting the config parameter > > >> dfs.support.append. This does not mean that "append" is bug free in > > >> the > > >> code, it just allows developers to continue testing with append > > >> functionality till the bugs are fixed. > > >> > > >> thanks, > > >> dhruba > > >> > > >> On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhema...@yahoo- > > >> inc.com>wrote: > > >> > > >>> +1 for HADOOP-5332. I am in the same position as Brian, as an > > >>> outside > > >>> observer. This will help us to move on Hadoop 0.20 which has a lot > > >>> of > > >> other > > >>> features as well that users are asking for. > > >>> > > >>> Thanks > > >>> hemanth > > >>> > > >>> > > >>> > > >>>> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote: > > >>>> > > >>>> > > >>>>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote: > > >>>>> > > >>>>> "Whipping out a patch" says nothing about its reliability. > > >>>>>> > > >>>>>> i would like some focus from the developer's community to > > >>>>>> properly > > >> fix > > >>>>>> this > > >>>>>> issue. I am willing to spend as much as time it takes ot get it > > >> fixed > > >>>>>> the > > >>>>>> right way, I but I would like even more constructive engagement > > >> from > > >>>>>> more > > >>>>>> people to get this one right. May I request you to see if you can > > >>>>>> volunteer > > >>>>>> to spend some time testing some of this code at scale ?(I have > > >> access to > > >>>>>> 10 > > >>>>>> machines only for testing). > > >>>>>> > > >>>>> > > >>>>> > > >>>> Dhruba, can you define "testing some of this code at scale"? Do you > > >> simply > > >>>> need access or folks who can run challenging jobs? Scaring up > > >>>> access > > >> to the > > >>>> cluster can be easy, but admin / user time isn't really available. > > >>>> > > >>>> Sorry, I can't commit any time/resources to this right now. Perhaps > > >> some > > >>>>> hbase folks can. In the meantime, can we make append > > >>>>> configurable in > > >> 0.19.2 > > >>>>> and 0.20.0? I filed > > >>>>> https://issues.apache.org/jira/browse/HADOOP-5332 > > >>>>> > > >>>> > > >>>> As an outside, irrelevant observer, I think this is a really good > > >>>> compromise. Helps out HBase but also would help prevent rushing. > > >>>> > > >>>> Brian > > >>>> > > >>>> > > >>>>> > > >>>>> Cheers, > > >>>>> Nige > > >>>>> > > >>>>> > > >>>>>> > > >>>>>> thanks > > >>>>>> dhruba > > >>>>>> > > >>>>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley < > [email protected] > > >>>>>> > > > >>>>>> wrote: > > >>>>>> > > >>>>>> > > >>>>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote: > > >>>>>>> > > >>>>>>> Hi Jim, > > >>>>>>> > > >>>>>>>> > > >>>>>>>> I can understand your problem. I can probably whip out a fix > > >>>>>>>> for > > >>>>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would > > >>>>>>>> be > > >> nice > > >>>>>>>> if > > >>>>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to > > >> discuss > > >>>>>>>> and > > >>>>>>>> review the patches/fixes. > > >>>>>>>> > > >>>>>>>> > > >>>>>>> "Whipping out a patch" doesn't give me any confidence that this > > >> feature > > >>>>>>> will be fixed properly. We're building a file system. Data > > >> reliability > > >>>>>>> and > > >>>>>>> accuracy are absolutely key. We know that this feature has been > > >> very > > >>>>>>> lightly tested. > > >>>>>>> > > >>>>>>> Nigel: wht is the proposed deadline for 0.20? > > >>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> March 6. > > >>>>>>> > > >>>>>>> Nige > > >>>>>>> > > >>>>>>> > > >>>>>>> thanks, > > >>>>>>> > > >>>>>>>> dhruba > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) < > > >>>>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>> --1 > > >>>>>>>> > > >>>>>>>>> > > >>>>>>>>> HBase really needs 4379. My testing to date indicates that it > > >> does > > >>>>>>>>> work > > >>>>>>>>> (although I have a bit more testing to do). > > >>>>>>>>> > > >>>>>>>>> I was ok with not putting it into 0.19.1 provided it was in > > >> 0.19.2 > > >>>>>>>>> and > > >>>>>>>>> 0.20.0. > > >>>>>>>>> > > >>>>>>>>> It's a big problem for us now and is hurting our ability to > > >>>>>>>>> keep > > >> our > > >>>>>>>>> community alive. (They will go to Cassandra or something > > >>>>>>>>> else to > > >>>>>>>>> ensure > > >>>>>>>>> reliability). > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -----Original Message----- > > >>>>>>>>> > > >>>>>>>>>> From: Nigel Daley [mailto:[email protected]] > > >>>>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM > > >>>>>>>>>> To: [email protected] > > >>>>>>>>>> Subject: Hadoop 0.20.0 > > >>>>>>>>>> > > >>>>>>>>>> Folks, > > >>>>>>>>>> > > >>>>>>>>>> Hadoop 0.19.1 is now available with the file append feature > > >>>>>>>>>> disabled. > > >>>>>>>>>> It's time to talk about a Hadoop 0.20.0 release. > > >>>>>>>>>> > > >>>>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. > > >>>>>>>>>> The > > >> last > > >>>>>>>>>> few blockers are now almost fixed (should be next week) > > >>>>>>>>>> except > > >> for > > >>>>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly > > >>>>>>>>>> implement > > >>>>>>>>>> file append. > > >>>>>>>>>> > > >>>>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and > > >> apply > > >>>>>>>>>> the > > >>>>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in > > >> place > > >>>>>>>>>> to > > >>>>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225). > > >>>>>>>>>> > > >>>>>>>>>> I will call a vote for 0.20.0 when blockers are fixed. > > >>>>>>>>>> > > >>>>>>>>>> Cheers, > > >>>>>>>>>> Nigel > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Folks, > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, > > >>>>>>>>>>> the > > >> 0.19 > > >>>>>>>>>>> branch has issues and a 0.19.1 release is needed. > > >>>>>>>>>>> > > >>>>>>>>>>> Quality issues in the changes made for the file append > > >>>>>>>>>>> feature > > >> have > > >>>>>>>>>>> prevented some from deploying Hadoop 0.19. One of these > > >> changes > > >>>>>>>>>>> (sync) has now been "fixed" by reducing its semantics in > > >> Hadoop > > >>>>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the > > >>>>>>>>>>> 0.18 > > >>>>>>>>>>> branch. > > >>>>>>>>>>> > > >>>>>>>>>>> I would like to propose that we apply this same "fix" to > > >>>>>>>>>>> sync > > >> in > > >>>>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full > > >>>>>>>>>>> semantics of > > >>>>>>>>>>> sync, I propose we also disable append (perhaps throw > > >>>>>>>>>>> UnsupportedOperationException from API?). Yes, this would > > >>>>>>>>>>> unfortunately be an incompatible change between 0.19.0 and > > >> 0.19.1. > > >>>>>>>>>>> We can then take the time needed to fix append properly in > > >> 0.21.0. > > >>>>>>>>>>> > > >>>>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are > > >> fixed. > > >>>>>>>>>>> > > >>>>>>>>>>> Nigel > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>> > > >>> > > > >
