Hey George: James Kennedy is working on getting transactional hbase working w/ hbase TRUNK. Watch HBASE-2641 for the drop of changes needed in core to make it so his github THBase can use HBase core.
St.Ack On Mon, Sep 20, 2010 at 5:43 PM, Ryan Rawson <ryano...@gmail.com> wrote: > hi, > > sorry i dont. i think the current transactional/indexed person is > working on bringing it up to 0.89, perhaps they would enjoy your help > in testing or porting the code? > > I'll poke a few people into replying. > > -ryan > > On Mon, Sep 20, 2010 at 5:19 PM, George P. Stathis <gstat...@traackr.com> > wrote: >> On Mon, Sep 20, 2010 at 4:55 PM, Ryan Rawson <ryano...@gmail.com> wrote: >> >>> When you say replication what exactly do you mean? In normal HDFS, as >>> you write the data is sent to 3 nodes yes, but with the flaw I >>> outlined, it doesnt matter because the datanodes and namenode will >>> pretend a data block just didnt exist if it wasnt closed properly. >>> >> >> That's the part I was not understanding. I do now. Thanks. >> >> >>> >>> So even with the most careful white glove handling of hbase, you will >>> eventually have a crash and you will lose data w/o 0.89/CDH3 et. al. >>> You can circumvent this by storing the data elsewhere and spooling >>> into hbase, or perhaps just not minding if you lose data (yes those >>> applications exist). >>> >>> Looking at those JIRAs in question, the first is already on trunk >>> which is 0.89. The second isn't alas. At this point the >>> transactional hbase just isnt being actively maintained by any >>> committer and we are reliant on kind people's contributions. So I >>> can't promise when it will hit 0.89/0.90. >>> >> >> Are you aware of any indexing alternatives in 0.89? >> >> >>> >>> -ryan >>> >>> >>> On Mon, Sep 20, 2010 at 1:21 PM, George P. Stathis <gstat...@traackr.com> >>> wrote: >>> > Thanks for the response Ryan. I have no doubt that 0.89 can be used in >>> > production and that it has strong support. I just wanted to avoid moving >>> to >>> > it now because we have limited resources and it would put a dent in our >>> > roadmap if we were to fast track the migration now. Specifically, we are >>> > using HBASE-2438 and HBASE-2426 to support pagination across indexes. So >>> we >>> > either have to migrate those to 0.89 or somehow go stock and be able to >>> > support pagination across region servers. >>> > >>> > Of course, if the choice is between migrating or losing more data, data >>> > safety comes first. But if we can buy two or three more months of time >>> and >>> > avoid region server crashes (like you did for a year), maybe we can go >>> that >>> > route for now. What do we need to do achieve that? >>> > >>> > -GS >>> > >>> > PS: Out of curiosity, I understand the WAL log append issue for a single >>> > regionserver when it comes to losing the data on a single node. But if >>> that >>> > data is also being replicated on another region server, why wouldn't it >>> be >>> > available there? Or is the WAL log shared across multiple region servers >>> > (maybe that's what I'm missing)? >>> > >>> > >>> > On Mon, Sep 20, 2010 at 3:52 PM, Ryan Rawson <ryano...@gmail.com> wrote: >>> > >>> >> Hey, >>> >> >>> >> The problem is that the stock 0.20 hadoop wont let you read from a >>> >> non-closed file. It will report that length as 0. So if a >>> >> regionserver crashes, that last WAL log that is still open becomes 0 >>> >> length and the data within in unreadable. That specifically is the >>> >> problem of data loss. You could always make it so your regionservers >>> >> rarely crash - this is possible btw and I did it for over a year. >>> >> >>> >> But you will want to run CDH3 or the append-branch releases to get the >>> >> series of patches that fix this hole. It also happens that only 0.89 >>> >> runs on it. I would like to avoid the hadoop "everyone uses 0.20 >>> >> forever" problem and talk about what we could do to help you get on >>> >> 0.89. Over here at SU we've made a commitment to the future of 0.89 >>> >> and are running it in production. Let us know what else you'd need. >>> >> >>> >> -ryan >>> >> >>> >> On Mon, Sep 20, 2010 at 12:39 PM, George P. Stathis >>> >> <gstat...@traackr.com> wrote: >>> >> > Thanks Todd. We are not quite ready to move to 0.89 yet. We have made >>> >> custom >>> >> > modifications to the transactional contrib sources which are now taken >>> >> out >>> >> > of 0.89. We are planning on moving to 0.90 when it comes out and at >>> that >>> >> > point, either migrate our customizations, or move back to the >>> out-of-the >>> >> box >>> >> > features (which will require a re-write of our code). >>> >> > >>> >> > We are well aware of the CDH distros but at the time we started with >>> >> hbase, >>> >> > there was none that included HBase. I think CDH3 the first one to >>> include >>> >> > HBase, correct? And is 0.89 the only one supported? >>> >> > >>> >> > Moreover, are we saying that there is no way to prevent stock hbase >>> >> 0.20.6 >>> >> > and hadoop 0.20.2 from losing data when a single node goes down? It >>> does >>> >> not >>> >> > matter if the data is replicated, it will still get lost? >>> >> > >>> >> > -GS >>> >> > >>> >> > On Sun, Sep 19, 2010 at 5:58 PM, Todd Lipcon <t...@cloudera.com> >>> wrote: >>> >> > >>> >> >> Hi George, >>> >> >> >>> >> >> The data loss problems you mentioned below are known issues when >>> running >>> >> on >>> >> >> stock Apache 0.20.x hadoop. >>> >> >> >>> >> >> You should consider upgrading to CDH3b2, which includes a number of >>> HDFS >>> >> >> patches that allow HBase to durably store data. You'll also have to >>> >> upgrade >>> >> >> to HBase 0.89 - we ship a version as part of CDH that will work well. >>> >> >> >>> >> >> Thanks >>> >> >> -Todd >>> >> >> >>> >> >> On Sun, Sep 19, 2010 at 6:57 AM, George P. Stathis < >>> >> gstat...@traackr.com >>> >> >> >wrote: >>> >> >> >>> >> >> > Hi folks. I'd like to run the following data loss scenario by you >>> to >>> >> see >>> >> >> if >>> >> >> > we are doing something obviously wrong with our setup here. >>> >> >> > >>> >> >> > Setup: >>> >> >> > >>> >> >> > - Hadoop 0.20.1 >>> >> >> > - HBase 0.20.3 >>> >> >> > - 1 Master Node running Nameserver, SecondaryNameserver, >>> JobTracker, >>> >> >> > HMaster and 1 Zookeeper (no zookeeper quorum right now) >>> >> >> > - 4 child nodes running a Datanode, TaskTracker and RegionServer >>> >> each >>> >> >> > - dfs.replication is set to 2 >>> >> >> > - Host: Amazon EC2 >>> >> >> > >>> >> >> > Up until yesterday, we were frequently experiencing >>> >> >> > HBASE-2077<https://issues.apache.org/jira/browse/HBASE-2077>, >>> >> >> > which kept bringing our RegionServers down. What we realized though >>> is >>> >> >> that >>> >> >> > we were losing data (a few hours worth) with just one out of four >>> >> >> > regionservers going down. This is problematic since we are supposed >>> to >>> >> >> > replicate at x2 out of 4 nodes, so at least one other node should >>> be >>> >> able >>> >> >> > to >>> >> >> > theoretically serve the data that the downed regionserver can't. >>> >> >> > >>> >> >> > Questions: >>> >> >> > >>> >> >> > - When a regionserver goes down unexpectedly, the only data that >>> >> >> > theoretically gets lost was whatever didn't make it to the WAL, >>> >> right? >>> >> >> Or >>> >> >> > wrong? E.g. >>> >> >> > >>> >> >> > >>> >> >> >>> >> >>> http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html >>> >> >> > - We ran a hadoop fsck on our cluster and verified the >>> replication >>> >> >> factor >>> >> >> > as well as that the were no under replicated blocks. So why was >>> our >>> >> >> data >>> >> >> > not >>> >> >> > available from another node? >>> >> >> > - If the log gets rolled every 60 minutes by default (we haven't >>> >> >> touched >>> >> >> > the defaults), how can we lose data from up to 24 hours ago? >>> >> >> > - When the downed regionserver comes back up, shouldn't that data >>> be >>> >> >> > available again? Ours wasn't. >>> >> >> > - In such scenarios, is there a recommended approach for >>> restoring >>> >> the >>> >> >> > regionserver that goes down? We just brought them back up by >>> logging >>> >> on >>> >> >> > the >>> >> >> > node itself an manually restarting them first. Now we have >>> automated >>> >> >> > crons >>> >> >> > that listen for their ports and restart them if they go down >>> within >>> >> two >>> >> >> > minutes. >>> >> >> > - Are there way to recover such lost data? >>> >> >> > - Are versions 0.89 / 0.90 addressing any of these issues? >>> >> >> > - Curiosity question: when a regionserver goes down, does the >>> master >>> >> >> try >>> >> >> > to replicate that node's data on another node to satisfy the >>> >> >> > dfs.replication >>> >> >> > ratio? >>> >> >> > >>> >> >> > For now, we have upgraded our HBase to 0.20.6, which is supposed to >>> >> >> contain >>> >> >> > the HBASE-2077 <https://issues.apache.org/jira/browse/HBASE-2077> >>> fix >>> >> >> (but >>> >> >> > no one has verified yet). Lars' blog also suggests that Hadoop >>> 0.21.0 >>> >> is >>> >> >> > the >>> >> >> > way to go to avoid the file append issues but it's not production >>> >> ready >>> >> >> > yet. Should we stick to 0.20.1? Upgrade to 0.20.2? >>> >> >> > >>> >> >> > Any tips here are definitely appreciated. I'll be happy to provide >>> >> more >>> >> >> > information as well. >>> >> >> > >>> >> >> > -GS >>> >> >> > >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Todd Lipcon >>> >> >> Software Engineer, Cloudera >>> >> >> >>> >> > >>> >> >>> > >>> >> >
