Andy, we will use LoadIncrementalHFiles, are you saying that this will never cause a split?
Joey, our policies will not allow us to deploy a Hadoop version with append, until there is an Apache release that includes both security and append. So t.isis not about cost of appending to the WAL - we simply don't have that option. Thanks -Andreas. On Wed, Jun 22, 2011 at 7:14 PM, Joey Echeverria <[email protected]> wrote: > > I agree that even though these are rare, they are not rare enough to take > a > > risk. But they could be rare enough to justify a less efficient > > implementation of the WAL. Would it be reasonable to use an > implementation > > of HLog that - at the price of performance - persists the WAL to HDFS > > without relying on append? > > Since edits to .META. should be rare if you're using bulkImport, why > not just pay the cost of appending to the WAL? My guess is performance > would not suffer greatly. Would be nice to benchmark my claim :) > > -Joey > On Wed, Jun 22, 2011 at 7:09 PM, Andrew Purtell <[email protected]> wrote: > > From: Andreas Neumann <[email protected]> > > > I guess that includes region splits but also > > reassignment of a region after its region server died. > > If you are not writing, then you won't see splits. > > Reassignment does involve writes to META but missing these is less serious > than missing a change in table structure. Stack could say more here but I > believe the Master will discover inconsistency and repair it. > > > But they could be rare enough to justify a less efficient > > implementation of the WAL. Would it be reasonable to use an > > implementation of HLog that - at the price of performance - > > persists the WAL to HDFS without relying on append? > > How would that work if not create/write/close on every WAL commit? Correct > me if I'm wrong, but there is no way to guarantee persistence in an HDFS > without append except to .close() the file. > > HBase can do the (near) equivalent of this without requiring any code > changes. Set hbase.regionserver.logroll.period to an insanely low value. > > - Andy > >
