> When HDFS gets tiered storage, we can revive this and put HBase's WAL on SSD > storage.
I think the archival storage work just committed (and ported into branch-2?) might be sufficient for a pilot of HBASE-6572. A few important changes were filed late, like APIs for apps for setting policy and persistence for changes in the edit log, so we should survey what is in there exactly. Related: HDFS-5682 On Sep 28, 2014, at 1:32 PM, lars hofhansl <[email protected]> wrote: >> Are anyone aware of any company who does not > use the hdfs default policy and flush every WAL sync. > > > It's a trade-off. You'll only lose data when and the wrong three machines die > around the same time (you'd have an outage that any block that exists only on > these three boxes). Not also that time of the data not being on disk is > bounded, eventually the OS the flush dirty pages, Linux does it every 15s by > default. > > So you'd have all machines die before a single of them manages to flush the > dirty pages to disk. Of course that can happen, for example during a data > center power outage. > > > A while ago I added HDFS-744 to HDFS, but never finished the parts in HBase > as nobody (including myself in the end) was interested in it. Reminds to > maybe take this up again in HBase 2.0 since now we support fewer versions of > Hadoop. > > > When HDFS gets tiered storage, we can revive this and put HBase's WAL on SSD > storage. > > > -- Lars > > > > ----- Original Message ----- > From: abhishek1015 <[email protected]> > To: [email protected] > Cc: > Sent: Sunday, September 28, 2014 9:13 AM > Subject: Re: hbase row locking > > Sorry for confusion. I meant that I am getting 6000 ops/sec throughput > overall using 4 machine. That is 1500 ops/sec/regionserver on average. > > I checked the ping response time between machines. It is approximately .09 > ms. > > Assuming that WAL sync thread tries to sync with two other hdfs node > sequentially, the row lock will be held for at least 0.18 ms, which will > still give a very high throughput per regionserver even if only one thread > is working and all other threads are blocked because of locking. > > It appears that bottleneck is then the hdfs disk flush. And, consequently, > above mentioned schema are equivalent w.r.t. performance. > > However, I have a question regarding the default hdfs policy of not flushing > every WAL sync. Are not people in industry afraid of data loss however small > probability of this happening. Are anyone aware of any company who does not > use the hdfs default policy and flush every WAL sync. > > Thanks > Abhishek > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/hbase-row-locking-tp4064432p4064458.html > > > > Sent from the HBase User mailing list archive at Nabble.com. >
