On Mon, Dec 21, 2015 at 2:31 AM, 鈴木俊裕 <brfrn...@gmail.com> wrote:
> St.Ack > > I am sorry for the late reply. > > Thank you for the reply. > This is the test code: > https://github.com/brfrn169/hbase-test This helps. The test has a different character to others we currently have (a thread keeps writing its row rather than a thread writing all rows in region or all threads writing a single row). Let me try it. > > > We applied the patch you can find below to HBase-1.0.0 to resolve the > performance degradation: > https://gist.github.com/brfrn169/15a874594be2fb9d6ea0 > > It showed a good performance. > > The patch is attractive because it keeps the change in MVCC class only. It is a pretty radical change though with some changes that look like they could go upstream. There are some questions below but no problem if you are busy, I can go do my own study. It looks like advanceMemstore had a "blind spot": i.e. we could complete a WriteEntry but it might not yet have had its write numbers assigned... Is that why you added the new advancedNoWriteNumberWriteEntries Set? + if (queueFirst.getWriteNumber() == NO_WRITE_NUMBER) {+ advancedNoWriteNumberWriteEntries.add(queueFirst); Are we sure that we'll move up through the write mvcc sequence numbers in order? We seem to be able to take from the advancedNoWriteNumberWriteEntries Set w/o concern for order. You remove the writeQueue notify and seem to flip instead to wait/notify on readWaiters. Less synchronization points seems good. We should do this upstream too (and your change of > to >= looking at nextReadValue?) Do you have a notion of how much more throughput you got with this change? Thank you 鈴木俊裕, St.Ack > I think the direct cause of the performance degradation is not a region > level lock because HBase-0.98.6 also has a region level lock. > I think lock contention is caused by HBASE-8763. > > The patch mitigates that lock contention. > > Thanks, > Toshihiro Suzuki > > > 2015-12-14 15:12 GMT+09:00 Stack <st...@duboce.net>: > > > On Tue, Sep 8, 2015 at 2:01 AM, 鈴木俊裕 <brfrn...@gmail.com> wrote: > > > > > ... > > > > > > Also we wrote performance test code for increment operation that > included > > > 100 threads and ran it in local mode. > > > > > > The result is shown below: > > > > > > CDH5.3.1(HBase0.98.6) > > > Throughput(op/s): 12757, Latency(ms): 7.975072509210629 > > > > > > CDH5.4.5(HBase1.0.0) > > > Throughput(op/s): 2027, Latency(ms): 49.11840157868772 > > > > > > > > Do you have this program Toshihiro Suzuki still? May I see it? I am > > interested in seeing what the 100 threads are doing, if they are all > > updating the same Cell or if they are spread over many rows (I see master > > branch is more than 2x slower than 0.94 if all threads are contending on > a > > single Cell but if not contending, master seems faster -- I must be doing > > something wrong over in my experiments on HBASE-14460). I would also be > > interested in what you loading is like in production if you can describe > it > > at all (send me offlist if you'd rather talk about it in public). In > > production, can you tell how much slowdown you are seeing? Is it 2x or 7x > > as per your test program? > > > > Thank you, > > St.Ack > > > > > > > > > > > > Thanks, > > > > > > Toshihiro Suzuki > > > > > >
