[
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994282#comment-13994282
]
stack edited comment on HBASE-11135 at 5/10/14 7:03 PM:
--------------------------------------------------------
[~jeffreyz] has been reviewing offline. Comments that if CP is slow doing
pre/post append, then as is, we will slow throughput. But as he notes, this is
as it is in 0.98 and not really a way around it since cp, and wal listeners are
inline w/ the actual append.
In this patch over most of the appendNoSyncs to use new method (so
edit/sequence id is availble on return out of append). Changed the append to
use a latch. This gained me back a few percent... maybe 5%... but this patch
makes us slower appending. When 200 threads contending, about half slower
again (180seconds vs 280 seconds to complete test):
{code}
$ for i in 1 3 5 10 50 200; do for j in 1 2 3; do perf stat
./hbase-0.99.0-SNAPSHOT/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads $i
-iterations 100000 -keySize 50 -valueSize 100 &>
"/tmp/wpatch_latch_barrier.${i}.${j}.txt"; done; done
{code}
I tried then also using a cyclicbarrier in the SyncFuture for the handler
thread and the the sync'ing thread to coordinate around but no noticeable
difference (perhaps a little slower).
I'd be good committing this for now since it simplifies the unification of mvcc
and sequence id. Even when we are half slower again we are still about 0.98
throughput in trunk since the claim in HBASE-10156 is that it doubled our
throughput at 200 threads.
After the unification happens, we can come back here again to do see if we can
get perf back.
was (Author: stack):
[~jeffreyz] has been reviewing offline. Comments that if CP is slow doing
pre/post append, then as is, we will slow throughput. But as he notes, this is
as it is in 0.98 and not really a way around it since cp, and wal listeners are
inline w/ the actual append.
In this patch over most of the appendNoSyncs to use new method (so
edit/sequence id is availble on return out of append). Changed the append to
use a latch. This gained me back a few percent... maybe 5%... but this patch
makes us slower appending. When 200 threads contending, about 1/3rd slower
again (180seconds vs 280 seconds to complete test):
{code}
$ for i in 1 3 5 10 50 200; do for j in 1 2 3; do perf stat
./hbase-0.99.0-SNAPSHOT/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads $i
-iterations 100000 -keySize 50 -valueSize 100 &>
"/tmp/wpatch_latch_barrier.${i}.${j}.txt"; done; done
{code}
I tried then also using a cyclicbarrier in the SyncFuture for the handler
thread and the the sync'ing thread to coordinate around but no noticeable
difference (perhaps a little slower).
I'd be good committing this for now since it simplifies the unification of mvcc
and sequence id. Even when we are 1/3rd slower we are still ahead of 0.98 in
trunk since the claim in HBASE-10156 is that it doubled our throughput at 200
threads.
After the unification happens, we can come back here again to do see if we can
get perf back.
> Change region sequenceid generation so happens earlier in the append cycle
> rather than just before added to file
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11135
> URL: https://issues.apache.org/jira/browse/HBASE-11135
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Reporter: stack
> Assignee: stack
> Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt
>
>
> Currently we assign the region edit/sequence id just before we put it in the
> WAL. We do it in the single thread that feeds from the ring buffer. Doing
> it at this point, we can ensure order, that the edits will be in the file in
> accordance w/ the ordering of the region sequence id.
> But the point at which region sequence id is assigned an edit is deep down in
> the WAL system and there is a lag between our putting an edit into the WAL
> system and the edit actually getting its edit/sequence id.
> This lag -- "late-binding" -- complicates the unification of mvcc and region
> sequence id, especially around async WAL writes (and, related, for no-WAL
> writes) -- the parent for this issue (For async, how you get the edit id in
> our system when the threads have all gone home -- unless you make them wait?)
> Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of
> getting the region sequence id near-immediately. We'll run two ringbuffers.
> The first will mesh all handler threads and the consumer will generate ids
> (we will have order on other side of this first ring buffer), and then if
> async or no sync, we will just let the threads return ... updating mvcc just
> before we let them go. All other calls will go up on to the second ring
> buffer to be serviced as now (batching, distribution out among the sync'ing
> threads). The first rb will have no friction and should turn at fast rates
> compared to the second. There should not be noticeable slowdown nor do I
> foresee this refactor intefering w/ our multi-WAL plans.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
