[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585090#comment-15585090
]
Yu Li commented on HBASE-16698:
-------------------------------
Thanks for coming back sir [~stack] :-)
bq. what if you added a new constructor on HLogKey, one that took a
WriteEntry...
I ever tried this way and believe me it will make the patch bigger because of
the new constructor (smile). And a new preAssignedWriteEntry won't interfere
with existing writeEntry so maybe easier to understand the logic?
bq. What to do for 1.3? Backport but flip the switch to false? We'd have to ask
Mikhail.
Yes, I guess false for now, and we could turn it to true latter if the perf
number shows no regression (I'm running the YCSB case but still haven't got all
numbers). And yes, I'd also like to hear Mikhail's opinion.
bq. For Master, should we try and do something better? Try batching up
sequenceid assign.
I also tried batch seqId assign but found it hard to pass the UT (I think
currently we have quite some logics depending on the fact of sequential seqId
assign, the {{SafePointZigZagLatch}} for example) so I applied current solution
here with lower risk to resolve our online issue (smile). But yes definitely
worthwhile to try since batch seqId assign is the real solution in my opinion.
bq. Apply a version of this patch in the meantime?
Yes, still a good work-around IMHO (smile).
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 1.2.3
> Reporter: Yu Li
> Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.patch, HBASE-16698.v2.patch,
> hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
