[
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355897#comment-15355897
]
Mikhail Antonov commented on HBASE-16142:
-----------------------------------------
>>>(more ideal would be a trace of the explicit slow queries showing call stack
>>>with timings dumped to a sink for later review; i.e. trigger an htrace when
>>>a query is slow...
Sounds interesting, but you can't trace only slow queries because you don't
know upfront on the client if the query is going to be slow so you can start
the trace? Maybe if tracing is already on. increase sampling in rpc client to
servers which experience high latency or something? Though in general idea of
programmatically increasing sampling ratio would be a bit scary. With hard
upper limit could be fine though.
> Trigger JFR session when under duress -- e.g. backed-up request queue count
> -- and dump the recording to log dir
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-16142
> URL: https://issues.apache.org/jira/browse/HBASE-16142
> Project: HBase
> Issue Type: Task
> Components: Operability
> Reporter: stack
> Priority: Minor
> Labels: beginner
>
> Chatting today w/ a mighty hbase operator on how to figure what is happening
> during transitory latency spike or any other transitory 'weirdness' in a
> server, the idea came up that a java flight recording during a spike would
> include a pretty good picture of what is going on during the time of duress
> (more ideal would be a trace of the explicit slow queries showing call stack
> with timings dumped to a sink for later review; i.e. trigger an htrace when a
> query is slow...).
> Taking a look, programmatically triggering a JFR recording seems doable, if
> awkward (MBean invocations). There is even a means of specifying 'triggers'
> based off any published mbean emission -- e.g. a query queue count threshold
> -- which looks nice. See
> https://community.oracle.com/thread/3676275?start=0&tstart=0 and
> https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH184
> This feature could start out as a blog post describing how to do it for one
> server. A plugin on Canary that looks at mbean values and if over a
> configured threshold, triggers a recording remotely could be next. Finally
> could integrate a couple of triggers that fire when issue via the trigger
> mechanism.
> Marking as beginner feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
