Hi Jeff, If the question is simply monitoring HDFS read/write latencies, please refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>, there's a patch but not committed yet, and probably cannot apply cleanly on current code base, but still some good reference IMHO, so JFYI.
To get an overview of how quickly the system could respond and what might be the root cause of the spikes, we only need to monitor the average/p99/p999 latency of below metrics (stages): 1. totalCallTime: time from request arriving at server to sending response 2. processCallTime: time for the server to process the call, regardless of the time this call being queued 3. queueCallTime: time the call has been queued 4. HDFS read/pread/write time: time of HFile reading/writing, added in HBASE-15160 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request However, for your original question, that to monitor the whole trace of a single request, I'm afraid no mature solution for the time being just as Stack mentioned. Hope my answer helps (smile). Best Regards, Yu On 4 March 2017 at 00:48, jeff saremi <jeffsar...@hotmail.com> wrote: > anything would help. thanks > > ________________________________ > From: saint....@gmail.com <saint....@gmail.com> on behalf of Stack < > st...@duboce.net> > Sent: Thursday, March 2, 2017 9:53:41 PM > To: Hbase-User > Subject: Re: Need guidance on getting detailed elapsed times in every > stage of processing a request > > On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsar...@hotmail.com> > wrote: > > > So i'd like to come back to my original question on how to get about > > separating the latency of HDFS from HBase. > > > > > That is a simple question to which we do not have an answer unfortunately > (we should). If interested, I could describe how you might do it. I don't > think it would take much work. > > St.Ack > > > > > Is there a most appropriate log4j TRACE option that could print out this > > information to the logs? > > Thanks > > > > ________________________________ > > From: jeff saremi <jeffsar...@hotmail.com> > > Sent: Thursday, March 2, 2017 12:45:59 PM > > To: Hbase-User > > Subject: Re: Need guidance on getting detailed elapsed times in every > > stage of processing a request > > > > Thanks so much for the advice! Looking forward to when Tracing gets > picked > > up again > > > > ________________________________ > > From: saint....@gmail.com <saint....@gmail.com> on behalf of Stack < > > st...@duboce.net> > > Sent: Thursday, March 2, 2017 12:17:35 PM > > To: Hbase-User > > Subject: Re: Need guidance on getting detailed elapsed times in every > > stage of processing a request > > > > HBase/HTrace integration once worked but has long since rotted. > > Refactorings of internals without proper respect for trace connections is > > the main culprit. Updates in htrace and hdfs that need attention > > reconnecting spans, etc., is another. On top of this, zipkin project has > > seen a burst of effort of late that would seem to offer much promise if > > someone of us spent some time rejiggering how HTrace and Zipkin relate. > > > > I would not waste any time on trying to setup HTrace for HBase at least > > until after HBASE-14451 goes in, an issue that has been put aside with a > > while now. Sorry if you've burned time on this to date. > > > > Yours, > > St.Ack > > > > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <jeffsar...@hotmail.com> > > wrote: > > > > > Where would i seek help for issues revolving around HTrace and zipkin? > > > Here? Because I have configured everything the way documentation said > > but i > > > see nothing in the zipkin server or in the logs. nothing at all > > > > > > ________________________________ > > > From: jeff saremi <jeffsar...@hotmail.com> > > > Sent: Tuesday, February 28, 2017 12:52:32 PM > > > To: user@hbase.apache.org > > > Subject: Re: Need guidance on getting detailed elapsed times in every > > > stage of processing a request > > > > > > No I had not. but it looks like what i needed. Thanks Ted. > > > > > > I'll see if I have any more questions after reading this. > > > > > > ________________________________ > > > From: Ted Yu <yuzhih...@gmail.com> > > > Sent: Tuesday, February 28, 2017 12:47:08 PM > > > To: user@hbase.apache.org > > > Subject: Re: Need guidance on getting detailed elapsed times in every > > > stage of processing a request > > > > > > Have you looked at: > > > http://hbase.apache.org/book.html#tracing > > > > > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <jeffsar...@hotmail.com> > > > wrote: > > > > > > > I think we need to get detailed information from HBase RegionServer > > logs > > > > on how a request (read or write) is processed. Specifically > speaking, i > > > > need to know of say 100 ms time spent in processing a write, how much > > of > > > it > > > > was spent waiting for the HDFS? > > > > What is the most efficient way of enabling this in log4j properties? > > Are > > > > there better mechanisms to get this information? > > > > > > > > If I can get this in the log, then I can process the logs offline or > in > > > > neartime and mount some dashboards on the top. > > > > > > > > thanks > > > > > > > > > > > > Jeff > > > > > > > > > >
