-user@hbase to bcc +dev@hbase Jeff,
These kinds of questions get into things that we only handle within the development community. I've copied in the appropriate dev@hbase mailing list, please ensure you are subscribed there so you can see any further discussion. On Fri, Mar 3, 2017 at 1:13 PM, jeff saremi <jeffsar...@hotmail.com> wrote: > Yu > Of the patches attached to HBASE-15160, do I need to apply all (v2, v3, ...) > or just HBASE-15160.patch ? > Also how would I know against what version this patch was created? > thanks > > > > > ________________________________ > From: jeff saremi <jeffsar...@hotmail.com> > Sent: Friday, March 3, 2017 10:34:00 AM > To: Hbase-User > Subject: Re: Need guidance on getting detailed elapsed times in every stage > of processing a request > > Thanks a lot Yu > > These are truly the metrics we care about about at this point. It is sad to > see that such important metrics were removed from the code. > > I will try to apply your patch on my own to the version of HBase we have. We > definitely need these. > > Other solutions like HTrace are not as urgent as having these few metrics you > talked about here. So if we can get these merged with the code we should be > happy. > > > ________________________________ > From: Yu Li <car...@gmail.com> > Sent: Friday, March 3, 2017 9:54:29 AM > To: Hbase-User > Subject: Re: Need guidance on getting detailed elapsed times in every stage > of processing a request > > Hi Jeff, > > If the question is simply monitoring HDFS read/write latencies, please > refer to HBASE-15160 <https://issues.apache.org/jira/browse/HBASE-15160>, > there's a patch but not committed yet, and probably cannot apply cleanly on > current code base, but still some good reference IMHO, so JFYI. > > To get an overview of how quickly the system could respond and what might > be the root cause of the spikes, we only need to monitor the > average/p99/p999 latency of below metrics (stages): > 1. totalCallTime: time from request arriving at server to sending response > 2. processCallTime: time for the server to process the call, regardless of > the time this call being queued > 3. queueCallTime: time the call has been queued > 4. HDFS read/pread/write time: time of HFile reading/writing, added in > HBASE-15160 > 5. WAL sync time: time of WAL sync to HDFS, critical path of writing request > > However, for your original question, that to monitor the whole trace of a > single request, I'm afraid no mature solution for the time being just as > Stack mentioned. > > Hope my answer helps (smile). > > Best Regards, > Yu > > On 4 March 2017 at 00:48, jeff saremi <jeffsar...@hotmail.com> wrote: > >> anything would help. thanks >> >> ________________________________ >> From: saint....@gmail.com <saint....@gmail.com> on behalf of Stack < >> st...@duboce.net> >> Sent: Thursday, March 2, 2017 9:53:41 PM >> To: Hbase-User >> Subject: Re: Need guidance on getting detailed elapsed times in every >> stage of processing a request >> >> On Thu, Mar 2, 2017 at 10:26 PM, jeff saremi <jeffsar...@hotmail.com> >> wrote: >> >> > So i'd like to come back to my original question on how to get about >> > separating the latency of HDFS from HBase. >> > >> > >> That is a simple question to which we do not have an answer unfortunately >> (we should). If interested, I could describe how you might do it. I don't >> think it would take much work. >> >> St.Ack >> >> >> >> > Is there a most appropriate log4j TRACE option that could print out this >> > information to the logs? >> > Thanks >> > >> > ________________________________ >> > From: jeff saremi <jeffsar...@hotmail.com> >> > Sent: Thursday, March 2, 2017 12:45:59 PM >> > To: Hbase-User >> > Subject: Re: Need guidance on getting detailed elapsed times in every >> > stage of processing a request >> > >> > Thanks so much for the advice! Looking forward to when Tracing gets >> picked >> > up again >> > >> > ________________________________ >> > From: saint....@gmail.com <saint....@gmail.com> on behalf of Stack < >> > st...@duboce.net> >> > Sent: Thursday, March 2, 2017 12:17:35 PM >> > To: Hbase-User >> > Subject: Re: Need guidance on getting detailed elapsed times in every >> > stage of processing a request >> > >> > HBase/HTrace integration once worked but has long since rotted. >> > Refactorings of internals without proper respect for trace connections is >> > the main culprit. Updates in htrace and hdfs that need attention >> > reconnecting spans, etc., is another. On top of this, zipkin project has >> > seen a burst of effort of late that would seem to offer much promise if >> > someone of us spent some time rejiggering how HTrace and Zipkin relate. >> > >> > I would not waste any time on trying to setup HTrace for HBase at least >> > until after HBASE-14451 goes in, an issue that has been put aside with a >> > while now. Sorry if you've burned time on this to date. >> > >> > Yours, >> > St.Ack >> > >> > On Thu, Mar 2, 2017 at 6:28 AM, jeff saremi <jeffsar...@hotmail.com> >> > wrote: >> > >> > > Where would i seek help for issues revolving around HTrace and zipkin? >> > > Here? Because I have configured everything the way documentation said >> > but i >> > > see nothing in the zipkin server or in the logs. nothing at all >> > > >> > > ________________________________ >> > > From: jeff saremi <jeffsar...@hotmail.com> >> > > Sent: Tuesday, February 28, 2017 12:52:32 PM >> > > To: u...@hbase.apache.org >> > > Subject: Re: Need guidance on getting detailed elapsed times in every >> > > stage of processing a request >> > > >> > > No I had not. but it looks like what i needed. Thanks Ted. >> > > >> > > I'll see if I have any more questions after reading this. >> > > >> > > ________________________________ >> > > From: Ted Yu <yuzhih...@gmail.com> >> > > Sent: Tuesday, February 28, 2017 12:47:08 PM >> > > To: u...@hbase.apache.org >> > > Subject: Re: Need guidance on getting detailed elapsed times in every >> > > stage of processing a request >> > > >> > > Have you looked at: >> > > http://hbase.apache.org/book.html#tracing >> > > >> > > On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi <jeffsar...@hotmail.com> >> > > wrote: >> > > >> > > > I think we need to get detailed information from HBase RegionServer >> > logs >> > > > on how a request (read or write) is processed. Specifically >> speaking, i >> > > > need to know of say 100 ms time spent in processing a write, how much >> > of >> > > it >> > > > was spent waiting for the HDFS? >> > > > What is the most efficient way of enabling this in log4j properties? >> > Are >> > > > there better mechanisms to get this information? >> > > > >> > > > If I can get this in the log, then I can process the logs offline or >> in >> > > > neartime and mount some dashboards on the top. >> > > > >> > > > thanks >> > > > >> > > > >> > > > Jeff >> > > > >> > > >> > >>
