The double-detach should be fixed in HBase as well.
On Thu, Feb 26, 2015 at 1:07 AM, Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:
> > perhaps a double detach.
>
> It turned out to be double detach of NullScope singleton and I filed
> HTRACE-119.
> I got the error on meta assignments on regionserver startup without
> starting trace spans..
>
> Thanks,
> Masatake
>
>
> On 2/26/15 09:58, Colin P. McCabe wrote:
>
>> Hmm. Looking at that error, my guess would be that there is an
>> incorrect usage of TraceScope#detach going on somewhere in hbase...
>> perhaps a double detach. But I could be wrong. We added some code
>> recently to catch issues like this.
>>
>> best,
>> Colin
>>
>> On Wed, Feb 25, 2015 at 12:28 AM, Masatake Iwasaki
>> <iwasak...@oss.nttdata.co.jp> wrote:
>>
>>> I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick
>>> fix
>>> to TestHTraceHooks).
>>> I got the error below in regionserver log.
>>> I will dig this tomorrow.::
>>>
>>> 2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0]
>>> htrace.Tracer: Tried to detach trace span null but it has already been
>>> detached.
>>> 2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0]
>>> handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.
>>> 1588230740,
>>> starting to roll back the global memstore size.
>>> java.lang.RuntimeException: Tried to detach trace span null but it has
>>> already been detached.
>>> at org.apache.htrace.Tracer.clientError(Tracer.java:61)
>>> at org.apache.htrace.TraceScope.detach(TraceScope.java:57)
>>> at
>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559)
>>> at
>>> org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(
>>> WALUtil.java:94)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> writeRegionOpenMarker(HRegion.java:910)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> openHRegion(HRegion.java:4911)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> openHRegion(HRegion.java:4874)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> openHRegion(HRegion.java:4845)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> openHRegion(HRegion.java:4801)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>> openHRegion(HRegion.java:4752)
>>> at
>>> org.apache.hadoop.hbase.regionserver.handler.
>>> OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
>>> at
>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(
>>> OpenRegionHandler.java:126)
>>> at
>>> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>>
>>> On 2/24/15 18:27, Colin P. McCabe wrote:
>>>
>>>> Thanks for trying this, Mastake. I've got HDFS working on my cluster
>>>> with tracing and LocalFileSpanReceiver. Did you try using HBase +
>>>> HDFS with LocalFileSpanReceiver? Be sure to use a build including
>>>> HTRACE-112 since LFSR was kind of busted prior to that.
>>>>
>>>> I'm going to do a longer writeup about getting HDFS + HBase working
>>>> with other span receivers just as soon as I finish stomping a few more
>>>> bugs.
>>>>
>>>> best,
>>>> Colin
>>>>
>>>> On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki
>>>> <iwasak...@oss.nttdata.co.jp> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks for trying this. I am sorry for late reply.
>>>>>
>>>>> I tried this today
>>>>> by hbase-1.0.1-SANPSHOT built with
>>>>> {{-Dhadoop-two.version=2.7.0-SNAPSHOT}}
>>>>> in pseudo distributed cluster
>>>>> but failed to get end-to-end trace.
>>>>>
>>>>> I checked that
>>>>> * tracing works for both of hbase and hdfs,
>>>>> * hbase runs with 2.7.0-SNAPSHOT jar of hadoop.
>>>>>
>>>>> When I did do put with tracing on,
>>>>> I saw span named "FSHLog.sync" with annotations such as
>>>>> "syncing writer" and "writer synced".
>>>>> The code for tracing in FSHLog worked at least.
>>>>>
>>>>> I'm still looking into this.
>>>>> If it turned out that tracing spans are not reached to
>>>>> actual HDFS writer thread in HBase, I will file a JIRA.
>>>>>
>>>>> # We need hadoop-2.6.0 or higher in order to trace HDFS.
>>>>> # Building hbase from source with {{-Dhadoop-two.version=2.6.0}}
>>>>> # is straight forward way to do this
>>>>> # because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars.
>>>>>
>>>>> Masatake
>>>>>
>>>>>
>>>>> On 2/11/15 08:56, Nick Dimiduk wrote:
>>>>>
>>>>>> Hi Joshua,
>>>>>>
>>>>>> In theory there's nothing special for you to do. Just issue your query
>>>>>> to
>>>>>> HBase with tracing enabled. The active span will go through HBase,
>>>>>> down
>>>>>> into HDFS, and back again. You'll need both systems collecting spans
>>>>>> into
>>>>>> the same place so that you can report on the complete trace tree.
>>>>>>
>>>>>> I've not recently tested the end-to-end, but I believe it's all there.
>>>>>> If
>>>>>> not, it's a bug -- this is an intended use case. Can you give it a try
>>>>>> and let us know how it goes?
>>>>>>
>>>>>> FYI, 0.99.x are preview releases of HBase and not for production use.
>>>>>> Just
>>>>>> so you know :)
>>>>>>
>>>>>> -n
>>>>>>
>>>>>> On Wednesday, February 11, 2015, Chunxu Tang <chunxut...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>>
>>>>>>> Now I’m exploiting HTrace to trace request level data flows in HBase
>>>>>>> and
>>>>>>> HDFS. I have successfully traced HBase and HDFS by using HTrace,
>>>>>>> respectively.
>>>>>>>
>>>>>>> After that, I combine HBase and HDFS together and I want to just
>>>>>>> send a
>>>>>>> PUT/GET request to HBase, but to trace the whole data flow in both
>>>>>>> HBase
>>>>>>> and HDFS. In my opinion, when I send a request such as Get to HBase,
>>>>>>> it
>>>>>>> will at last try to read the blocks on HDFS, so I can construct a
>>>>>>> whole
>>>>>>> data flow tracing through HBase and HDFS. While, the fact is that I
>>>>>>> can
>>>>>>> only get tracing data of HBase, with no data of HDFS.
>>>>>>>
>>>>>>> Could you give me any suggestions on how to trace the data flow in
>>>>>>> both
>>>>>>> HBase and HDFS? Does anyone have similar experience? Do I need to
>>>>>>> modify
>>>>>>> the source code? And maybe which part(s) should I touch? If I need to
>>>>>>> modify the code, I will try to create a patch for that.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>> My Configurations:
>>>>>>> Hadoop version: 2.6.0
>>>>>>> HBase version: 0.99.2
>>>>>>> HTrace version: htrace-master
>>>>>>> OS: Ubuntu 12.04
>>>>>>>
>>>>>>>
>>>>>>> Joshua
>>>>>>>
>>>>>>>
>
