> perhaps a double detach.
It turned out to be double detach of NullScope singleton and I filed
HTRACE-119.
I got the error on meta assignments on regionserver startup without
starting trace spans..
Thanks,
Masatake
On 2/26/15 09:58, Colin P. McCabe wrote:
Hmm. Looking at that error, my guess would be that there is an
incorrect usage of TraceScope#detach going on somewhere in hbase...
perhaps a double detach. But I could be wrong. We added some code
recently to catch issues like this.
best,
Colin
On Wed, Feb 25, 2015 at 12:28 AM, Masatake Iwasaki
<iwasak...@oss.nttdata.co.jp> wrote:
I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick fix
to TestHTraceHooks).
I got the error below in regionserver log.
I will dig this tomorrow.::
2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0]
htrace.Tracer: Tried to detach trace span null but it has already been
detached.
2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0]
handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740,
starting to roll back the global memstore size.
java.lang.RuntimeException: Tried to detach trace span null but it has
already been detached.
at org.apache.htrace.Tracer.clientError(Tracer.java:61)
at org.apache.htrace.TraceScope.detach(TraceScope.java:57)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559)
at
org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:94)
at
org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:910)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4911)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4874)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
On 2/24/15 18:27, Colin P. McCabe wrote:
Thanks for trying this, Mastake. I've got HDFS working on my cluster
with tracing and LocalFileSpanReceiver. Did you try using HBase +
HDFS with LocalFileSpanReceiver? Be sure to use a build including
HTRACE-112 since LFSR was kind of busted prior to that.
I'm going to do a longer writeup about getting HDFS + HBase working
with other span receivers just as soon as I finish stomping a few more
bugs.
best,
Colin
On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki
<iwasak...@oss.nttdata.co.jp> wrote:
Hi,
Thanks for trying this. I am sorry for late reply.
I tried this today
by hbase-1.0.1-SANPSHOT built with
{{-Dhadoop-two.version=2.7.0-SNAPSHOT}}
in pseudo distributed cluster
but failed to get end-to-end trace.
I checked that
* tracing works for both of hbase and hdfs,
* hbase runs with 2.7.0-SNAPSHOT jar of hadoop.
When I did do put with tracing on,
I saw span named "FSHLog.sync" with annotations such as
"syncing writer" and "writer synced".
The code for tracing in FSHLog worked at least.
I'm still looking into this.
If it turned out that tracing spans are not reached to
actual HDFS writer thread in HBase, I will file a JIRA.
# We need hadoop-2.6.0 or higher in order to trace HDFS.
# Building hbase from source with {{-Dhadoop-two.version=2.6.0}}
# is straight forward way to do this
# because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars.
Masatake
On 2/11/15 08:56, Nick Dimiduk wrote:
Hi Joshua,
In theory there's nothing special for you to do. Just issue your query
to
HBase with tracing enabled. The active span will go through HBase, down
into HDFS, and back again. You'll need both systems collecting spans
into
the same place so that you can report on the complete trace tree.
I've not recently tested the end-to-end, but I believe it's all there.
If
not, it's a bug -- this is an intended use case. Can you give it a try
and let us know how it goes?
FYI, 0.99.x are preview releases of HBase and not for production use.
Just
so you know :)
-n
On Wednesday, February 11, 2015, Chunxu Tang <chunxut...@gmail.com>
wrote:
Hi all,
Now I’m exploiting HTrace to trace request level data flows in HBase
and
HDFS. I have successfully traced HBase and HDFS by using HTrace,
respectively.
After that, I combine HBase and HDFS together and I want to just send a
PUT/GET request to HBase, but to trace the whole data flow in both
HBase
and HDFS. In my opinion, when I send a request such as Get to HBase, it
will at last try to read the blocks on HDFS, so I can construct a whole
data flow tracing through HBase and HDFS. While, the fact is that I can
only get tracing data of HBase, with no data of HDFS.
Could you give me any suggestions on how to trace the data flow in both
HBase and HDFS? Does anyone have similar experience? Do I need to
modify
the source code? And maybe which part(s) should I touch? If I need to
modify the code, I will try to create a patch for that.
Thank you.
My Configurations:
Hadoop version: 2.6.0
HBase version: 0.99.2
HTrace version: htrace-master
OS: Ubuntu 12.04
Joshua
