[
https://issues.apache.org/jira/browse/HDFS-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129265#comment-14129265
]
Colin Patrick McCabe commented on HDFS-6880:
--------------------------------------------
bq. Tracing is supposed to be controlled from the point of origin. The HTrace
API provides sample rate selection. See the use of ProbabilitySampler in
https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java#L947
I think we are all in agreement that "tracing is supposed to be controlled from
the point of origin." After all, this patch modifies DFSClient, which is the
point of origin for HDFS, to create trace spans. Nobody is proposing
controlling tracing on the server.
What I think is problematic here is that, if I understand correctly, the entire
DFSOutputStream is being considered as a single trace span, which is continued
with Trace.continueSpan whenever the stream does something. I don't think this
is the right way to do things in HDFS, since output streams can be open for
hours, or days, at a time. If we used something like {{Trace.startSpan("test
stream span", traceSampler);}}, we could get 1% of all streams sampled,
perhaps, but they might not be the 1% we want. The HBase WAL stays open for
days at a time... if it happens to be one of the streams that sampled, things
will get really slow. On the other hand, if it doesn't get sampled, we'll have
no information about it in our trace data.
We need finer granularity. Does that make sense?
A DFSOutputStream is more like an HBase *client* than like an HBase *row
operation*. You clearly would not want to have a single trace span for the
whole hbase client, and I think you don't want a single trace span for the
whole DFSOutputStream.
bq. Because DFSInputStream does not use Thread in normal path as
DFSOutputStream do, it is not crucial to add the code (for passing tracing
infos between threads) in it. But it might be better if the code is symmetric
for input and output. I will consider to add tracing span to DFSInputStream on
updating the patch.
I understand that in the special case of HBase (or another application that
uses HTrace), the DFSClient may have already created a trace span. But we
should consider other clients, which haven't yet been instrumented with HTrace,
that want to see input stream data show up.
> Adding tracing to DataNode data transfer protocol
> -------------------------------------------------
>
> Key: HDFS-6880
> URL: https://issues.apache.org/jira/browse/HDFS-6880
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Masatake Iwasaki
> Assignee: Masatake Iwasaki
> Attachments: HDFS-6880-0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
