[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Resolution: Fixed Status: Resolved (was: Patch Available) > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: BB2015-05-TBR > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, > HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, > HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, > HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-5274: --- Labels: BB2015-05-TBR (was: ) > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: BB2015-05-TBR > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, > HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, > HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, > HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-17.patch I updated the patch - rebased to trunk - fixed the init/close logic of SpanReceiverHost - bumped the version of htrace to 3.0.4 > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, > HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, > HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, > HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-16.patch > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, > HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, > HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, > HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace > d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-15.patch I added htrace-zipkin to dependency. All of the dependency of htrace-zipkin except libthrift is already bundled with Hadoop. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, > HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, > HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace > a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, > ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-14.patch attaching updated patch. - rebased to current trunk - bumped the version of htrace to 3.0.3 - added documentation of tracing [~stack] I did not add htrace-zipkin to the dependency of Hadoop in order to keep hadoop-common as clean as possible. I added the documentation including the setup procedure of htrace-zipkin instead. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-14.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, > HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, > HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: 3node_put_200mb.png > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, > HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, > HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace > d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: 3node_get_200mb.png 3node_put_200mb.png adding screen shots of tracing on 3node cluster: 3node_put_200mb.png and 3node_get_200mb.png > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: 3node_get_200mb.png, 3node_put_200mb.png, > HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, > HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, > HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, > HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace > a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, > ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-13.patch added tests for the cases tracing disabled. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, > HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, > HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, > HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace > d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-12.patch fixed the test if tracing in Receiver. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, > ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HDFS-5274: Attachment: HDFS-5274-11.txt + Fix double-close in Server.java + Minor fix of javadoc comment + Fix few places where we were not testing if trace enabled. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-2.patch, HDFS-5274-3.patch, > HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, > HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace > a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, > ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HDFS-5274: Attachment: HDFS-5274-10.patch Rebase and updated the patch to refer to released htrace (which is now in maven central so no need of the sonatype repo reference). > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, > HDFS-5274-10.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, > HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, > HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-9.patch attaching updated patch. * rebased to trunk, * changed package name from org.cloudera.htrace to org.htrace, * added repository to top level pom. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, > ss-5274v8-get.png, ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-8.patch > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace > a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, > ss-5274v8-put.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-5274:
---
Attachment: ss-5274v8-get.png
ss-5274v8-put.png
HDFS-5274-8.patch
I am attaching updated patch and screen shots of trace of putting and getting a
200MB file.
bq. Fix these in next patch:
fixed.
bq. Is formatting ok here?
fixed.
bq. In BlockReceiver, should traceSpan be getting closed?
added description to span and calling close().
{quote}
Is it possible that below throws an exception?
+ scope.getSpan().addKVAnnotation(
+ "stream".getBytes(),
+ jas.getCurrentStream().toString().getBytes());
i..e. we can hope out w/o closing the span since the try/finally only happens
later.
This is in JournalSet in a few places.
{quote}
I moved these code in try block to make sure.
bq. TraceInfo and RPCTInfo seem to be same datastructure? Should we define it
onetime only and share?'
I prefer keeping this as is because of simplicity and independency between
datatransfer protocol and o.a.h.ipc.
bq. I checked the trace of putting and getting a big file by Zipkin today.
There seems to be too many spans concerning "DFSInputStream.read" and
"DFSOutputStream.write". I will fix this in the next version of patch.
just removed those spans from DFSInputStream and DFSOutputStream.
> Add Tracing to HDFS
> ---
>
> Key: HDFS-5274
> URL: https://issues.apache.org/jira/browse/HDFS-5274
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
> HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
> HDFS-5274-7.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png,
> Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
>
>
> Since Google's Dapper paper has shown the benefits of tracing for a large
> distributed system, it seems like a good time to add tracing to HDFS. HBase
> has added tracing using HTrace. I propose that the same can be done within
> HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-5274: --- Attachment: HDFS-5274-7.patch I am attaching the patch rebased and updated based on review comments. bq. Any reason we take config on construction and in init for SpanReceiverHost? I removed conf from constructor argument. bq. SpanReceiverHost is on only when trace is enabled, right? If so, say so in class comment. SpanReceiverHost is always on, though it do nothing if no SpanReceiver is configured. I added a line in class comment. bq. Has to be a shutdown hook? ShutdownHookManager.get().addShutdownHook ? This is fine unless we envision someone having to override it which I suppose should never happen for an optionally enabled, rare, trace function? Overriding SpanReceiverHost is not necessary, though there could be someone who implement SpanReceiver. I think it is useful to wait for receivers to process all the tracing data on crash scenario. bq. HTraceConfiguration is for testing only? Should be @visiblefortesting only or a comment at least? HTraceConfiguration is used by SpanReceiver implementation, not for testing only. bq. Should there be defines for a few of these? "DFSInputStream.close" seems fine... only used once DFSInputStream.read? I think it is fine not to define "DFSInputStream.read" now. There are some fixes in addition to above such as, * removed timing dependency from TestTracing. * added guard by Trace.isTracing() around startSpan() in DFSInputStream, FsShell and WritableRpcEngine. * removed SpanReceiverHost from FsShell and DFSClient. I will add options or config properties to turn on tracing from shell later on another JIRA issue. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace > d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-6.patch Here's a patch that adds annotations for DFSInputStream.seek > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-5.patch > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, Zipkin Trace > a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: Zipkin Trace d0f0d66b8a258a69.png Another example > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, Zipkin Trace a06e941b0172ec73.png, > Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-4.patch Here's a lot more rigorous testing. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, Zipkin Trace a06e941b0172ec73.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-3.patch Fix for tests. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, Zipkin Trace a06e941b0172ec73.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-2.patch Instrumented Sender and Receiver (Though some of those code paths are not hit as well). better read side instrumentation. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > Zipkin Trace a06e941b0172ec73.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: Zipkin Trace a06e941b0172ec73.png Here's an example of what I have currently. I'm still trying to balance what should be instrumented. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, Zipkin Trace > a06e941b0172ec73.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-1.patch WIP path. This one has testing for the read and write paths started. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Affects Version/s: 2.1.1-beta Status: Patch Available (was: Open) > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-0.patch Here's an initial implementation of the tracing. Some more annotations and instrumentation could be added if needed. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
