[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259848#comment-17259848 ] Steve Loughran commented on HADOOP-13845: - HADOOP-17271 does this for HEAD and LIST requests, plus time for the GET to start. Anything O(data), O(objects) etc is tricky > s3a to instrument duration of HTTP calls > > > Key: HADOOP-13845 > URL: https://issues.apache.org/jira/browse/HADOOP-13845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13844 proposes pulling out the swift duration classes for reuse; this > patch proposes instrumenting s3a with it. > One interesting question: what to do with the values. For now, they could > just be printed, but it might be interesting to include in FS stats collected > at the end of a run. However, those are all assumed to be simple counters > where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109776#comment-16109776 ] Steve Loughran commented on HADOOP-13845: - Like your thoughts about live viewing. something like this, though of course it'll make the code even more complex. If you are looking at improving metrics in S3A, how about taking a look at what's mentioned in HADOOP-13551: working out how to fetch the SDK internal metrics and pump them out. The SDK team have done stuff internally, and it seems good to get at all this stuff before we move to instrumenting our own code more. They're working at a lower level than we can instrument, so could do things we can't (and vice-versa). > s3a to instrument duration of HTTP calls > > > Key: HADOOP-13845 > URL: https://issues.apache.org/jira/browse/HADOOP-13845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13844 proposes pulling out the swift duration classes for reuse; this > patch proposes instrumenting s3a with it. > One interesting question: what to do with the values. For now, they could > just be printed, but it might be interesting to include in FS stats collected > at the end of a run. However, those are all assumed to be simple counters > where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[
https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098416#comment-16098416
]
Yonger commented on HADOOP-13845:
-
Does this make sense?
{code:java}
Duration duration = new Duration();
ObjectListing listing= s3.listObjects(request);
duration.finished();
durationStats.add(method.getName()+" " + reason, duration, success);
return listing;
{code}
> s3a to instrument duration of HTTP calls
>
>
> Key: HADOOP-13845
> URL: https://issues.apache.org/jira/browse/HADOOP-13845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> HADOOP-13844 proposes pulling out the swift duration classes for reuse; this
> patch proposes instrumenting s3a with it.
> One interesting question: what to do with the values. For now, they could
> just be printed, but it might be interesting to include in FS stats collected
> at the end of a run. However, those are all assumed to be simple counters
> where merging is a matter of addition. These are more metrics
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095706#comment-16095706 ] Yonger commented on HADOOP-13845: - Another usage is push http duration into database(such as influxdb), and get a realtime show via grafana. that you can know the duration of one kind(path) of request when is longest when suffering some bottleneck, and then you can do trouble shooting work in the range of higher duration time. which can not be done if you just only output the final result in FS stats. And what's going on for HADOOP-13844? i expect to put http duration into s3a metrics. > s3a to instrument duration of HTTP calls > > > Key: HADOOP-13845 > URL: https://issues.apache.org/jira/browse/HADOOP-13845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13844 proposes pulling out the swift duration classes for reuse; this > patch proposes instrumenting s3a with it. > One interesting question: what to do with the values. For now, they could > just be printed, but it might be interesting to include in FS stats collected > at the end of a run. However, those are all assumed to be simple counters > where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
