steveloughran commented on pull request #2069:
URL: https://github.com/apache/hadoop/pull/2069#issuecomment-661077889
The latest update records the min/max and mean times to initiate the
(long-haul) http get request
```
2020-07-17 17:56:30,433 [JUnit] INFO scale.ITestS3AInputStreamPerformance
(ITestS3AInputStreamPerformance.java:dumpIOStatistics(135)) -
Aggregate Stream Statistics counters=((stream_aborted=2)
(stream_read_bytes=47870126) (stream_read_bytes_backwards_on_seek=12713984)
(stream_read_bytes_discarded_in_abort=43889622)
(stream_read_bytes_read_in_close=252395)
(stream_read_bytes_skipped_on_seek=55054163)
(stream_read_close_operations=0) (stream_read_closed=12)
(stream_read_exceptions=0) (stream_read_fully_operations=8)
(stream_read_opened=14) (stream_read_operations=3415)
(stream_read_operations_incomplete=3362)
(stream_read_seek_backward_operations=4)
(stream_read_seek_bytes_read=45092691)
(stream_read_seek_forward_operations=175) (stream_read_seek_operations=179)
(stream_read_seek_policy_changed=8) (stream_read_total_bytes=93215212)
(stream_read_version_mismatches=0));
gauges=((stream_read_gauge_input_policy=6));
minimums=((op_http_get_request.min=29));
maximums=((op_http_get_request.max=753));
means=((op_http_get_request.mean=MeanStatistic{sum=2420, samples=14,
mean=172.85714285714286}));
```
S3A also collects it for listings, and pass that all the way back through
LocationStatusFetcher
```
2020-07-20 14:48:40,563 [JUnit-testLocatedFileStatusFourThreads[raw]] INFO
s3a.ITestLocatedFileStatusFetcher
(ITestLocatedFileStatusFetcher.java:assertListCount(184))
- Statistics of fetcher: counters=((op_http_list_request=4)); gauges=();
minimums=((op_http_list_request.min=29));
maximums=((op_http_list_request.max=114));
means=((op_http_list_request.mean=sum=274, samples=4, mean=68.500000));
```
also goes through LineReader and the codec in/out streams.
This means that applications using the MR classes can now ask for FS
performance values.
1. o.a.h.fs.statistics API is ready for review; this is the bit we need to
keep stable.
2. o.a.h.fs.statistics.impl is also up for looking at. This is where we can
be agile about change, but we should still look for obvious issues.
3. S3A stats migration is complete. It's a big part of this patch, but can
be reviewed independently. It's just driven the work, especially the .impl
package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
