[
https://issues.apache.org/jira/browse/HADOOP-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684040#comment-17684040
]
Viraj Jasani commented on HADOOP-18617:
---------------------------------------
{quote}Ideally we should actually move the IOStatisticsStore interface into
org.apache.hadoop.fs.statistics and the builder to match -but we can't do that
without causing trauma elsewhere (google gcs).
Strategy there: Add a new interface IOStatisticsCollector in .impl which is
then implemented by IOStatisticsStore, and a new builder API which forwards to
IOStatisticsStoreBuilder.
{quote}
If we do this
* Create new interface IOStatisticsCollector in .impl
* Move interface IOStatisticsStore to org.apache.hadoop.fs.statistics
* Make interface IOStatisticsStore implement IOStatisticsCollector (which now
belongs to .impl)
We would essentially let an interface at *_xyz_* package implement another
interface from *_xyz.impl_* package. I wonder if this makes the structure look
a bit tricky.
> Make IOStatisticsStore and binding APIs public for use beyond our code
> ----------------------------------------------------------------------
>
> Key: HADOOP-18617
> URL: https://issues.apache.org/jira/browse/HADOOP-18617
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.3.5
> Reporter: Steve Loughran
> Priority: Major
>
> it's really useful to be able to collect iostats in things other than the FS
> classes -we do it in the S3A and manifest committers.
> But external code -such as the spark committers can't use the methods in
> {{org.apache.hadoop.fs.statistics.impl))
> Proposed
> Make some classes/interfaces public
> * IOStatisticsBinding
> * IOStatisticsStore
> * IOStatisticsStoreBuilder
> Ideally we should actually move the IOStatisticsStore interface into
> org.apache.hadoop.fs.statistics and the builder to match -but we can't do
> that without causing trauma elsewhere (google gcs).
> Strategy there: Add a new interface IOStatisticsCollector in .impl which is
> then implemented by IOStatisticsStore, and a new builder API which forwards
> to IOStatisticsStoreBuilder.
> Side issue: we don't make any use of the "clever, elegant functional" bit of
> DynamicIOStatisticsBuilder/DynamicIOStatistics, where every counter is mapped
> to a function which is then invoked to get at the atomic longs. It's used in
> IOStatisticsStoreImpl, but only with AtomicLong and MeanStatistic instances.
> If we just move to simple maps we will save on lambda-expressions and on
> lookup overhead. The original intent was something like coda hale metrics
> where we could add dynamic lookup to other bits of instrumentation; in
> practise we measure durations and build counts/min/max.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
