[
https://issues.apache.org/jira/browse/HDDS-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16872:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: Report EC read bytes and decode time in filesystem statistics
> -----------------------------------------------------------------
>
> Key: HDDS-16872
> URL: https://issues.apache.org/jira/browse/HDDS-16872
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> {\{OzoneFSInputStream}} only calls
> \{{FileSystem.Statistics.incrementBytesRead}}.
> Engines and users that read EC data cannot see, per filesystem or per stream:
> * how many bytes were read from EC keys;
> * how much time was spent decoding;
> * how many blocks were read in reconstruction mode.
> The EC reconstruction counters that exist today are client-wide, in
> {\{XceiverClientMetrics}}.
> h2. HDFS behavior
> {\{DFSStripedInputStream}} adds EC decoding time to \{{ReadStatistics}}, and
> {\{DFSClient.updateFileSystemECReadStats}} calls
> {\{FileSystem.Statistics.incrementBytesReadErasureCoded}}.
> h2. Proposal
> * Call \{{incrementBytesReadErasureCoded}} for reads of EC keys. That method
> exists only in Hadoop 3, so put the call in the Hadoop 3 filesystem path (for
> example \{{CapableOzoneFSInputStream}}) or guard it, so \{{ozonefs-hadoop2}}
> keeps working. The stream needs to know the key is EC, which the key lookup
> that opens it already knows.
> * Collect per-stream EC counters in the client (bytes decoded, decode time,
> blocks read in reconstruction mode, failovers) and expose them on the
> filesystem input stream as \{{IOStatistics}} (\{{IOStatisticsSource}}). That
> interface is Hadoop 3.3+, so check the oldest Hadoop 3 runtime the
> filesystem jar has to load on and guard the class if needed.
> * Keep the existing \{{XceiverClientMetrics}} counters unchanged.
> h2. Compatibility
> Client only. No wire or data change. New statistics only; existing ones keep
> their names and values.
> h2. Testing
> * Unit: reading an EC key increments \{{bytesRead}} and
> {\{bytesReadErasureCoded}} by the same amount; a replicated key leaves
> {\{bytesReadErasureCoded}} at 0.
> * Unit: a degraded read reports non-zero decode time and the reconstruction
> block count in the stream's \{{IOStatistics}}.
> * The \{{ozonefs-hadoop2}} build and its tests keep passing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
