[
https://issues.apache.org/jira/browse/TEZ-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743248#comment-17743248
]
Steve Loughran commented on TEZ-4451:
-------------------------------------
you can log filesystem stats in filesystem close for abfs and s3a. what the
stream now does is let you collect aggregate stream statistics for each thread,
so if you do different work in different threads, you can collect the isolated
work.
if you want all work for the entire life of a FileSystem instance, that is much
easier. FileSystem.getIOStatistics() will return io stats or null; you can
create a snapshot of that which can be marshalled as a java serializable or to
json and back. enjoy
{code}
IOStatisticsSupport.snapshotIOStatistics(FileSystem..getIOStatistics())
{code}
> ThreadLevel IO Stats Support for TEZ
> ------------------------------------
>
> Key: TEZ-4451
> URL: https://issues.apache.org/jira/browse/TEZ-4451
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harshit Gupta
> Priority: Major
>
> Dump IO Statistics for each of the tasks in the log.
> This will requires upgrading Tez to use Hadoop-3.3.9-SNAPSHOT
>
> cc: [~rbalamohan] [~abstractdog] [~mthakur]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
