[ https://issues.apache.org/jira/browse/TEZ-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743248#comment-17743248 ]
Steve Loughran commented on TEZ-4451: ------------------------------------- you can log filesystem stats in filesystem close for abfs and s3a. what the stream now does is let you collect aggregate stream statistics for each thread, so if you do different work in different threads, you can collect the isolated work. if you want all work for the entire life of a FileSystem instance, that is much easier. FileSystem.getIOStatistics() will return io stats or null; you can create a snapshot of that which can be marshalled as a java serializable or to json and back. enjoy {code} IOStatisticsSupport.snapshotIOStatistics(FileSystem..getIOStatistics()) {code} > ThreadLevel IO Stats Support for TEZ > ------------------------------------ > > Key: TEZ-4451 > URL: https://issues.apache.org/jira/browse/TEZ-4451 > Project: Apache Tez > Issue Type: Improvement > Reporter: Harshit Gupta > Priority: Major > > Dump IO Statistics for each of the tasks in the log. > This will requires upgrading Tez to use Hadoop-3.3.9-SNAPSHOT > > cc: [~rbalamohan] [~abstractdog] [~mthakur] -- This message was sent by Atlassian Jira (v8.20.10#820010)