[
https://issues.apache.org/jira/browse/TEZ-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775320#comment-17775320
]
Ayush Saxena commented on TEZ-4451:
-----------------------------------
Sounds cool, I checked the Hadoop code, it does log the FileSystem statistics
on every FS close, based on a config. I was thinking we could have logged the
entire {{TezChild}} statistics somewhere here:
[https://github.com/apache/tez/blob/master/tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java#L319]
It does {{{}FileSystem.closeAllForUGI(childUGI);{}}}, So, an aggregated
statistics per {{TezChild}} could have been logged. But I think there isn't
anything like that, unless I coin a util at Tez, which gets all the instances &
then do an aggregate, I think FS doesn't expose all FS per ugi either....
The stream one I think I can get it done. This can be put in
{{TaskRunner2Callable}} in finally block.
{code:java}
String ioStats =
IOStatisticsContext.getCurrentIOStatisticsContext().snapshot().toString();
if (StringUtils.isNotEmpty(ioStats)) {
LOG.info("TaskAttemptId={}, IOStatistics={}", task.getTaskAttemptID(),
ioStats);
}
{code}
Maybe cover it with a conf, if it is costly. I will take this up in a few days
unless anyone else wants to take it.
> ThreadLevel IO Stats Support for TEZ
> ------------------------------------
>
> Key: TEZ-4451
> URL: https://issues.apache.org/jira/browse/TEZ-4451
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harshit Gupta
> Priority: Major
>
> Dump IO Statistics for each of the tasks in the log.
> This will requires upgrading Tez to use Hadoop-3.3.9-SNAPSHOT
>
> cc: [~rbalamohan] [~abstractdog] [~mthakur]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
