[
https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888032#action_12888032
]
Suresh Srinivas commented on HADOOP-6859:
-----------------------------------------
I am planning to introduce the following additional statistics, that is
accumulated at the client per file system as it is done now.
# read operations - number of read operations such as listStatus,
getFileBlockLocations, open etc.
# write operations - number of write operations such as create, append,
setPermission etc.
# large read operations - on file system, most of the operations are small
except listFiles for a large directory. Iterative listFiles was introduced in
HDFS to break down a single large operation into smaller steps. This counter is
incremented is incremented for every iteration of listFiles, when listing files
under a large directory.
These statistics are collected in job history for analysis of how HDFS is
loaded by map reduce tasks. This is useful in the interim to identify jobs that
heavily load HDFS. In future this could also be used to throttle the load at
the map reduce framework.
> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
> Key: HADOOP-6859
> URL: https://issues.apache.org/jira/browse/HADOOP-6859
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional
> statistics that gives summary of operations performed will be useful for
> tracking file system use.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
