[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated TEZ-3331: --- Attachment: TEZ-3331.8.patch > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch, TEZ-3331.8.patch, > TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, > TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated TEZ-3331: --- Attachment: TEZ-3331.7.patch > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch, > TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, > TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated TEZ-3331: --- Attachment: TEZ-3331.6.patch > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.wip.2.patch, > TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, > TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-3331: Assignee: Hitesh Shah > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3331: - Attachment: TEZ-3331.wip.5.patch Fix typo in test file. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3331: - Attachment: TEZ-3331.wip.4.patch Some tests added. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3331: - Attachment: TEZ-3331.wip.3.patch Removed use of isTracked() > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-3331:
-
Attachment: TEZ-3331.wip.2.patch
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3331: - Attachment: TEZ-3331.wip.patch [~mingma] Agreed that this feature requires a 2.8.0 release but it is possible to try and add it in via a shim which can use the existing functionality in versions below 2.8.0 and switch to the new stats tracker for 2.8.0 onwards. Attached a wip patch for the potential changes. This has not been shimmed out yet as the current shim approach does not work as the shim impl may need access to tez counters and the enum changes probably need to be thought out a bit better to provide translations to the op name via a shim too. \cc [~sseth] > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated TEZ-3331: -- Description: Hadoop has added several operation specific counters in the FileSystem statistics (HADOOP-13065). These counters are useful to track file system operations more granularly. It would be great to track these counters for Tez and expose them via UI as well. (was: Hadoop has added several operation specific counters in the FileSystem statistics. These counters are useful to track file system operations more granularly. It would be great to track these counters for Tez and expose them via UI as well.) > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
