[jira] [Comment Edited] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371550#comment-15371550
]
Hitesh Shah edited comment on TEZ-3331 at 7/11/16 8:32 PM:
---
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls if we can make the group name a bit more understandable.
was (Author: hitesh):
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls.
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371550#comment-15371550
]
Hitesh Shah edited comment on TEZ-3331 at 7/11/16 8:32 PM:
---
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls.
was (Author: hitesh):
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368605#comment-15368605 ] Ming Ma edited comment on TEZ-3331 at 7/11/16 3:32 AM: --- To support this, it seems Tez needs to change its dependency to hadoop 2.8. In addition, there are several other 2.8 YARN and HDFS features Tez can benefit from. Maybe the next major release Tez can switch from hadoop 2.6 to hadoop 2.8? was (Author: mingma): For support this, it seems Tez needs to change its dependency to hadoop 2.8. In addition, there are several other 2.8 YARN and HDFS features Tez can benefit from. Maybe the next major release Tez can switch from hadoop 2.6 to hadoop 2.8? > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
