[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503701#comment-16503701 ] Prasanth Jayachandran commented on TEZ-3331: TestTaskScheduler seems to be crashing in master as well. So I guess its not related to this patch. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch, TEZ-3331.8.patch, > TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, > TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503119#comment-16503119
]
TezQA commented on TEZ-3331:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12926702/TEZ-3331.8.patch
against master revision 09102e5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.rm.TestTaskScheduler
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2832//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2832//console
This message is automatically generated.
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Hitesh Shah
>Priority: Major
> Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch, TEZ-3331.8.patch,
> TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch,
> TEZ-3331.wip.5.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503032#comment-16503032
]
TezQA commented on TEZ-3331:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12926689/TEZ-3331.7.patch
against master revision 09102e5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.metrics.TestFileSystemStatisticUpdater2
The following test timeouts occurred in :
org.apache.tez.dag.app.rm.TestTaskScheduler
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2831//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2831//console
This message is automatically generated.
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Hitesh Shah
>Priority: Major
> Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch,
> TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch,
> TEZ-3331.wip.5.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502933#comment-16502933 ] Prasanth Jayachandran commented on TEZ-3331: Fixed failing test TestTaskExecution2. TestUnorderedPartitionedKVWriter is timeout issue and is passing on my laptop. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.7.patch, > TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, > TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502715#comment-16502715
]
TezQA commented on TEZ-3331:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12926663/TEZ-3331.6.patch
against master revision 09102e5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.task.TestTaskExecution2
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
The following test timeouts occurred in :
org.apache.tez.dag.app.rm.TestTaskScheduler
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2829//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2829//console
This message is automatically generated.
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Hitesh Shah
>Priority: Major
> Attachments: TEZ-3331.6.patch, TEZ-3331.wip.2.patch,
> TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch,
> TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502663#comment-16502663 ] Prasanth Jayachandran commented on TEZ-3331: Rebased patch. Just removed the changes to root pom.xml from .5 patch which changes hadoop version. Since master is already at hadoop 3.0.2 we no longer required the root pom.xml changes. [~EricWohlstadter] / [~gopalv] can someone please review and commit this patch? > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah >Priority: Major > Attachments: TEZ-3331.6.patch, TEZ-3331.wip.2.patch, > TEZ-3331.wip.3.patch, TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, > TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425441#comment-15425441 ] Hitesh Shah commented on TEZ-3331: -- Missed commenting on this earlier. Yes - we need to shim this out. For the counter removal - yes - they probably do need to be but are being used in the analyzers. I think they were accidentally added in one of the analyzer patches. Would need some minor changes there to remove the dependency on this enum and then remove the counters. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Hitesh Shah > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379771#comment-15379771
]
Ming Ma commented on TEZ-3331:
--
[~hitesh], thanks for the explanation. So to make this patch compiled against
the default version of hadoop, we need to refactor the reference of
{{org.apache.hadoop.fs.StorageStatistics.CommonStatisticNames}} from
{{FileSystemCounter}} to {{HadoopShim28}}?
Also wonder if the following counters can be removed from {{FileSystemCounter}}
given {{FileSystemCounterGroup}} will take care of expanding FileSystemCounter
name to {{FileSystemScheme_FileSystemCounterName}} counters.
{noformat}
HDFS_BYTES_READ,
HDFS_BYTES_WRITTEN,
FILE_BYTES_READ,
FILE_BYTES_WRITTEN
{noformat}
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Hitesh Shah
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch,
> TEZ-3331.wip.4.patch, TEZ-3331.wip.5.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373981#comment-15373981 ] Hitesh Shah commented on TEZ-3331: -- Also, TEZ-3168 has a wip patch that shows how the shims could be enhanced to make use of an API not in the default version of hadoop that we compile against. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373947#comment-15373947 ] Ming Ma commented on TEZ-3331: -- [~hitesh] thanks for the info about hadoop shim. bq. Mind adding more details on which features in particular? I have opened TEZ-3340, TEZ-3341, TEZ-3342 and followed up on [~sseth]'s email thread about release. Do you know if hadoop shim can supports additions of these features? > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.4.patch, TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370185#comment-15370185 ] Hitesh Shah commented on TEZ-3331: -- bq. Maybe the next major release Tez can switch from hadoop 2.6 to hadoop 2.8? This is probably something better to discuss on the dev@ list. However, IMHO, retaining a minimum dependency on 2.6 and making use of new features via shims where possible would be a better approach. Most users tend not to jump to a newer version straight away and .0 releases tend to be problematic to say the least in terms of stability. > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370181#comment-15370181 ] Hitesh Shah commented on TEZ-3331: -- bq. In addition, there are several other 2.8 YARN and HDFS features Tez can benefit from [~mingma] Mind adding more details on which features in particular? > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368605#comment-15368605 ] Ming Ma commented on TEZ-3331: -- For support this, it seems Tez needs to change its dependency to hadoop 2.8. In addition, there are several other 2.8 YARN and HDFS features Tez can benefit from. Maybe the next major release Tez can switch from hadoop 2.6 to hadoop 2.8? > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
