[jira] [Commented] (TEZ-3274) Vertex with MRInput and broadcast input does not respect slow start
[
https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061457#comment-16061457
]
Jonathan Eagles commented on TEZ-3274:
--
I think this approach is correct given the comments above regarding making
minimal changes and leaving ShuffleVertexManger as is.
A few minor comments on the code ported from ShuffleVertexManager
{code:title=RootInputVertexManager.java#L68}
public static final float
TEZ_ROOT_INPUT_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT = 0.951f;
public static final float
TEZ_ROOT_INPUT_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT = 0.951f;
{code}
// differs from the ShuffleVertexManager value of 0.25f and 0.75f respectively
{code:title=RootInputVertexManager.java#L231}
public void initialize() {
conf = new Configuration();
{code}
// conf is double initialized here and below. This one can be removed
[~ebadger], there are a number of other style/design comments that are
artifacts of copying the code from the ShuffleVertexManager that should be
cleaned up. I'll file a separate JIRA to address those issues at that time.
> Vertex with MRInput and broadcast input does not respect slow start
> ---
>
> Key: TEZ-3274
> URL: https://issues.apache.org/jira/browse/TEZ-3274
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Eric Badger
> Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch,
> TEZ-3274.003.patch, TEZ-3274.004.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and
> not ShuffleVertexManager) and start containers and tasks immediately. In this
> scenario, resources can be wasted since they do not respect
> tez.shuffle-vertex-manager.min-src-fraction
> tez.shuffle-vertex-manager.max-src-fraction.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3771) Tez UI: WASB/ADLS counters should be listed on the Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061338#comment-16061338
]
TezQA commented on TEZ-3771:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874281/TEZ-3771.2.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2541//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2541//console
This message is automatically generated.
> Tez UI: WASB/ADLS counters should be listed on the Tez UI
> --
>
> Key: TEZ-3771
> URL: https://issues.apache.org/jira/browse/TEZ-3771
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3771.1.patch, TEZ-3771.2.patch
>
>
> Under Group : org.apache.tez.common.counters.FileSystemCounter
> Counter names : WASB_BYTES_READ, WASB_BYTES_WRITTEN, ADL_BYTES_READ and
> ADL_BYTES_WRITTEN must be added
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3771 PreCommit Build #2541
Jira: https://issues.apache.org/jira/browse/TEZ-3771
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2541/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 339.45 KB...]
[INFO]
[INFO] Total time: 01:01 h
[INFO] Finished at: 2017-06-23T18:16:26Z
[INFO] Final Memory: 79M/1254M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874281/TEZ-3771.2.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2541//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2541//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
d437c361634890fca75cfaf7de83b2b0ed8b80e6 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3771) Tez UI: WASB/ADLS counters should be listed on the Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3771: Attachment: TEZ-3771.2.patch There was an object reference issue while trying to add counters from the config.env. Attaching a fresh patch with a dependency up-gradation to fix the same. Guess the patch can be submitted shortly. > Tez UI: WASB/ADLS counters should be listed on the Tez UI > -- > > Key: TEZ-3771 > URL: https://issues.apache.org/jira/browse/TEZ-3771 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3771.1.patch, TEZ-3771.2.patch > > > Under Group : org.apache.tez.common.counters.FileSystemCounter > Counter names : WASB_BYTES_READ, WASB_BYTES_WRITTEN, ADL_BYTES_READ and > ADL_BYTES_WRITTEN must be added -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3771) Tez UI: WASB/ADLS counters should be listed on the Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060781#comment-16060781 ] Sreenath Somarajapuram commented on TEZ-3771: - bq. Actually, Is there a way where a site deployment can add specific fields. Yes. That can be done via the config/configs.env file. > Tez UI: WASB/ADLS counters should be listed on the Tez UI > -- > > Key: TEZ-3771 > URL: https://issues.apache.org/jira/browse/TEZ-3771 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3771.1.patch > > > Under Group : org.apache.tez.common.counters.FileSystemCounter > Counter names : WASB_BYTES_READ, WASB_BYTES_WRITTEN, ADL_BYTES_READ and > ADL_BYTES_WRITTEN must be added -- This message was sent by Atlassian JIRA (v6.4.14#64029)
