[jira] [Commented] (TEZ-3452) Auto-reduce parallelism calculation can overflow with large inputs
[
https://issues.apache.org/jira/browse/TEZ-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582893#comment-15582893
]
Jonathan Eagles commented on TEZ-3452:
--
[~mingma], can you have a look at the latest implementation based on BigInteger?
> Auto-reduce parallelism calculation can overflow with large inputs
> --
>
> Key: TEZ-3452
> URL: https://issues.apache.org/jira/browse/TEZ-3452
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3452.1.patch, TEZ-3452.2.patch, TEZ-3452.3.patch
>
>
> Overflow can occur when the numTasks is high (say 45000) and outputSize is
> high (say 311TB) and slow start is set to 1.0.
> {code:title=ShuffleVertexManager}
> for (Map.Entry vInfo : getBipartiteInfo()) {
> SourceVertexInfo srcInfo = vInfo.getValue();
> if (srcInfo.numTasks > 0 && srcInfo.numVMEventsReceived > 0) {
> // this assumes that 1 vmEvent is received per completed task -
> TEZ-2961
> expectedTotalSourceTasksOutputSize +=
> (srcInfo.numTasks * srcInfo.outputSize) /
> srcInfo.numVMEventsReceived;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3419: Attachment: TEZ-3419.5.patch Thanks [~hitesh] Attaching a fresh patch with the comments addressed. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, > TEZ-3419.3.patch, TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, > Tez data missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-3419 PreCommit Build #2041
Jira: https://issues.apache.org/jira/browse/TEZ-3419
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2041/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4824 lines...]
[INFO] Tez SUCCESS [ 0.026 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56:40 min
[INFO] Finished at: 2016-10-17T22:34:36+00:00
[INFO] Final Memory: 85M/1451M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12833806/TEZ-3419.5.patch
against master revision 48208dc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2041//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
d69b56fd95ba9b7c754bac0c4266d92f5cd5ad3b logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3419
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[
https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583718#comment-15583718
]
TezQA commented on TEZ-3419:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12833806/TEZ-3419.5.patch
against master revision 48208dc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2041//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//console
This message is automatically generated.
> Tez UI: Applications page shows error, for users with only DAG level ACL
> permission.
>
>
> Key: TEZ-3419
> URL: https://issues.apache.org/jira/browse/TEZ-3419
> Project: Apache Tez
> Issue Type: Sub-task
>Affects Versions: 0.7.0
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot
> 2016-10-13 at 4.37.09 PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch,
> TEZ-3419.3.patch, TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch,
> Tez data missing.png, YARN & Tez data missing.png, YARN data missing.png
>
>
> Follow this logic and display better message:
> On loading app details page, send a request to
> /ws/v1/timeline/TEZ_APPLICATION/tez_
> - If it succeed, display the details page as we do now.
> - If it fails, send a request to
> /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A
> -- If it succeed, then we know that DAGs under the app are available and
> assume that the user doesn't have permission to access app level data.
> --- If AHS is accessible, display application data from there in the details
> page.
> --- else if AHS is not accessible, display a message in app details tab,
> something like "Data is not available. Check if you are authorized to access
> application data!".
> --- Also display the DAGs tab, for the user to see DAGs under that app.
> -- If it fails, display error message as we do now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583809#comment-15583809 ] Hitesh Shah edited comment on TEZ-3419 at 10/17/16 11:17 PM: - - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now ( as compared to an error message indicating no data or permission issue ) but just wanted to make sure that this was the intention of the patch and not an accidental change. was (Author: hitesh): - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now but just wanted to make sure that this was the intention of the patch and not an accidental change. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, Screen Shot 2016-10-17 at 4.11.29 PM.png, > Screen Shot 2016-10-17 at 4.11.59 PM.png, Screen Shot 2016-10-17 at 4.12.23 > PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, TEZ-3419.3.patch, > TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, Tez data > missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3419: - Attachment: Screen Shot 2016-10-17 at 4.12.23 PM.png Screen Shot 2016-10-17 at 4.11.59 PM.png Screen Shot 2016-10-17 at 4.11.29 PM.png - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now but just wanted to make sure that this was the intention of the patch and not an accidental change. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, Screen Shot 2016-10-17 at 4.11.29 PM.png, > Screen Shot 2016-10-17 at 4.11.59 PM.png, Screen Shot 2016-10-17 at 4.12.23 > PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, TEZ-3419.3.patch, > TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, Tez data > missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3469) Tez UI: Bump Phantom JS version to 2.1.1
[ https://issues.apache.org/jira/browse/TEZ-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583952#comment-15583952 ] Hitesh Shah commented on TEZ-3469: -- +1 > Tez UI: Bump Phantom JS version to 2.1.1 > > > Key: TEZ-3469 > URL: https://issues.apache.org/jira/browse/TEZ-3469 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3469.1.patch > > > Phantom JS have two parts - the original binary and a wrapper that interfaces > the binary and node world. > - Currently we use PhantomJS wrapper v1.9.19, which internally uses v1.9.8 of > PhantomJS build for running UTs. > - This ticket is to upgrade the wrapper package to v2.1.13, and thus use > PhantomJS 2.1.1 for running UTs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3458) Auto grouping for cartesian product edge(unpartitioned case)
[ https://issues.apache.org/jira/browse/TEZ-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3458: -- Summary: Auto grouping for cartesian product edge(unpartitioned case) (was: Auto reduce for cartesian product edge(unpartitioned case)) > Auto grouping for cartesian product edge(unpartitioned case) > > > Key: TEZ-3458 > URL: https://issues.apache.org/jira/browse/TEZ-3458 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3458.1.patch > > > Original CartesianProductVertexManagerUnpartitioned set parallelism as > product of all source vertices parallelism which may explode to insane > number. We should do auto reduce as in ShuffleVertexManager to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3458) Auto grouping for cartesian product edge(unpartitioned case)
[ https://issues.apache.org/jira/browse/TEZ-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3458: -- Attachment: TEZ-3458.1.patch > Auto grouping for cartesian product edge(unpartitioned case) > > > Key: TEZ-3458 > URL: https://issues.apache.org/jira/browse/TEZ-3458 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3458.1.patch > > > Original CartesianProductVertexManagerUnpartitioned set parallelism as > product of all source vertices parallelism which may explode to insane > number. We should do auto reduce as in ShuffleVertexManager to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
