[jira] [Commented] (TEZ-3452) Auto-reduce parallelism calculation can overflow with large inputs
[ https://issues.apache.org/jira/browse/TEZ-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582893#comment-15582893 ] Jonathan Eagles commented on TEZ-3452: -- [~mingma], can you have a look at the latest implementation based on BigInteger? > Auto-reduce parallelism calculation can overflow with large inputs > -- > > Key: TEZ-3452 > URL: https://issues.apache.org/jira/browse/TEZ-3452 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3452.1.patch, TEZ-3452.2.patch, TEZ-3452.3.patch > > > Overflow can occur when the numTasks is high (say 45000) and outputSize is > high (say 311TB) and slow start is set to 1.0. > {code:title=ShuffleVertexManager} > for (Map.Entry vInfo : getBipartiteInfo()) { > SourceVertexInfo srcInfo = vInfo.getValue(); > if (srcInfo.numTasks > 0 && srcInfo.numVMEventsReceived > 0) { > // this assumes that 1 vmEvent is received per completed task - > TEZ-2961 > expectedTotalSourceTasksOutputSize += > (srcInfo.numTasks * srcInfo.outputSize) / > srcInfo.numVMEventsReceived; > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3419: Attachment: TEZ-3419.5.patch Thanks [~hitesh] Attaching a fresh patch with the comments addressed. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, > TEZ-3419.3.patch, TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, > Tez data missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-3419 PreCommit Build #2041
Jira: https://issues.apache.org/jira/browse/TEZ-3419 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2041/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4824 lines...] [INFO] Tez SUCCESS [ 0.026 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 56:40 min [INFO] Finished at: 2016-10-17T22:34:36+00:00 [INFO] Final Memory: 85M/1451M [INFO] {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12833806/TEZ-3419.5.patch against master revision 48208dc. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. d69b56fd95ba9b7c754bac0c4266d92f5cd5ad3b logged out == == Finished build. == == Archiving artifacts [description-setter] Description set: TEZ-3419 Recording test results Email was triggered for: Success Sending email for trigger: Success ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583718#comment-15583718 ] TezQA commented on TEZ-3419: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12833806/TEZ-3419.5.patch against master revision 48208dc. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2041//console This message is automatically generated. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, > TEZ-3419.3.patch, TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, > Tez data missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583809#comment-15583809 ] Hitesh Shah edited comment on TEZ-3419 at 10/17/16 11:17 PM: - - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now ( as compared to an error message indicating no data or permission issue ) but just wanted to make sure that this was the intention of the patch and not an accidental change. was (Author: hitesh): - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now but just wanted to make sure that this was the intention of the patch and not an accidental change. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, Screen Shot 2016-10-17 at 4.11.29 PM.png, > Screen Shot 2016-10-17 at 4.11.59 PM.png, Screen Shot 2016-10-17 at 4.12.23 > PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, TEZ-3419.3.patch, > TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, Tez data > missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3419: - Attachment: Screen Shot 2016-10-17 at 4.12.23 PM.png Screen Shot 2016-10-17 at 4.11.59 PM.png Screen Shot 2016-10-17 at 4.11.29 PM.png - 2 screenshots show spurious data being shown in the UI. - 3rd screenshot is for the configs. Configs are not accessible due to permission issues but UI says no records found. I think this is a reasonable approach for now but just wanted to make sure that this was the intention of the patch and not an accidental change. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: Screen Shot 2016-10-13 at 4.25.31 PM.png, Screen Shot > 2016-10-13 at 4.37.09 PM.png, Screen Shot 2016-10-17 at 4.11.29 PM.png, > Screen Shot 2016-10-17 at 4.11.59 PM.png, Screen Shot 2016-10-17 at 4.12.23 > PM.png, TEZ-3419.1.patch, TEZ-3419.2.patch, TEZ-3419.3.patch, > TEZ-3419.4.patch, TEZ-3419.5.patch, TEZ-3419.wip.1.patch, Tez data > missing.png, YARN & Tez data missing.png, YARN data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3469) Tez UI: Bump Phantom JS version to 2.1.1
[ https://issues.apache.org/jira/browse/TEZ-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583952#comment-15583952 ] Hitesh Shah commented on TEZ-3469: -- +1 > Tez UI: Bump Phantom JS version to 2.1.1 > > > Key: TEZ-3469 > URL: https://issues.apache.org/jira/browse/TEZ-3469 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3469.1.patch > > > Phantom JS have two parts - the original binary and a wrapper that interfaces > the binary and node world. > - Currently we use PhantomJS wrapper v1.9.19, which internally uses v1.9.8 of > PhantomJS build for running UTs. > - This ticket is to upgrade the wrapper package to v2.1.13, and thus use > PhantomJS 2.1.1 for running UTs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3458) Auto grouping for cartesian product edge(unpartitioned case)
[ https://issues.apache.org/jira/browse/TEZ-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3458: -- Summary: Auto grouping for cartesian product edge(unpartitioned case) (was: Auto reduce for cartesian product edge(unpartitioned case)) > Auto grouping for cartesian product edge(unpartitioned case) > > > Key: TEZ-3458 > URL: https://issues.apache.org/jira/browse/TEZ-3458 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3458.1.patch > > > Original CartesianProductVertexManagerUnpartitioned set parallelism as > product of all source vertices parallelism which may explode to insane > number. We should do auto reduce as in ShuffleVertexManager to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3458) Auto grouping for cartesian product edge(unpartitioned case)
[ https://issues.apache.org/jira/browse/TEZ-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3458: -- Attachment: TEZ-3458.1.patch > Auto grouping for cartesian product edge(unpartitioned case) > > > Key: TEZ-3458 > URL: https://issues.apache.org/jira/browse/TEZ-3458 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3458.1.patch > > > Original CartesianProductVertexManagerUnpartitioned set parallelism as > product of all source vertices parallelism which may explode to insane > number. We should do auto reduce as in ShuffleVertexManager to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)