[jira] [Created] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Rajesh Balamohan (JIRA)
Rajesh Balamohan created TEZ-2834: - Summary: tez app hangs at large scale (~30TB) Key: TEZ-2834 URL: https://issues.apache.org/jira/browse/TEZ-2834 Project: Apache Tez Issue Type: Bug Aff

[jira] [Updated] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2834: -- Attachment: application_1442254312093_0095.1.log.gz application_1442254312093_0095.

[jira] [Updated] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2834: -- Description: Will attach the DAG. Repro for reference: TPC-DS q_70 @ 30 TB scale. "Map 7" comple

[jira] [Updated] (TEZ-2732) DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers

2015-09-16 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2732: - Affects Version/s: (was: 0.8.0-alpha) > DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort

[jira] [Updated] (TEZ-2732) DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers

2015-09-16 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2732: - Affects Version/s: 0.5.0 0.6.0 0.7.0 0.8

[jira] [Commented] (TEZ-2833) Dont create extra directory during ATS file download

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790762#comment-14790762 ] Bikas Saha commented on TEZ-2833: - Couldn't understand the scenario :) The file names are al

[jira] [Commented] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790778#comment-14790778 ] Bikas Saha commented on TEZ-2834: - If this cluster has latest YARN then the am logs can be s

[jira] [Commented] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790783#comment-14790783 ] Gopal V commented on TEZ-2834: -- [~bikassaha]: YARN-4149? That was fixed last night, it's not de

[jira] [Commented] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790819#comment-14790819 ] Bikas Saha commented on TEZ-2834: - The preemption code logs are all debug. This issue cannot

[jira] [Commented] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790823#comment-14790823 ] Bikas Saha commented on TEZ-2834: - Was the cluster fully occupied when this was happening. M

[jira] [Updated] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2834: Assignee: (was: Bikas Saha) > tez app hangs at large scale (~30TB) > -

[jira] [Assigned] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned TEZ-2834: --- Assignee: Siddharth Seth > tez app hangs at large scale (~30TB) > -

[jira] [Updated] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2834: Assignee: Bikas Saha (was: Siddharth Seth) > tez app hangs at large scale (~30TB) > -

[jira] [Assigned] (TEZ-2834) tez app hangs at large scale (~30TB)

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha reassigned TEZ-2834: --- Assignee: Bikas Saha > tez app hangs at large scale (~30TB) > > >

[jira] [Updated] (TEZ-2830) Backport TEZ-2774 to branch-0.7

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2830: Attachment: TEZ-2830.1.txt [~bikassaha] - could you please scan through the backport for sanity. > Ba

[jira] [Comment Edited] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790965#comment-14790965 ] Bikas Saha edited comment on TEZ-2774 at 9/16/15 7:09 PM: -- Attachin

[jira] [Updated] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2774: Attachment: TEZ-2774.addendum.patch Attaching an addendum patch that periodically logs in preemption related c

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790974#comment-14790974 ] Siddharth Seth commented on TEZ-2774: - Looks fine. > Reduce logging in the AM, and part

[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791022#comment-14791022 ] Bikas Saha commented on TEZ-2774: - Thanks! commit 1a065b9d87d84645363d0c65ae021a6a514169a8 A

[jira] [Updated] (TEZ-2830) Backport TEZ-2774 to branch-0.7

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2830: Attachment: TEZ-2830.2.txt Updated with the addendum to 2774 > Backport TEZ-2774 to branch-0.7 >

[jira] [Commented] (TEZ-2830) Backport TEZ-2774 to branch-0.7

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791099#comment-14791099 ] Bikas Saha commented on TEZ-2830: - lgtm. found one missing item. perhaps its not relevant to

[jira] [Updated] (TEZ-2826) save the status for completed dags in a session

2015-09-16 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2826: - Description: currently we store the list of dags completed. If we store the state of the dag too, it would

[jira] [Created] (TEZ-2835) [Timeline ACLs] Session-level entities should not be tied to the dag's domain

2015-09-16 Thread Hitesh Shah (JIRA)
Hitesh Shah created TEZ-2835: Summary: [Timeline ACLs] Session-level entities should not be tied to the dag's domain Key: TEZ-2835 URL: https://issues.apache.org/jira/browse/TEZ-2835 Project: Apache Tez

[jira] [Commented] (TEZ-2830) Backport TEZ-2774 to branch-0.7

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791209#comment-14791209 ] Siddharth Seth commented on TEZ-2830: - That's not relevant to branch-0.7, only for threa

[jira] [Created] (TEZ-2836) Avoid setting framework/system counters for tasks running in threads

2015-09-16 Thread Siddharth Seth (JIRA)
Siddharth Seth created TEZ-2836: --- Summary: Avoid setting framework/system counters for tasks running in threads Key: TEZ-2836 URL: https://issues.apache.org/jira/browse/TEZ-2836 Project: Apache Tez

[jira] [Updated] (TEZ-2836) Avoid setting framework/system counters for tasks running in threads

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2836: Attachment: TEZ-2836.1.txt [~rajesh.balamohan], [~hitesh] - please review. This disables the final up

[jira] [Resolved] (TEZ-2830) Backport TEZ-2774 to branch-0.7

2015-09-16 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved TEZ-2830. - Resolution: Fixed Fix Version/s: 0.7.1 > Backport TEZ-2774 to branch-0.7 > --

Failed: TEZ-2836 PreCommit Build #1145

2015-09-16 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-2836 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1145/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Commented] (TEZ-2836) Avoid setting framework/system counters for tasks running in threads

2015-09-16 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791435#comment-14791435 ] TezQA commented on TEZ-2836: {color:red}-1 overall{color}. Here are the results of testing the

[jira] [Updated] (TEZ-814) Improve heuristic for determining a task has failed outputs

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-814: --- Attachment: TEZ-814.1.patch > Improve heuristic for determining a task has failed outputs > -

[jira] [Assigned] (TEZ-814) Improve heuristic for determining a task has failed outputs

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha reassigned TEZ-814: -- Assignee: Bikas Saha > Improve heuristic for determining a task has failed outputs > -

[jira] [Commented] (TEZ-814) Improve heuristic for determining a task has failed outputs

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791456#comment-14791456 ] Bikas Saha commented on TEZ-814: Heuristics are mainly designed to prevent inadvertent flurry

[jira] [Updated] (TEZ-814) Improve heuristic for determining a task has failed outputs

2015-09-16 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-814: --- Fix Version/s: 0.7.1 > Improve heuristic for determining a task has failed outputs >

Failed: TEZ-814 PreCommit Build #1146

2015-09-16 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-814 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1146/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Commented] (TEZ-814) Improve heuristic for determining a task has failed outputs

2015-09-16 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791544#comment-14791544 ] TezQA commented on TEZ-814: --- {color:red}-1 overall{color}. Here are the results of testing the lat

[jira] [Created] (TEZ-2837) TEZ UI: First Task Start Time is not available

2015-09-16 Thread Jeff Zhang (JIRA)
Jeff Zhang created TEZ-2837: --- Summary: TEZ UI: First Task Start Time is not available Key: TEZ-2837 URL: https://issues.apache.org/jira/browse/TEZ-2837 Project: Apache Tez Issue Type: Improvement

[jira] [Updated] (TEZ-2837) TEZ UI: First Task Start Time is not available

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2837: Attachment: 2015-09-17_1326.png > TEZ UI: First Task Start Time is not available > ---

[jira] [Updated] (TEZ-2837) TEZ UI: First Task Start Time is not available

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2837: Issue Type: Sub-task (was: Improvement) Parent: TEZ-2760 > TEZ UI: First Task Start Time is not avail

[jira] [Updated] (TEZ-2838) Tez UI: Finish Time and Duration is not available on DAG Details

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2838: Attachment: 2015-09-17_1338.png > Tez UI: Finish Time and Duration is not available on DAG Details > -

[jira] [Created] (TEZ-2838) Tez UI: Finish Time and Duration is not available on DAG Details

2015-09-16 Thread Jeff Zhang (JIRA)
Jeff Zhang created TEZ-2838: --- Summary: Tez UI: Finish Time and Duration is not available on DAG Details Key: TEZ-2838 URL: https://issues.apache.org/jira/browse/TEZ-2838 Project: Apache Tez Issue

[jira] [Updated] (TEZ-2838) Tez UI: Finish Time and Duration is not available on DAG Details

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2838: Priority: Minor (was: Major) > Tez UI: Finish Time and Duration is not available on DAG Details > ---

[jira] [Updated] (TEZ-2838) Tez UI: Finished Time is not updated in real-time

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2838: Description: I have to refresh the page to see the finished time and duration. Same for DAG/Vertex/Task/TaskA

[jira] [Updated] (TEZ-2838) Tez UI: Finished Time is not updated in real-time

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2838: Summary: Tez UI: Finished Time is not updated in real-time (was: Tez UI: Finish Time and Duration is not avai

[jira] [Created] (TEZ-2839) Tez UI: Use another kind of bar to represent dag killed/failed

2015-09-16 Thread Jeff Zhang (JIRA)
Jeff Zhang created TEZ-2839: --- Summary: Tez UI: Use another kind of bar to represent dag killed/failed Key: TEZ-2839 URL: https://issues.apache.org/jira/browse/TEZ-2839 Project: Apache Tez Issue Ty

[jira] [Updated] (TEZ-2839) Tez UI: Use another kind of bar to represent dag killed/failed

2015-09-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2839: Attachment: 2015-09-17_1359.png > Tez UI: Use another kind of bar to represent dag killed/failed > ---