[jira] [Updated] (TEZ-2784) micro optimization: optimize TaskImpl.isFinished()
[ https://issues.apache.org/jira/browse/TEZ-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2784: -- Attachment: TEZ-2784.1.patch AM_Profiler_snapshot.jpg > micro optimization: optimize TaskImpl.isFinished() > -- > > Key: TEZ-2784 > URL: https://issues.apache.org/jira/browse/TEZ-2784 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: AM_Profiler_snapshot.jpg, TEZ-2784.1.patch > > > getInternalState() gets called multiple times within the same method within > read lock. This shows up in the AM profiler when executing large jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2784) micro optimization: optimize TaskImpl.isFinished()
Rajesh Balamohan created TEZ-2784: - Summary: micro optimization: optimize TaskImpl.isFinished() Key: TEZ-2784 URL: https://issues.apache.org/jira/browse/TEZ-2784 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan getInternalState() gets called multiple times within the same method within read lock. This shows up in the AM profiler when executing large jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2784) micro optimization: optimize TaskImpl.isFinished()
[ https://issues.apache.org/jira/browse/TEZ-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2784: -- Attachment: (was: TEZ-2784.1.patch) > micro optimization: optimize TaskImpl.isFinished() > -- > > Key: TEZ-2784 > URL: https://issues.apache.org/jira/browse/TEZ-2784 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: AM_Profiler_snapshot.jpg, TEZ-2784.1.patch > > > getInternalState() gets called multiple times within the same method within > read lock. This shows up in the AM profiler when executing large jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2784) micro optimization: optimize TaskImpl.isFinished()
[ https://issues.apache.org/jira/browse/TEZ-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2784: -- Attachment: TEZ-2784.1.patch > micro optimization: optimize TaskImpl.isFinished() > -- > > Key: TEZ-2784 > URL: https://issues.apache.org/jira/browse/TEZ-2784 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: AM_Profiler_snapshot.jpg, TEZ-2784.1.patch > > > getInternalState() gets called multiple times within the same method within > read lock. This shows up in the AM profiler when executing large jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2769) TEZ-UI Hive SQL is only displayed to line 11
[ https://issues.apache.org/jira/browse/TEZ-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734394#comment-14734394 ] Jeff Zhang commented on TEZ-2769: - +1, patch verified. > TEZ-UI Hive SQL is only displayed to line 11 > > > Key: TEZ-2769 > URL: https://issues.apache.org/jira/browse/TEZ-2769 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Affects Versions: 0.8.1 >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: 2015-09-02_1608.png, 2015-09-02_1609.png, > TEZ-2769.1.patch > > > In the DAG Details tab, hive sql is only displayed to line 11. I have to > click on the places under line 11, the remaining sql will show. It happens in > chrome & firefox, but works fine in safari. > \cc [~Sreenath] [~pramachandran] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735434#comment-14735434
]
Siddharth Seth commented on TEZ-2774:
-
Kept the following.
{code}
-LOG.info("Comparing client version with AM version"
- LOG.info(taskAttemptID + " given a go for committing the task
output.");
- LOG.info("Removing commit attempt: " + commitAttempt);
if (numEventsSent > 0) {
-StringBuilder builder = new StringBuilder();
-builder.append("Sending ").append(attemptID).append(" numEvents:
").append(numEventsSent)
-.append(" from: ").append(fromEventId).append(" to:
").append(nextFromEventId)
-.append(" out of ").append(currEventCount).append(" on-demand
events in vertex: ")
-.append(getLogIdentifier());
-LOG.info("No taskRequests. Container's idle timeout delay expired or
is new. " +
- LOG.info("Container's idle timeout expired. Releasing container"
- LOG.info("Releasing held container as either there are pending
but "
-LOG.info("Deallocating task: " + task + " before allocation");
+LOG.debug("Grouping input splits for input " +
getContext().getInputName()); | Moved into a consolidated message
{code}
Task Attempt messages generated the most amount of bulk, so have been removed.
Reasons for retaining the rest? Most of these statements are the ones which
generated a bulk of the logs - and are very standard flow. Also, some of the
information is available in other log lines - please try looking at the output
of a generated log.
bq. Lets not duplicate the logging code in checkTasksForCompletion() in
VertexImpl. Lets construct it in one place.
Removed the duplication.
bq. Would be instructive to create a histogram of log stmt occurrences over a
set of jobs to see which logs are creating bulk.
I had initially planned on doing this, except it's not trivial given the
unstructured log lines with variables all over the place. It's much simpler to
identify bulk by classname, clean them up and then look at the logs.
> Reduce logging in the AM
>
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735412#comment-14735412 ] Siddharth Seth commented on TEZ-2774: - bq. this has been useful at times when debugging rolling upgrade goof-ups. One-time line per AM instance so would be useful to retain. Moving back to INFO bq. Does this need any additional context on what is being logged? i.e. taskAttemptId or taskId? The TaskSpec has the attemptId, that should provide adequate context. bq. should this be "modulo max(50, numTasks/50)" i.e log this at max 50 times? Will change this. bq. should enhance to log container exit code, diagnostics etc. maybe log failed containers at info ( not only in this case but others that were converted to debug )? Added the status to the debug message. Failed containers are being logged in AMContainerImpl bq. might be worth retaining if this is not a very frequent log. If frequent, then yes we can remove. might be worth retaining with a less heavy message. Kept both. bq. typo: timelineAclsClas - change to TimelineACLManagerClass Fixed. bq. non logging change? This is to consolidate the log lines. bq. from 2 seconds to 30 seconds? That's not the most useful message, especially for sessions which sit around for 5 minutes by default and have pre-warmed containers. Will change the title of the jira. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735417#comment-14735417 ] Siddharth Seth commented on TEZ-2774: - bq. YarnTaskSchedulerService - Changes related to "Releasing container", "Container's idle timeout expired. Releasing container", "Releasing held container as either there are pending" could be used for debugging. Can we retain them in INFO logging? Retained both. bq. Also need to check if changes to tez-container-log4j.properties would have any impact on the tez-tools (e.g swimlane tool). Any insight into whether the tools will break. Can revert the container-log4j changes. The change is essentially to make it easier to parse. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735438#comment-14735438 ] Siddharth Seth commented on TEZ-2774: - Also for events being sent to tasks - either in the next iteration, or a follow up jira - I plan to make this less verbose. Log the first message of events being sent, followed by a message every 50-100 events that are dispatched, instead of on each event being sent out. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735536#comment-14735536 ] Bikas Saha commented on TEZ-2774: - the logs I had mentioned in my comment were the ones I remember using for debugging at various points of time. Please retain the logs I had mentioned in my comment. We can slow roll log deletion and be on the safe side instead of being aggressive now and not having information when its needed. If you have examples of specific logs (among the ones I had mentioned) that are generating bulk then please mention them. We can try to figure out if that information (including time of occurrence) is available somewhere else or if some other log could be augmented with the information. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735418#comment-14735418 ] Siddharth Seth commented on TEZ-2774: - bq. If I run standalone tez app (like wordcount), I can see the "main" thread name in the log. But if run hive on tez, I see a different thread name This is session vs non-session mode. Session mode receives requests over RPC - hence the PRC thread name shows up for some of these log messages. Non-session runs in the main thread. Nothing in this patch changes this. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2787) Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks
Jason Lowe created TEZ-2787: --- Summary: Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks Key: TEZ-2787 URL: https://issues.apache.org/jira/browse/TEZ-2787 Project: Apache Tez Issue Type: Bug Affects Versions: 0.7.0 Reporter: Jason Lowe TezRuntimeChildJVM ensures that tasks are launched with -Djava.io.tmpdir=./tmp, but there's no corresponding code to ensure the Tez AM also has a similar tmpdir setting. The client should setup the AM launch context to have -Djava.io.tmpdir=./tmp to be consistent with the tasks and to prevent accidental leaking of files in /tmp by the Tez AM if it crashes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[ https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saikat updated TEZ-2643: Attachment: TEZ-2643.1.patch > Minimize number of empty spills in Pipelined Sorter > --- > > Key: TEZ-2643 > URL: https://issues.apache.org/jira/browse/TEZ-2643 > Project: Apache Tez > Issue Type: Improvement >Reporter: Saikat >Assignee: Saikat > Attachments: TEZ-2643.1.patch, TEZ-2643.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2097) TEZ-UI Add dag logs
[ https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-2097: - Attachment: TEZ-2097.3.patch [~hitesh], reworked the patch so that 1) inprogressurl is now per attempt so that recovery will work and 2) added verification in the ats History tests to verify data is propagated from event to timeline event. However, I have not considered how to display this in the Tez UI. [~pramachandran], [~Sreenath], How to display DAG container logs in the UI. There are potentially several AM attempt logs that will correspond to a single DAG due to recovery. Should we display a comma separated list of logs, should we create a new multi-logs-cell that can display this more complicated cell data? Not sure I have the UI skill to add something this complex. Any help will be appreciated once we are set on the interface. > TEZ-UI Add dag logs > --- > > Key: TEZ-2097 > URL: https://issues.apache.org/jira/browse/TEZ-2097 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Jonathan Eagles >Priority: Critical > Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch > > > If dag fails due to AM error, there's no way to check the dag logs on tez-ui. > Users have to grab the app logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735571#comment-14735571
]
Saikat commented on TEZ-2643:
-
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only event is sent out
for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many chanes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2643 PreCommit Build #1087
Jira: https://issues.apache.org/jira/browse/TEZ-2643
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1087/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2941 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754727/TEZ-2643.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1087//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1087//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
c0deb1cdd089f0f40ef9db6ab7393a751063282a logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1084
Archived 53 artifacts
Archive block size is 32768
Received 8 blocks and 2956093 bytes
Compression is 8.1%
Took 0.83 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:135)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:248)
[jira] [Commented] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735617#comment-14735617
]
TezQA commented on TEZ-2643:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754727/TEZ-2643.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1087//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1087//console
This message is automatically generated.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[ https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saikat updated TEZ-2643: Attachment: TEZ-2643.2.patch > Minimize number of empty spills in Pipelined Sorter > --- > > Key: TEZ-2643 > URL: https://issues.apache.org/jira/browse/TEZ-2643 > Project: Apache Tez > Issue Type: Improvement >Reporter: Saikat >Assignee: Saikat > Attachments: TEZ-2643.1.patch, TEZ-2643.2.patch, TEZ-2643.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735571#comment-14735571
]
Saikat edited comment on TEZ-2643 at 9/8/15 8:42 PM:
-
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only one event is sent
out for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many changes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
was (Author: saikatr):
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only one event is sent
out for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many chanes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735571#comment-14735571
]
Saikat edited comment on TEZ-2643 at 9/8/15 8:41 PM:
-
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only one event is sent
out for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many chanes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
was (Author: saikatr):
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only event is sent out
for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many chanes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735675#comment-14735675 ] Bikas Saha commented on TEZ-2788: - [~rajesh.balamohan] Straightforward patch. Please review. > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[ https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735684#comment-14735684 ] Bikas Saha commented on TEZ-2783: - lgtm. > Refactor analyzers to extend TezAnalyzerBase > > > Key: TEZ-2783 > URL: https://issues.apache.org/jira/browse/TEZ-2783 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2783.1.patch > > > Refactoring other analyzers to extend from TezAnalyzerBase allows it be to > run in standalone cmdline mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2789) Backport events added in TEZ-2612 to branch-0.7
Bikas Saha created TEZ-2789: --- Summary: Backport events added in TEZ-2612 to branch-0.7 Key: TEZ-2789 URL: https://issues.apache.org/jira/browse/TEZ-2789 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Having the events in the 0.7 line will allow them to be persisted to ATS or SimpleHistory logging. After that, the latest analyzers from master or 0.8 could be used to analyze them. At some point when the analzyers are stable, they could move into the UI directly or be back-ported in bulk to the 0.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2789) Backport events added in TEZ-2612 to branch-0.7
[ https://issues.apache.org/jira/browse/TEZ-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2789: Attachment: TEZ-2789.1.patch > Backport events added in TEZ-2612 to branch-0.7 > --- > > Key: TEZ-2789 > URL: https://issues.apache.org/jira/browse/TEZ-2789 > Project: Apache Tez > Issue Type: Task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2789.1.patch > > > Having the events in the 0.7 line will allow them to be persisted to ATS or > SimpleHistory logging. After that, the latest analyzers from master or 0.8 > could be used to analyze them. At some point when the analzyers are stable, > they could move into the UI directly or be back-ported in bulk to the 0.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2789) Backport events added in TEZ-2612 to branch-0.7
[
https://issues.apache.org/jira/browse/TEZ-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735659#comment-14735659
]
Bikas Saha commented on TEZ-2789:
-
[~rajesh.balamohan] could you please take a quick look in case I have missed
something. Thanks!
Pulled in the following under TEZ-2612 and deleted all files touched under
plugins or analyzers since they dont exist in branch-0.7
Tested this by running jobs with branch-0.7 jars with simple history logging.
Then analyzing the simple history events using critical path analyzer from
master.
{noformat}commit 0ecb6e2e85fc32151ce648b158e08e66ea474c6f
Author: Bikas Saha
Date: Mon Sep 7 16:50:33 2015 -0700
TEZ-2778. Improvements to handle multiple read errors with complex DAGs
(bikas)
(cherry picked from commit 171d4850443a9e67aa5a68d4c8482779c0caa8eb)
Conflicts:
CHANGES.txt
tez-dag/src/main/java/org/apache/tez/dag/app/TaskCommunicatorManager.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskAttemptInfo.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/utils/Utils.java
tez-plugins/tez-history-parser/src/test/java/org/apache/tez/history/TestHistoryParser.java
tez-tools/analyzers/job-analyzer/src/main/java/org/apache/tez/analyzer/plugins/CriticalPathAnalyzer.java
tez-tools/analyzers/job-analyzer/src/test/java/org/apache/tez/analyzer/TestAnalyzer.java
commit 5fde1b5cd06fc780c7c7ea83651f098e4065d933
Author: Bikas Saha
Date: Thu Aug 13 17:53:52 2015 -0700
TEZ-2701. Add time at which container was allocated to attempt (bikas)
(cherry picked from commit 6b67b0bc1eb010f6dc8af2936ae738909e1244ff)
Conflicts:
CHANGES.txt
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskAttemptInfo.java
tez-plugins/tez-history-parser/src/test/java/org/apache/tez/history/TestHistoryParser.java
commit cdc450f95bc071620af71a5a20fe5fabf059c0f2
Author: Bikas Saha
Date: Wed Aug 5 12:35:30 2015 -0700
TEZ-2650. Timing details on Vertex state changes (bikas)
(cherry picked from commit 6970fb01bc7bd8a838dc55c5e38cbad41b1c3140)
Conflicts:
CHANGES.txt
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/BaseInfo.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/BaseParser.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/DagInfo.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskAttemptInfo.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskInfo.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/VertexInfo.java
tez-plugins/tez-history-parser/src/test/java/org/apache/tez/history/TestATSFileParser.java
tez-tools/analyzers/job-analyzer/src/main/java/org/apache/tez/analyzer/plugins/SlowestVertexAnalyzer.java
commit 25a6b66f80a1b93e843b1e8f810bc66c2f916dd6
Author: Bikas Saha
Date: Tue Aug 4 10:42:40 2015 -0700
TEZ-2647. Add input causality dependency for attempts (bikas)
(cherry picked from commit 6a99798f250310f8bd819aae89123e6146364983)
Conflicts:
CHANGES.txt
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskAttemptInfo.java
tez-plugins/tez-history-parser/src/test/java/org/apache/tez/history/TestATSFileParser.java
commit f8511b7538f782d97a01b27acd479551fe3c68b1
Author: Bikas Saha
Date: Mon Aug 3 11:38:15 2015 -0700
TEZ-2646. Add scheduling casual dependency for attempts (bikas)
(cherry picked from commit 4d381d778f82d8d17ea9f718bbac3938ce4cc40d)
Conflicts:
CHANGES.txt
tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/datamodel/TaskAttemptInfo.java
tez-plugins/tez-history-parser/src/test/java/org/apache/tez/history/TestATSFileParser.java
{noformat}
> Backport events added in TEZ-2612 to branch-0.7
> ---
>
> Key: TEZ-2789
> URL: https://issues.apache.org/jira/browse/TEZ-2789
> Project: Apache Tez
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: TEZ-2789.1.patch
>
>
> Having the events in the 0.7 line will allow them to be persisted to ATS or
> SimpleHistory logging. After that, the latest analyzers from master or 0.8
> could be used to analyze them. At some point when the
[jira] [Commented] (TEZ-2774) Reduce logging in the AM
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735671#comment-14735671 ] Siddharth Seth commented on TEZ-2774: - All of these are still available at DEBUG level. A bunch of the logs you'd mentioned are from tasks, and some of that information is available in history, or via the scheduler. Please try generating logs with a large job with the patch applied, rather than just scanning through the patch. For instance, a bunch of information from individual task logs is already available. We should aim to have enough information for hung jobs, rather than trying to maintain information which helps with debugging locality issues. LOG statements like sending events x of y assists towards this. OTOH, task asked for container does not help much. Most of that can be inferred from the Vertex transition logs which are fairly detailed. > Reduce logging in the AM > > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-2643 PreCommit Build #1088
Jira: https://issues.apache.org/jira/browse/TEZ-2643
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1088/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3451 lines...]
[INFO] Final Memory: 95M/1246M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754737/TEZ-2643.2.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1088//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1088//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6a466535a0945c8799e03b2a42a0ed92932842f8 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1084
Archived 53 artifacts
Archive block size is 32768
Received 8 blocks and 2987348 bytes
Compression is 8.1%
Took 0.7 sec
Description set: TEZ-2643
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Created] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
Bikas Saha created TEZ-2788: --- Summary: Allow TezAnalyzerBase to parse SimpleHistory logs Key: TEZ-2788 URL: https://issues.apache.org/jira/browse/TEZ-2788 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735571#comment-14735571
]
Saikat edited comment on TEZ-2643 at 9/8/15 9:18 PM:
-
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.2
comment 2:
a. move the spillrecords init to after the check for ignoreSpillIfNeeded.
b. renamed to ignoreEmptySpills
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only one event is sent
out for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
would need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many changes, so I return a boolean value from spill, to let
the caller know if there was actually a spill or not, and then the caller can
take a decision to send events and if its a last event etc.
was (Author: saikatr):
Thanks [~rajesh.balamohan] for the review comments.
Made the following changes in patchset 2643.1
comment 2: move the spillrecords init to after the check for
ignoreSpillIfNeeded.
Comment 1:
I didnt want to put the sendPipelinedShuffleEvents() inside spill because of
the following scenario:
a. in flush(), if sendPipelinedShuffleEvents() is called in spill(), then we
need to change the if (!isFinalMergeEnabled) {} where only one event is sent
out for last spill.
b. Also, in sendPipelinedShuffleEvents() hard codes isLastEvent to false, so we
wud need to pass lastEvent flag to sendPipelinedShuffleEvents().
There would be too many changes, so I return a boolean value from spill, to let
the caller know it there was actually a spill, and then the caller can take a
decision to send events and if its a last event etc.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.2.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2788: Attachment: TEZ-2788.1.patch > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735762#comment-14735762 ] Rajesh Balamohan commented on TEZ-2788: --- lgtm. +1 > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2643) Minimize number of empty spills in Pipelined Sorter
[
https://issues.apache.org/jira/browse/TEZ-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735765#comment-14735765
]
TezQA commented on TEZ-2643:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754737/TEZ-2643.2.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1088//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1088//console
This message is automatically generated.
> Minimize number of empty spills in Pipelined Sorter
> ---
>
> Key: TEZ-2643
> URL: https://issues.apache.org/jira/browse/TEZ-2643
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Saikat
>Assignee: Saikat
> Attachments: TEZ-2643.1.patch, TEZ-2643.2.patch, TEZ-2643.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2774 PreCommit Build #1091
Jira: https://issues.apache.org/jira/browse/TEZ-2774
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1091/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3134 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754758/TEZ-2774.2.txt
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1091//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1091//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
317b454a908ebfe2e66e866239622f5cb44aa276 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 4 blocks and 3084388 bytes
Compression is 4.1%
Took 0.7 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TaskAttempt, hashCode: 1221904230,
,
Container: [ContainerId: container_1_0001_01_02, NodeId: host3:0,
NodeHttpAddress: host3:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:686)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:537)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:537)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:537)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:538)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:538)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 657736953
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseWithTaskSpecificLaunchCmdOption(TestContainerReuse.java:538)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735828#comment-14735828
]
TezQA commented on TEZ-2774:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754758/TEZ-2774.2.txt
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1091//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1091//console
This message is automatically generated.
> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2097) TEZ-UI Add dag logs
[ https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735797#comment-14735797 ] Hitesh Shah commented on TEZ-2097: -- [~jeagles] Maybe we can split this jira into 2. One for the ATS publish changes and another for displaying the log links in the UI? > TEZ-UI Add dag logs > --- > > Key: TEZ-2097 > URL: https://issues.apache.org/jira/browse/TEZ-2097 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Jonathan Eagles >Priority: Critical > Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch > > > If dag fails due to AM error, there's no way to check the dag logs on tez-ui. > Users have to grab the app logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2774: Attachment: TEZ-2774.2.txt Updated patch with comments addressed as indicated in replies. Small addition for AM PID to be logged. > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2097) TEZ-UI Add dag logs
[
https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735802#comment-14735802
]
Hitesh Shah commented on TEZ-2097:
--
Minor nit:
{code}
atsEntity.addOtherInfo(ATSConstants.IN_PROGRESS_LOGS_URL + "_"
+ event.getApplicationAttemptId().toString(), event.getContainerLogs());
{code}
- Should this be using event.getApplicationAttemptId().getId() instead of
event.getApplicationAttemptId() ?
> TEZ-UI Add dag logs
> ---
>
> Key: TEZ-2097
> URL: https://issues.apache.org/jira/browse/TEZ-2097
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch
>
>
> If dag fails due to AM error, there's no way to check the dag logs on tez-ui.
> Users have to grab the app logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2788: Attachment: TEZ-2788.2.patch [~rajesh.balamohan] Could you please take another look? I have added support to download the ATS data to reduce the number of steps needed to use the analyzer. Thanks! > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2097 PreCommit Build #1090
Jira: https://issues.apache.org/jira/browse/TEZ-2097
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1090/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3469 lines...]
[INFO]
[INFO] Total time: 41:43 min
[INFO] Finished at: 2015-09-08T22:56:06+00:00
[INFO] Final Memory: 85M/1122M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754747/TEZ-2097.3.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1090//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1090//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
f8c3e064c9a6c4c837917c146cbf6b9183e3c17e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-TEZ-Build #1089
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster.testDisabledACls
Error Message:
test timed out after 5 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 5 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:151)
at com.sun.proxy.$Proxy89.getNewApplication(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)
at
org.apache.tez.client.TezYarnClient.createApplication(TezYarnClient.java:66)
at org.apache.tez.client.TezClient.createApplication(TezClient.java:893)
at org.apache.tez.client.TezClient.start(TezClient.java:407)
at
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster.testDisabledACls(TestATSHistoryWithMiniCluster.java:155)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[
https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735780#comment-14735780
]
TezQA commented on TEZ-2788:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754742/TEZ-2788.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1089//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1089//console
This message is automatically generated.
> Allow TezAnalyzerBase to parse SimpleHistory logs
> -
>
> Key: TEZ-2788
> URL: https://issues.apache.org/jira/browse/TEZ-2788
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: TEZ-2788.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2097) TEZ-UI Add dag logs
[
https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735781#comment-14735781
]
TezQA commented on TEZ-2097:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754747/TEZ-2097.3.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1090//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1090//console
This message is automatically generated.
> TEZ-UI Add dag logs
> ---
>
> Key: TEZ-2097
> URL: https://issues.apache.org/jira/browse/TEZ-2097
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch
>
>
> If dag fails due to AM error, there's no way to check the dag logs on tez-ui.
> Users have to grab the app logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2788 PreCommit Build #1089
Jira: https://issues.apache.org/jira/browse/TEZ-2788
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1089/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3459 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754742/TEZ-2788.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1089//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1089//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
fb6dee0ed6b2c65969349b2c8f76fc8da504b079 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 6 blocks and 3069704 bytes
Compression is 6.0%
Took 0.73 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Comment Edited] (TEZ-2097) TEZ-UI Add dag logs
[ https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735797#comment-14735797 ] Hitesh Shah edited comment on TEZ-2097 at 9/8/15 11:09 PM: --- [~jeagles] If needed, we can split this jira into 2. One for the ATS publish changes and another for displaying the log links in the UI? Assuming the code change for displaying the data is small, a single jira should also be fine. was (Author: hitesh): [~jeagles] Maybe we can split this jira into 2. One for the ATS publish changes and another for displaying the log links in the UI? > TEZ-UI Add dag logs > --- > > Key: TEZ-2097 > URL: https://issues.apache.org/jira/browse/TEZ-2097 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Jonathan Eagles >Priority: Critical > Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch > > > If dag fails due to AM error, there's no way to check the dag logs on tez-ui. > Users have to grab the app logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2774: Summary: Reduce logging in the AM, and parts of the runtime (was: Reduce logging in the AM) > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735829#comment-14735829 ] Bikas Saha commented on TEZ-2774: - I will reiterate my initial comments about the logs I have mentioned. Let us keep them there at info level. Logging at debug level is not always useful because it may be an intermittent issue or too expensive to reproduce. Some of the removed logs like which attempt succeeded and which got killed will only be relevant during speculation and not logged all the time. Removing the kill log is not going to reduce general logging size but will be sorely missed if there is a bug in speculation. The task scheduler logs for containers have been useful when debugging why min-held or pre-warm containers were not working as expected. The input-spec log is used when figuring out how many events should have actually come to an input and how many actually did for debugging hung jobs. Then correlating it with event routing log from the AM to see where the error is. The event routing log from the AM was removed in the initial patch while the the next version introduces it back but continues to hide the task spec log. This seems arbitrary. It is hard to explain all the debugging context from previous experience. Hence, my suggestion is to accept the comments as is instead of debating them subjectively. With time we can remove more logs. But I am quite wary of doing such a bulk removal and not having logs to debug when I need them. > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735968#comment-14735968 ] Rajesh Balamohan commented on TEZ-2788: --- >> (user running an analyzer for a dag on a gateway box) Got it. Patch .2 lgtm. +1 In case of any errors during download (due to config dependency) or so, user can look at the error message and act accordingly. > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[ https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2783: -- Attachment: TEZ-2783.2.rebased.patch Attaching rebased patch. > Refactor analyzers to extend TezAnalyzerBase > > > Key: TEZ-2783 > URL: https://issues.apache.org/jira/browse/TEZ-2783 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2783.1.patch, TEZ-2783.2.rebased.patch > > > Refactoring other analyzers to extend from TezAnalyzerBase allows it be to > run in standalone cmdline mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[ https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2783: -- Target Version/s: 0.8.1 > Refactor analyzers to extend TezAnalyzerBase > > > Key: TEZ-2783 > URL: https://issues.apache.org/jira/browse/TEZ-2783 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2783.1.patch, TEZ-2783.2.rebased.patch > > > Refactoring other analyzers to extend from TezAnalyzerBase allows it be to > run in standalone cmdline mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[
https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735871#comment-14735871
]
TezQA commented on TEZ-2788:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754760/TEZ-2788.2.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1092//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1092//console
This message is automatically generated.
> Allow TezAnalyzerBase to parse SimpleHistory logs
> -
>
> Key: TEZ-2788
> URL: https://issues.apache.org/jira/browse/TEZ-2788
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2788 PreCommit Build #1092
Jira: https://issues.apache.org/jira/browse/TEZ-2788
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1092/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2942 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754760/TEZ-2788.2.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1092//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1092//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a572f6a2a86f51ab122047babe923780ea3b0af5 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 10 blocks and 2889629 bytes
Compression is 10.2%
Took 0.67 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:135)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:248)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735928#comment-14735928 ] Rajesh Balamohan commented on TEZ-2788: --- >> added support to download the ATS data to reduce the number of steps needed >> to use the analyzer. - Should this be clubbed with analyzers? Reason is, earlier users could run the analyzer in any machine without cluster config dependency. Adding this would introduce this dependency (users need to run this on edge/gateway node or on a machine where cluster configs are setup correctly). - Also, users could specific y different timelineserver URL to download DAG from different ATS instance (e.g machine having access to 2 cluster ATS instances). > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735943#comment-14735943 ] Bikas Saha commented on TEZ-2788: - The analysers can still be used separately with the ATSImportTool. So the above 2 cases would still work. This is an attempt to optionally make the common case (user running an analyzer for a dag on a gateway box) simpler - 1 step. Does that clarify? If you think this is undesirable, then I can commit v1 patch only. > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2097) TEZ-UI Add dag logs
[
https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-2097:
-
Attachment: TEZ-2097.4.patch
[~hitesh], here is the patch with the getAttemptId int replacing the whole
applicationAttemptId string. It also removes all the Tez UI code changes to be
left for another JIRA.
Below is an example of what the new REST api entry looks like.
{code}
otherInfo: {
inProgressLogsURL_1":
"localhost:8042/node/containerlogs/container_1441762810631_0001_01_01/jeagles",
inProgressLogsURL_2":
"localhost:8042/node/containerlogs/container_1441762810631_0001_01_02/jeagles",
}
{code}
> TEZ-UI Add dag logs
> ---
>
> Key: TEZ-2097
> URL: https://issues.apache.org/jira/browse/TEZ-2097
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch,
> TEZ-2097.4.patch
>
>
> If dag fails due to AM error, there's no way to check the dag logs on tez-ui.
> Users have to grab the app logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2788) Allow TezAnalyzerBase to parse SimpleHistory logs
[ https://issues.apache.org/jira/browse/TEZ-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735927#comment-14735927 ] Bikas Saha commented on TEZ-2788: - Tests not added. I have verified on a real cluster that the new command line single tool approach works. [~rajesh.balamohan] Please review. > Allow TezAnalyzerBase to parse SimpleHistory logs > - > > Key: TEZ-2788 > URL: https://issues.apache.org/jira/browse/TEZ-2788 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2788.1.patch, TEZ-2788.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[
https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736050#comment-14736050
]
TezQA commented on TEZ-2783:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754790/TEZ-2783.2.rebased.patch
against master revision 56f7847.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1094//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1094//artifact/patchprocess/newPatchFindbugsWarningsjob-analyzer.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1094//console
This message is automatically generated.
> Refactor analyzers to extend TezAnalyzerBase
>
>
> Key: TEZ-2783
> URL: https://issues.apache.org/jira/browse/TEZ-2783
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2783.1.patch, TEZ-2783.2.rebased.patch
>
>
> Refactoring other analyzers to extend from TezAnalyzerBase allows it be to
> run in standalone cmdline mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2097) TEZ-UI Add dag logs
[ https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736115#comment-14736115 ] Jonathan Eagles commented on TEZ-2097: -- Looks like the test failure is from TEZ-2686. > TEZ-UI Add dag logs > --- > > Key: TEZ-2097 > URL: https://issues.apache.org/jira/browse/TEZ-2097 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Jonathan Eagles >Priority: Critical > Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch, > TEZ-2097.4.patch > > > If dag fails due to AM error, there's no way to check the dag logs on tez-ui. > Users have to grab the app logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2097) TEZ-UI Add dag logs
[
https://issues.apache.org/jira/browse/TEZ-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736033#comment-14736033
]
TezQA commented on TEZ-2097:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754787/TEZ-2097.4.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1093//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1093//console
This message is automatically generated.
> TEZ-UI Add dag logs
> ---
>
> Key: TEZ-2097
> URL: https://issues.apache.org/jira/browse/TEZ-2097
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: TEZ-2097.1.patch, TEZ-2097.2.patch, TEZ-2097.3.patch,
> TEZ-2097.4.patch
>
>
> If dag fails due to AM error, there's no way to check the dag logs on tez-ui.
> Users have to grab the app logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2097 PreCommit Build #1093
Jira: https://issues.apache.org/jira/browse/TEZ-2097
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1093/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3453 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754787/TEZ-2097.4.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1093//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1093//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4ceb541032b1a024420f20a64bcccfdef93615b2 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 10 blocks and 2945716 bytes
Compression is 10.0%
Took 0.69 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2784) optimize TaskImpl.isFinished()
[
https://issues.apache.org/jira/browse/TEZ-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736060#comment-14736060
]
TezQA commented on TEZ-2784:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754581/TEZ-2784.1.patch
against master revision 56f7847.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1095//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1095//console
This message is automatically generated.
> optimize TaskImpl.isFinished()
> --
>
> Key: TEZ-2784
> URL: https://issues.apache.org/jira/browse/TEZ-2784
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: AM_Profiler_snapshot.jpg, TEZ-2784.1.patch
>
>
> getInternalState() gets called multiple times within the same method within
> read lock. This shows up in the AM profiler when executing large jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2784 PreCommit Build #1095
Jira: https://issues.apache.org/jira/browse/TEZ-2784
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1095/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3455 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754581/TEZ-2784.1.patch
against master revision 56f7847.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1095//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1095//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
86a7c1b5defaa9d0704b0623d371afe1c03739c8 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 6 blocks and 3054904 bytes
Compression is 6.0%
Took 0.89 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2785) Tez UI: Implement an abstract controller to enclose basic generic functionalities.
[ https://issues.apache.org/jira/browse/TEZ-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2785: Description: Move setup, reset is Active from TablePageController to this abstract controller. > Tez UI: Implement an abstract controller to enclose basic generic > functionalities. > -- > > Key: TEZ-2785 > URL: https://issues.apache.org/jira/browse/TEZ-2785 > Project: Apache Tez > Issue Type: Task >Reporter: Sreenath Somarajapuram >Priority: Minor > > Move setup, reset is Active from TablePageController to this abstract > controller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2785) Tez UI: Implement an abstract controller to enclose basic generic functionalities.
Sreenath Somarajapuram created TEZ-2785: --- Summary: Tez UI: Implement an abstract controller to enclose basic generic functionalities. Key: TEZ-2785 URL: https://issues.apache.org/jira/browse/TEZ-2785 Project: Apache Tez Issue Type: Task Reporter: Sreenath Somarajapuram Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735223#comment-14735223 ] Prakash Ramachandran commented on TEZ-2780: --- [~Sreenath] still reviewing the changes, one quick question though. Why does the polling class need to be a mixin? this has two disadvantages. 1. there can be only a single instance of the polling. 2. its tied to the type of controller it is used in. IMHO it would be better if its not a mixin, instantiate it in init. and depending on if polling is required, do a start/stop. if you are concerned about code reuse across task/task attempts write a helper function that takes in the required arguments. > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > > Add AM API for fetching realtime tasks info: > - API endpoint : /ws/v2/tez/tasksInfo > - Query Params: > -- dagID: The complete dagID, (mandatory). > -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. > -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex > -- limit: Maximum number of items to be returned (Defaults to 100). > - If taskMinID is passed: All (capped by limit) the specified tasks will be > returned. vertexMinID if present wont be considered. > - IF vertexMinID is passed: All (capped by limit) tasks under the vertices > will be returned. > - If just dagID is passed: All (capped by limit) tasks under the DAG will be > returned. > - Data returned: complete task id, progress, status > Modify table component to automatically update cell based on model change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2780: Summary: Tez UI: Update All Tasks page while in progress. (was: Tez UI: Update tasks page while in progress.) > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2784) micro optimization: optimize TaskImpl.isFinished()
[ https://issues.apache.org/jira/browse/TEZ-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735149#comment-14735149 ] Bikas Saha commented on TEZ-2784: - lgtm. > micro optimization: optimize TaskImpl.isFinished() > -- > > Key: TEZ-2784 > URL: https://issues.apache.org/jira/browse/TEZ-2784 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: AM_Profiler_snapshot.jpg, TEZ-2784.1.patch > > > getInternalState() gets called multiple times within the same method within > read lock. This shows up in the AM profiler when executing large jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2780) Tez UI: Update tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2780: Attachment: TEZ-2780.1.patch 1. Added AM API for fetching realtime tasks info. 2. Upgraded pollster to manage polling for a specific entity. 3. Added progress column to All Tasks table. 4. Updated table to automatically reflect model change - Just need to set observePath to true in defaultColumnConfigs. 5. Updated table logic to send action on row change. [~pramachandran] Please review. > Tez UI: Update tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2780: Description: Add AM API for fetching realtime tasks info: - API endpoint : /ws/v2/tez/tasksInfo - Query Params: -- dagID: The complete dagID, (mandatory). -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex -- limit: Maximum number of items to be returned (Defaults to 100). - If taskMinID is passed: All (capped by limit) the specified tasks will be returned. vertexMinID if present wont be considered. - IF vertexMinID is passed: All (capped by limit) tasks under the vertices will be returned. - If just dagID is passed: All (capped by limit) tasks under the DAG will be returned. - Data returned: complete task id, progress, status > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > > Add AM API for fetching realtime tasks info: > - API endpoint : /ws/v2/tez/tasksInfo > - Query Params: > -- dagID: The complete dagID, (mandatory). > -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. > -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex > -- limit: Maximum number of items to be returned (Defaults to 100). > - If taskMinID is passed: All (capped by limit) the specified tasks will be > returned. vertexMinID if present wont be considered. > - IF vertexMinID is passed: All (capped by limit) tasks under the vertices > will be returned. > - If just dagID is passed: All (capped by limit) tasks under the DAG will be > returned. > - Data returned: complete task id, progress, status -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735161#comment-14735161 ] Sreenath Somarajapuram edited comment on TEZ-2780 at 9/8/15 5:04 PM: - 1. Added AM API for fetching realtime tasks info. 2. Upgraded pollster to manage polling for a specific entity. 3. Added progress column to All Tasks table. 4. Updated table to automatically reflect model change - Just need to set observePath to true in defaultColumnConfigs. 5. Updated table logic to send action on row change. [~pramachandran] Please review. was (Author: sreenath): 1. Added AM API for fetching realtime tasks info. 2. Upgraded pollster to manage polling for a specific entity. 3. Added progress column to All Tasks table. 4. Updated table to automatically reflect model change - Just need to set observePath to true in defaultColumnConfigs. 5. Updated table logic to send action on row change. [~pramachandran] Please review. > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > > Add AM API for fetching realtime tasks info: > - API endpoint : /ws/v2/tez/tasksInfo > - Query Params: > -- dagID: The complete dagID, (mandatory). > -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. > -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex > -- limit: Maximum number of items to be returned (Defaults to 100). > - If taskMinID is passed: All (capped by limit) the specified tasks will be > returned. vertexMinID if present wont be considered. > - IF vertexMinID is passed: All (capped by limit) tasks under the vertices > will be returned. > - If just dagID is passed: All (capped by limit) tasks under the DAG will be > returned. > - Data returned: complete task id, progress, status > Modify table component to automatically update cell based on model change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2780: Description: Add AM API for fetching realtime tasks info: - API endpoint : /ws/v2/tez/tasksInfo - Query Params: -- dagID: The complete dagID, (mandatory). -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex -- limit: Maximum number of items to be returned (Defaults to 100). - If taskMinID is passed: All (capped by limit) the specified tasks will be returned. vertexMinID if present wont be considered. - IF vertexMinID is passed: All (capped by limit) tasks under the vertices will be returned. - If just dagID is passed: All (capped by limit) tasks under the DAG will be returned. - Data returned: complete task id, progress, status Modify table component to automatically update cell based on model change. was: Add AM API for fetching realtime tasks info: - API endpoint : /ws/v2/tez/tasksInfo - Query Params: -- dagID: The complete dagID, (mandatory). -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex -- limit: Maximum number of items to be returned (Defaults to 100). - If taskMinID is passed: All (capped by limit) the specified tasks will be returned. vertexMinID if present wont be considered. - IF vertexMinID is passed: All (capped by limit) tasks under the vertices will be returned. - If just dagID is passed: All (capped by limit) tasks under the DAG will be returned. - Data returned: complete task id, progress, status > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > > Add AM API for fetching realtime tasks info: > - API endpoint : /ws/v2/tez/tasksInfo > - Query Params: > -- dagID: The complete dagID, (mandatory). > -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. > -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex > -- limit: Maximum number of items to be returned (Defaults to 100). > - If taskMinID is passed: All (capped by limit) the specified tasks will be > returned. vertexMinID if present wont be considered. > - IF vertexMinID is passed: All (capped by limit) tasks under the vertices > will be returned. > - If just dagID is passed: All (capped by limit) tasks under the DAG will be > returned. > - Data returned: complete task id, progress, status > Modify table component to automatically update cell based on model change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2786) Tez UI: Update task details page while in progress.
Sreenath Somarajapuram created TEZ-2786: --- Summary: Tez UI: Update task details page while in progress. Key: TEZ-2786 URL: https://issues.apache.org/jira/browse/TEZ-2786 Project: Apache Tez Issue Type: Sub-task Reporter: Sreenath Somarajapuram -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2633) Allow VertexManagerPlugins to receive and report based on attempts instead of tasks
[ https://issues.apache.org/jira/browse/TEZ-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2633: Issue Type: Task (was: Sub-task) Parent: (was: TEZ-2612) > Allow VertexManagerPlugins to receive and report based on attempts instead of > tasks > --- > > Key: TEZ-2633 > URL: https://issues.apache.org/jira/browse/TEZ-2633 > Project: Apache Tez > Issue Type: Task >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.8.0-alpha > > Attachments: TEZ-2633.1.patch, TEZ-2633.2.patch, TEZ-2633.3.patch, > TEZ-2633.4.patch > > > If the same event is sent from an attempt and its retry then there is no way > to differentiate between them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2744) Logging cleanup and enhancements
[ https://issues.apache.org/jira/browse/TEZ-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735299#comment-14735299 ] Siddharth Seth commented on TEZ-2744: - In terms of the history events - they provide a good amount of information about which attempt succeeded, which task is completed etc. That allows for some of the other messages to be removed, and access to information in the same logs. So I think it's better to keep these in the main log file. For yarn related logs, and hadoop related logs - yes they should be at a different level. See previous comment about handling RackResolver. Users need to be able to set these to whatever level they want; the custom handling of the log level which exists today should provide this. > Logging cleanup and enhancements > > > Key: TEZ-2744 > URL: https://issues.apache.org/jira/browse/TEZ-2744 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Siddharth Seth > > There's far too much logging, some of which is redundant information - which > leads to large log files for jobs. > Also, some logging can be enhanced to include additional context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[
https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735303#comment-14735303
]
TezQA commented on TEZ-2780:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754671/TEZ-2780.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1086//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1086//console
This message is automatically generated.
> Tez UI: Update All Tasks page while in progress.
>
>
> Key: TEZ-2780
> URL: https://issues.apache.org/jira/browse/TEZ-2780
> Project: Apache Tez
> Issue Type: Sub-task
> Components: UI
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch
>
>
> Add AM API for fetching realtime tasks info:
> - API endpoint : /ws/v2/tez/tasksInfo
> - Query Params:
> -- dagID: The complete dagID, (mandatory).
> -- vertexMinID: A comma separated list. vertexMinID = vertexIndex.
> -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex
> -- limit: Maximum number of items to be returned (Defaults to 100).
> - If taskMinID is passed: All (capped by limit) the specified tasks will be
> returned. vertexMinID if present wont be considered.
> - IF vertexMinID is passed: All (capped by limit) tasks under the vertices
> will be returned.
> - If just dagID is passed: All (capped by limit) tasks under the DAG will be
> returned.
> - Data returned: complete task id, progress, status
> Modify table component to automatically update cell based on model change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2780 PreCommit Build #1086
Jira: https://issues.apache.org/jira/browse/TEZ-2780
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1086/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3469 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754671/TEZ-2780.1.patch
against master revision c07d6b7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1086//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1086//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
c10e0f0017172e678dfc9ea7eadd647c20d40e6a logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1084
Archived 53 artifacts
Archive block size is 32768
Received 8 blocks and 2990368 bytes
Compression is 8.1%
Took 0.79 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2780) Tez UI: Update All Tasks page while in progress.
[ https://issues.apache.org/jira/browse/TEZ-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735305#comment-14735305 ] Hitesh Shah commented on TEZ-2780: -- It would be good if the AM webservices and the UI changes could be split out into 2 jiras. > Tez UI: Update All Tasks page while in progress. > > > Key: TEZ-2780 > URL: https://issues.apache.org/jira/browse/TEZ-2780 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2780.1.patch, TEZ-2780.wip.1.patch > > > Add AM API for fetching realtime tasks info: > - API endpoint : /ws/v2/tez/tasksInfo > - Query Params: > -- dagID: The complete dagID, (mandatory). > -- vertexMinID: A comma separated list. vertexMinID = vertexIndex. > -- taskMinID: A comma separated list. taskMinID = vertexIndex_taskIndex > -- limit: Maximum number of items to be returned (Defaults to 100). > - If taskMinID is passed: All (capped by limit) the specified tasks will be > returned. vertexMinID if present wont be considered. > - IF vertexMinID is passed: All (capped by limit) tasks under the vertices > will be returned. > - If just dagID is passed: All (capped by limit) tasks under the DAG will be > returned. > - Data returned: complete task id, progress, status > Modify table component to automatically update cell based on model change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2783 PreCommit Build #1096
Jira: https://issues.apache.org/jira/browse/TEZ-2783
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1096/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3465 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754817/TEZ-2783.3.patch
against master revision 9aa3ae6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1096//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1096//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
fff893cd0b517adae528f2798d8e22b3c6a96e23 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1088
Archived 53 artifacts
Archive block size is 32768
Received 26 blocks and 2399094 bytes
Compression is 26.2%
Took 0.58 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[
https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736228#comment-14736228
]
TezQA commented on TEZ-2783:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754817/TEZ-2783.3.patch
against master revision 9aa3ae6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1096//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1096//console
This message is automatically generated.
> Refactor analyzers to extend TezAnalyzerBase
>
>
> Key: TEZ-2783
> URL: https://issues.apache.org/jira/browse/TEZ-2783
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2783.1.patch, TEZ-2783.2.rebased.patch,
> TEZ-2783.3.patch
>
>
> Refactoring other analyzers to extend from TezAnalyzerBase allows it be to
> run in standalone cmdline mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2783) Refactor analyzers to extend TezAnalyzerBase
[ https://issues.apache.org/jira/browse/TEZ-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2783: -- Attachment: TEZ-2783.3.patch Fixing findbug warning (related to string concat) > Refactor analyzers to extend TezAnalyzerBase > > > Key: TEZ-2783 > URL: https://issues.apache.org/jira/browse/TEZ-2783 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2783.1.patch, TEZ-2783.2.rebased.patch, > TEZ-2783.3.patch > > > Refactoring other analyzers to extend from TezAnalyzerBase allows it be to > run in standalone cmdline mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
