[jira] [Created] (TEZ-2839) Tez UI: Use another kind of bar to represent dag killed/failed
Jeff Zhang created TEZ-2839: --- Summary: Tez UI: Use another kind of bar to represent dag killed/failed Key: TEZ-2839 URL: https://issues.apache.org/jira/browse/TEZ-2839 Project: Apache Tez Issue Type: Sub-task Reporter: Jeff Zhang Priority: Minor Currently tez-ui use a blue animation bar to indicate the progress of dag. It would be better to use another kind (red one and without animation ?) of bar in the case of dag failed/killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2839) Tez UI: Use another kind of bar to represent dag killed/failed
[ https://issues.apache.org/jira/browse/TEZ-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2839: Attachment: 2015-09-17_1359.png > Tez UI: Use another kind of bar to represent dag killed/failed > -- > > Key: TEZ-2839 > URL: https://issues.apache.org/jira/browse/TEZ-2839 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Reporter: Jeff Zhang >Priority: Minor > Attachments: 2015-09-17_1359.png > > > Currently tez-ui use a blue animation bar to indicate the progress of dag. > It would be better to use another kind (red one and without animation ?) of > bar in the case of dag failed/killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2581) Umbrella for Tez Recovery Redesign
[ https://issues.apache.org/jira/browse/TEZ-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2581: Attachment: TEZ-2581-WIP-4.patch rebase patch > Umbrella for Tez Recovery Redesign > -- > > Key: TEZ-2581 > URL: https://issues.apache.org/jira/browse/TEZ-2581 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-2581-WIP-1.patch, TEZ-2581-WIP-2.patch, > TEZ-2581-WIP-3.patch, TEZ-2581-WIP-4.patch, TezRecoveryRedesignProposal.pdf, > TezRecoveryRedesignV1.1.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2838) Tez UI: Finished Time is not updated when dag is completed
[ https://issues.apache.org/jira/browse/TEZ-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2838: Summary: Tez UI: Finished Time is not updated when dag is completed (was: Tez UI: Finished Time is not updated in real-time) > Tez UI: Finished Time is not updated when dag is completed > -- > > Key: TEZ-2838 > URL: https://issues.apache.org/jira/browse/TEZ-2838 > Project: Apache Tez > Issue Type: Sub-task > Components: UI >Affects Versions: 0.8.1 >Reporter: Jeff Zhang >Priority: Minor > Attachments: 2015-09-17_1338.png > > > I have to refresh the page to see the finished time and duration. > Same for DAG/Vertex/Task/TaskAttempt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2581 PreCommit Build #1147
Jira: https://issues.apache.org/jira/browse/TEZ-2581
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1147/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3002 lines...]
against master revision 1a065b9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 142 javac
compiler warnings (more than the master's current 140 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1147//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
5c01438ca8b976fe6db8baefa1e308cd2a8be0a6 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1144
Archived 54 artifacts
Archive block size is 32768
Received 4 blocks and 3136136 bytes
Compression is 4.0%
Took 0.96 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:135)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:248)
[jira] [Commented] (TEZ-2581) Umbrella for Tez Recovery Redesign
[
https://issues.apache.org/jira/browse/TEZ-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14802891#comment-14802891
]
TezQA commented on TEZ-2581:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757064/TEZ-2581-WIP-4.patch
against master revision 1a065b9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 142 javac
compiler warnings (more than the master's current 140 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1147//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1147//console
This message is automatically generated.
> Umbrella for Tez Recovery Redesign
> --
>
> Key: TEZ-2581
> URL: https://issues.apache.org/jira/browse/TEZ-2581
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Attachments: TEZ-2581-WIP-1.patch, TEZ-2581-WIP-2.patch,
> TEZ-2581-WIP-3.patch, TEZ-2581-WIP-4.patch, TezRecoveryRedesignProposal.pdf,
> TezRecoveryRedesignV1.1.pdf
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803330#comment-14803330 ] Bikas Saha commented on TEZ-2832: - [~rajesh.balamohan] Please review when convenient! > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2774: Attachment: TEZ-2774.addendum.2.patch > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.8.1 > > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, > TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, > TEZ-2774.addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2816) Build with hadoop 2.4 fails
[
https://issues.apache.org/jira/browse/TEZ-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803327#comment-14803327
]
Bikas Saha commented on TEZ-2816:
-
Ping!
> Build with hadoop 2.4 fails
> ---
>
> Key: TEZ-2816
> URL: https://issues.apache.org/jira/browse/TEZ-2816
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jeff Zhang
>Assignee: Bikas Saha
> Attachments: TEZ-2816.1.patch, TEZ-2816.2.patch, TEZ-2816.3.patch
>
>
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/170/console
> {noformat}
> Running org.apache.tez.analyzer.TestAnalyzer
> Tests run: 13, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 99.595 sec
> <<< FAILURE!
> testBasicInputFailureWithoutExit(org.apache.tez.analyzer.TestAnalyzer) Time
> elapsed: 6.276 sec <<< FAILURE!
> java.lang.AssertionError: v2 : 00_0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.tez.analyzer.TestAnalyzer.verifyCriticalPath(TestAnalyzer.java:273)
> at
> org.apache.tez.analyzer.TestAnalyzer.runDAGAndVerify(TestAnalyzer.java:220)
> at
> org.apache.tez.analyzer.TestAnalyzer.testBasicInputFailureWithoutExit(TestAnalyzer.java:399)
> testCascadingInputFailureWithExitSuccess(org.apache.tez.analyzer.TestAnalyzer)
> Time elapsed: 5.986 sec <<< FAILURE!
> java.lang.AssertionError: v3 : 00_1
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.tez.analyzer.TestAnalyzer.verifyCriticalPath(TestAnalyzer.java:273)
> at
> org.apache.tez.analyzer.TestAnalyzer.runDAGAndVerify(TestAnalyzer.java:220)
> at
> org.apache.tez.analyzer.TestAnalyzer.testCascadingInputFailureWithExitSuccess(TestAnalyzer.java:561)
> Results :
> Failed tests:
>
> TestAnalyzer.testBasicInputFailureWithoutExit:399->runDAGAndVerify:220->verifyCriticalPath:273
> v2 : 00_0
>
> TestAnalyzer.testCascadingInputFailureWithExitSuccess:561->runDAGAndVerify:220->verifyCriticalPath:273
> v3 : 00_1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804316#comment-14804316
]
TezQA commented on TEZ-2774:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757126/TEZ-2774.addendum.2.patch
against master revision 1a065b9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1148//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1148//console
This message is automatically generated.
> Reduce logging in the AM, and parts of the runtime
> --
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 0.8.1
>
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt,
> TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch,
> TEZ-2774.addendum.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804367#comment-14804367 ] Bikas Saha edited comment on TEZ-2774 at 9/17/15 8:19 PM: -- [~hitesh] [~sseth] Can you take a quick look at the findbugs fix addendum please. Warning is: Inconsistent synchronization of org.apache.tez.dag.app.rm.YarnTaskSchedulerService.numHeartbeats; locked 57% of time was (Author: bikassaha): [~hitesh] [~sseth] Can you take a quick look at the findbugs fix addendum please. > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.8.1 > > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, > TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, > TEZ-2774.addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804367#comment-14804367 ] Bikas Saha commented on TEZ-2774: - [~hitesh] [~sseth] Can you take a quick look at the findbugs fix addendum please. > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.8.1 > > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, > TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, > TEZ-2774.addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2840 PreCommit Build #1149
Jira: https://issues.apache.org/jira/browse/TEZ-2840
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1149/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3456 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757159/TEZ-2840.patch
against master revision 1a065b9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1149//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1149//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1149//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6cd19a86f17ed312891b18576a6f0c2fd0abce9d logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1144
Archived 53 artifacts
Archive block size is 32768
Received 31 blocks and 2247193 bytes
Compression is 31.1%
Took 0.57 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2840) MRInputLegacy.init should set splitInfoViaEvents
[
https://issues.apache.org/jira/browse/TEZ-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804658#comment-14804658
]
TezQA commented on TEZ-2840:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757159/TEZ-2840.patch
against master revision 1a065b9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1149//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1149//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1149//console
This message is automatically generated.
> MRInputLegacy.init should set splitInfoViaEvents
>
>
> Key: TEZ-2840
> URL: https://issues.apache.org/jira/browse/TEZ-2840
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: TEZ-2840.patch
>
>
> Only one path that uses initializeInternal and thus this field actually sets
> it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2840) MRInputLegacy.init should set splitInfoViaEvents
[ https://issues.apache.org/jira/browse/TEZ-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-2840: -- Description: Only one path that uses initializeInternal and thus this field actually sets it. > MRInputLegacy.init should set splitInfoViaEvents > > > Key: TEZ-2840 > URL: https://issues.apache.org/jira/browse/TEZ-2840 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Only one path that uses initializeInternal and thus this field actually sets > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2840) MRInputLegacy.init should set splitInfoViaEvents
Sergey Shelukhin created TEZ-2840: - Summary: MRInputLegacy.init should set splitInfoViaEvents Key: TEZ-2840 URL: https://issues.apache.org/jira/browse/TEZ-2840 Project: Apache Tez Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2840) MRInputLegacy.init should set splitInfoViaEvents
[ https://issues.apache.org/jira/browse/TEZ-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804514#comment-14804514 ] Sergey Shelukhin commented on TEZ-2840: --- [~sseth] [~hitesh] can you take a look? > MRInputLegacy.init should set splitInfoViaEvents > > > Key: TEZ-2840 > URL: https://issues.apache.org/jira/browse/TEZ-2840 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2840.patch > > > Only one path that uses initializeInternal and thus this field actually sets > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha reopened TEZ-2774: - the addendum caused a findbugs warning. re-opening to trigger jenkins on addendum.2 to verify fix for findbugs. > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.8.1 > > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, > TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, > TEZ-2774.addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-814) Improve heuristic for determining a task has failed outputs
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-814: --- Attachment: TEZ-814.2.patch Fix for ignoring findbugs. > Improve heuristic for determining a task has failed outputs > --- > > Key: TEZ-814 > URL: https://issues.apache.org/jira/browse/TEZ-814 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.7.1 > > Attachments: TEZ-814.1.patch, TEZ-814.2.patch > > > Currently 25% of consumers need to report failure. However we may not always > have those many error reports. Eg. this is the last consumer and it the > source is lost. Or some consumers are cut off from the source. The job may > hang on those consumers waiting for a re-run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2832: Attachment: (was: TEZ-2832.2.patch) > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2832: Attachment: TEZ-2832.2.patch Patch removes confusing error_color and makes it consistent with existing scheme. > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2832: Attachment: TEZ-2832.2.patch > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-814) Improve heuristic for determining a task has failed outputs
[
https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804874#comment-14804874
]
TezQA commented on TEZ-814:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757188/TEZ-814.2.patch
against master revision 35cf023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
orgTests
org.apache.tez.tests.TestExternalTezServices
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1150//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1150//console
This message is automatically generated.
> Improve heuristic for determining a task has failed outputs
> ---
>
> Key: TEZ-814
> URL: https://issues.apache.org/jira/browse/TEZ-814
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 0.7.1
>
> Attachments: TEZ-814.1.patch, TEZ-814.2.patch
>
>
> Currently 25% of consumers need to report failure. However we may not always
> have those many error reports. Eg. this is the last consumer and it the
> source is lost. Or some consumers are cut off from the source. The job may
> hang on those consumers waiting for a re-run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-814 PreCommit Build #1150
Jira: https://issues.apache.org/jira/browse/TEZ-814
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1150/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3447 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757188/TEZ-814.2.patch
against master revision 35cf023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
orgTests
org.apache.tez.tests.TestExternalTezServices
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1150//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1150//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
dc665e089d322541b73047662fd308c8746000ba logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #1144
Archived 53 artifacts
Archive block size is 32768
Received 10 blocks and 2939223 bytes
Compression is 10.0%
Took 0.84 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-814) Improve heuristic for determining a task has failed outputs
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804876#comment-14804876 ] Rajesh Balamohan commented on TEZ-814: -- lgtm. +1. Even when tez.task.max.allowed.output.failures & tez.task.max.allowed.output.failures.fraction are not converging, this would end up restarting producer after 300 seconds in case of output read-error. Should this be backported to 0.6 and 0.5 as well? > Improve heuristic for determining a task has failed outputs > --- > > Key: TEZ-814 > URL: https://issues.apache.org/jira/browse/TEZ-814 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.7.1 > > Attachments: TEZ-814.1.patch, TEZ-814.2.patch > > > Currently 25% of consumers need to report failure. However we may not always > have those many error reports. Eg. this is the last consumer and it the > source is lost. Or some consumers are cut off from the source. The job may > hang on those consumers waiting for a re-run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2832 PreCommit Build #1151
Jira: https://issues.apache.org/jira/browse/TEZ-2832
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1151/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 1528 lines...]
Running tests
/home/jenkins/tools/maven/latest/bin/mvn clean install -fn -DTezPatchProcess
cat:
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/../patchprocess/testrun.txt:
No such file or directory
awk: fatal: cannot open file
`/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/../patchprocess/testrun.txt'
for reading (No such file or directory)
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757191/TEZ-2832.2.patch
against master revision 35cf023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1151//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1151//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
43277f702e970f2a084750b9918d2f67f5ca8a0b logged out
==
==
Finished build.
==
==
Archiving artifacts
ERROR: No artifacts found that match the file pattern "patchprocess/*.*".
Configuration error?
ERROR: ‘patchprocess/*.*’ doesn’t match anything, but ‘*.*’ does. Perhaps
that’s what you mean?
Build step 'Archive the artifacts' changed build result to FAILURE
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[
https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804882#comment-14804882
]
TezQA commented on TEZ-2832:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12757191/TEZ-2832.2.patch
against master revision 35cf023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1151//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1151//console
This message is automatically generated.
> Support tests for both SimpleHistory logging and ATS logging
>
>
> Key: TEZ-2832
> URL: https://issues.apache.org/jira/browse/TEZ-2832
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch
>
>
> This will ensure that both kinds of logging continue to be valid for using
> with the cp analyzer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2816) Build with hadoop 2.4 fails
[
https://issues.apache.org/jira/browse/TEZ-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804700#comment-14804700
]
Jeff Zhang commented on TEZ-2816:
-
Sorry [~bikassaha] Will review it today.
> Build with hadoop 2.4 fails
> ---
>
> Key: TEZ-2816
> URL: https://issues.apache.org/jira/browse/TEZ-2816
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jeff Zhang
>Assignee: Bikas Saha
> Attachments: TEZ-2816.1.patch, TEZ-2816.2.patch, TEZ-2816.3.patch
>
>
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/170/console
> {noformat}
> Running org.apache.tez.analyzer.TestAnalyzer
> Tests run: 13, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 99.595 sec
> <<< FAILURE!
> testBasicInputFailureWithoutExit(org.apache.tez.analyzer.TestAnalyzer) Time
> elapsed: 6.276 sec <<< FAILURE!
> java.lang.AssertionError: v2 : 00_0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.tez.analyzer.TestAnalyzer.verifyCriticalPath(TestAnalyzer.java:273)
> at
> org.apache.tez.analyzer.TestAnalyzer.runDAGAndVerify(TestAnalyzer.java:220)
> at
> org.apache.tez.analyzer.TestAnalyzer.testBasicInputFailureWithoutExit(TestAnalyzer.java:399)
> testCascadingInputFailureWithExitSuccess(org.apache.tez.analyzer.TestAnalyzer)
> Time elapsed: 5.986 sec <<< FAILURE!
> java.lang.AssertionError: v3 : 00_1
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.tez.analyzer.TestAnalyzer.verifyCriticalPath(TestAnalyzer.java:273)
> at
> org.apache.tez.analyzer.TestAnalyzer.runDAGAndVerify(TestAnalyzer.java:220)
> at
> org.apache.tez.analyzer.TestAnalyzer.testCascadingInputFailureWithExitSuccess(TestAnalyzer.java:561)
> Results :
> Failed tests:
>
> TestAnalyzer.testBasicInputFailureWithoutExit:399->runDAGAndVerify:220->verifyCriticalPath:273
> v2 : 00_0
>
> TestAnalyzer.testCascadingInputFailureWithExitSuccess:561->runDAGAndVerify:220->verifyCriticalPath:273
> v3 : 00_1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2775) Reduce logging in runtime components
[ https://issues.apache.org/jira/browse/TEZ-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2775: -- Attachment: TEZ-2775.3.txt - Added logProgress() in ShuffleEventHandler. Added it in UnorderedKVInput.close(), ShuffleEventHandlerImpl.handleEvent, ShuffleInputEventHandlerOrderedGrouped.handleEvent. By default it should print every 50 times; it would print it before closing as well. - Added logProgress() in ShuffleScheduler's close() call as well. In case of any abrupt failures, it should log how many are copied. - Retained logs in MergeManager.closeInMemoryFile. - Left "FetcherOrderedGrouped for decomp len" unchanged as it ends up occupying lots of space. However, this information would be useful for debugging corner cases issues of downloading multiple attempts from same connection. (Since it is not a common scenario, retaining at DEBUG level itself) - No changes to ShuffleUtils:logIndividualFetchComplete() (need to change perf tool later) Without Patch: Query_75 @ 10 TB scale (with hive patch for l4j in SMB) application_1439860407967_1259: 418,061,352 (compressed) : 6,375,796,132 (uncompressed) With .1 Patch: application_1439860407967_1260: 142,492,745 (compressed) : 1,365,908,923 (uncompressed) With .3 patch: application_1439860407967_1280: 219,133,295 (compressed) : 2,410,892,797 (uncompressed) > Reduce logging in runtime components > > > Key: TEZ-2775 > URL: https://issues.apache.org/jira/browse/TEZ-2775 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2775.1.txt, TEZ-2775.3.txt > > > Specifically Shuffle, which logs a lot for each event being processed and > data being fetched. > Also PipelinedShuffle is fairly noisy - some of the information from here > could be consolidated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2774) Reduce logging in the AM, and parts of the runtime
[ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804760#comment-14804760 ] Hitesh Shah commented on TEZ-2774: -- +1 > Reduce logging in the AM, and parts of the runtime > -- > > Key: TEZ-2774 > URL: https://issues.apache.org/jira/browse/TEZ-2774 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 0.8.1 > > Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt, > TEZ-2774.4.patch, TEZ-2774.5.patch, TEZ-2774.addendum.2.patch, > TEZ-2774.addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2840) MRInputLegacy.init should set splitInfoViaEvents
[ https://issues.apache.org/jira/browse/TEZ-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804944#comment-14804944 ] Hitesh Shah commented on TEZ-2840: -- +1 - committing shortly. > MRInputLegacy.init should set splitInfoViaEvents > > > Key: TEZ-2840 > URL: https://issues.apache.org/jira/browse/TEZ-2840 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-2840.patch > > > Only one path that uses initializeInternal and thus this field actually sets > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805008#comment-14805008 ] Rajesh Balamohan commented on TEZ-2832: --- lgtm. +1. Were the changes related to "tracking multiple waves that were run on the same container" added for any specific debugging reason?. > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-814) Improve heuristic for determining a task has failed outputs
[ https://issues.apache.org/jira/browse/TEZ-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805024#comment-14805024 ] Bikas Saha commented on TEZ-814: Thanks! I will backport this to 0.5 > Improve heuristic for determining a task has failed outputs > --- > > Key: TEZ-814 > URL: https://issues.apache.org/jira/browse/TEZ-814 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.7.1 > > Attachments: TEZ-814.1.patch, TEZ-814.2.patch > > > Currently 25% of consumers need to report failure. However we may not always > have those many error reports. Eg. this is the last consumer and it the > source is lost. Or some consumers are cut off from the source. The job may > hang on those consumers waiting for a re-run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2832) Support tests for both SimpleHistory logging and ATS logging
[ https://issues.apache.org/jira/browse/TEZ-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805026#comment-14805026 ] Bikas Saha commented on TEZ-2832: - Thanks! I am trying to improve the common case where a large allocation overhead shows up in the critical path. Running multiple waves of a given vertex would imply that the job could run faster if it had more capacity. This is just a first step. We could probably analyze the container allocations to derive more insights. > Support tests for both SimpleHistory logging and ATS logging > > > Key: TEZ-2832 > URL: https://issues.apache.org/jira/browse/TEZ-2832 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-2832.1.patch, TEZ-2832.2.patch > > > This will ensure that both kinds of logging continue to be valid for using > with the cp analyzer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
