[jira] [Created] (TEZ-2384) Add warning message in the case of prewarn under non-session mode
Jeff Zhang created TEZ-2384: --- Summary: Add warning message in the case of prewarn under non-session mode Key: TEZ-2384 URL: https://issues.apache.org/jira/browse/TEZ-2384 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2383) release sort buffers on close
[
https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518825#comment-14518825
]
TezQA commented on TEZ-2383:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729042/TEZ-2383.1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/571//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/571//console
This message is automatically generated.
> release sort buffers on close
> -
>
> Key: TEZ-2383
> URL: https://issues.apache.org/jira/browse/TEZ-2383
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2383.1.patch
>
>
> Currently they get released when sorter object gets GC-ed, but it might be
> good to explicitly release them on close as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2383) release sort buffers on close
[ https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2383: -- Attachment: TEZ-2383.2.patch Attaching revised patch, which cleans up the processorcontext in LogicalIOProcessorRuntimeTask. This would also remove any stale references to objects related to inputs/outputs, which would internally release the memory buffers as well. [~gopalv], [~sseth] - Please review. > release sort buffers on close > - > > Key: TEZ-2383 > URL: https://issues.apache.org/jira/browse/TEZ-2383 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2383.1.patch, TEZ-2383.2.patch > > > Currently they get released when sorter object gets GC-ed, but it might be > good to explicitly release them on close as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1564) State machine error: Invalid event: T_SCHEDULE at SCHEDULED
[
https://issues.apache.org/jira/browse/TEZ-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518772#comment-14518772
]
Jeff Zhang edited comment on TEZ-1564 at 4/29/15 5:52 AM:
--
Check it again, vertex (Map 5) is scheduled 2 times. And both are in
onSourceTaskCompleted, and in the same thread (AsyncDispatcher event handler),
not sure why it is scheduled 2 times. [~rajesh.balamohan] Have you meet this
error again ?
{noformat}
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_05_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
2014-09-09 21:47:16,755 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_06_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,756 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
{noformat}
was (Author: zjffdu):
Check it again, vertex (Map 5) is scheduled 2 times. And both are in
onSourceTaskCompleted, and in the same thread (AsyncDispatcher event handler),
not sure why it is scheduled 2 times. [~rajesh.balamohan] Have you see this
error again ?
{noformat}
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_05_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
2014-09-09 21:47:16,755 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_06_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,756 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
{noformat}
> State machine error: Invalid event: T_SCHEDULE at SCHEDULED
> ---
>
> Key: TEZ-1564
> URL: https://issues.apache.org/jira/browse/TEZ-1564
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Critical
> Attachments: applogs.txt.tar.gz, dag.dot
>
>
> ERROR [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.TaskImpl: Can't handle this event at current
> state for task_1409722953518_0162_1_07_00
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> T_SCHEDULE at SCHEDULED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:827)
> at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:95)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1604)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1590)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:724)
> I will attach the dag + app logs soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2377) RandomWriter ends up using TextOutputFormat instead of SequenceFileOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518785#comment-14518785
]
Jeff Zhang commented on TEZ-2377:
-
This issue only happen when translating MR job to Tez. MROutput may be
associated with either mapper or reducer while MRInput can only be associated
with mapper. So MRInput don't have this kind of issue.
> RandomWriter ends up using TextOutputFormat instead of
> SequenceFileOutputFormat
> ---
>
> Key: TEZ-2377
> URL: https://issues.apache.org/jira/browse/TEZ-2377
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2377.1.patch
>
>
> {code}
> yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar randomwriter
> "-Dmapreduce.randomwriter.totalbytes=10737418" /tmp/test1
> {code}
> This ends up generating TextOutputFormat.
> {code}
> yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar sort
> "-Dmapreduce.framework.name=yarn-tez" -r 5 /tmp/test1 /tmp/test_sorted
> {code}
> This ends up throwing error
> {noformat}
> Failure while running task:java.io.IOException:
> hdfs://tez-vm:56565/tmp/test1/part-0 not a SequenceFile
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1851)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1811)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1760)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1774)
> at
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapReduce.setupNewRecordReader(MRReaderMapReduce.java:149)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapReduce.(MRReaderMapReduce.java:78)
> at
> org.apache.tez.mapreduce.input.MRInput.initializeInternal(MRInput.java:475)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2360 PreCommit Build #572
Jira: https://issues.apache.org/jira/browse/TEZ-2360
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/572/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2770 lines...]
[INFO] Final Memory: 70M/927M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729055/TEZ-2360.3.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/572//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/572//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
022980fbbe4695093812a1faba1e540b6ec43076 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #570
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2517319 bytes
Compression is 9.4%
Took 2.1 sec
Description set: TEZ-2360
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Failed: TEZ-2383 PreCommit Build #571
Jira: https://issues.apache.org/jira/browse/TEZ-2383
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/571/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2772 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729042/TEZ-2383.1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/571//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/571//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
414e8a49d79c073eb823ed76f601ef03de55966a logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #570
Archived 44 artifacts
Archive block size is 32768
Received 26 blocks and 1896706 bytes
Compression is 31.0%
Took 1.1 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Failed: TEZ-2384 PreCommit Build #574
Jira: https://issues.apache.org/jira/browse/TEZ-2384
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/574/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2557 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729070/TEZ-2384-1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/574//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/574//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
15e0fe36ef162c519da41cc592a9336703adf0a3 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #573
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2476842 bytes
Compression is 9.6%
Took 1.7 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
6 tests failed.
REGRESSION:
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:675)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:118)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:248)
REGRESSION:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:675)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:118)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices(TestFaultTolerance.java:672)
REGRESSION:
org.apache.tez.test.TestFaultTolerance.testMultipleInputFailureWithoutExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:675)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:118)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testMultipleInputFailureWithoutExit(TestFaultTolerance.java:297)
REGRESS
Success: TEZ-2305 PreCommit Build #570
Jira: https://issues.apache.org/jira/browse/TEZ-2305
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/570/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2771 lines...]
[INFO] Final Memory: 72M/927M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729029/TEZ-2305-5.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/570//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/570//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
727b15df0e299f69213c0f6f9398ccfca7a3343b logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #567
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2556723 bytes
Compression is 7.1%
Took 1.3 sec
Description set: TEZ-2305
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[
https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518892#comment-14518892
]
TezQA commented on TEZ-1521:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729064/TEZ-1521-1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/573//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/573//console
This message is automatically generated.
> VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
> ---
>
> Key: TEZ-1521
> URL: https://issues.apache.org/jira/browse/TEZ-1521
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: TEZ-1521-1.patch
>
>
> The TezEvents may be added to pendingTaskEvents and route again later when
> task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent
> will been logged twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-1521 PreCommit Build #573
Jira: https://issues.apache.org/jira/browse/TEZ-1521
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/573/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2769 lines...]
[INFO] Final Memory: 69M/915M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729064/TEZ-1521-1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/573//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/573//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4280f658357a0e0fcaef97a6d477bfbddb923e33 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #572
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2489034 bytes
Compression is 9.5%
Took 2.2 sec
Description set: TEZ-1521
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2384) Add warning message in the case of prewarn under non-session mode
[ https://issues.apache.org/jira/browse/TEZ-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2384: Attachment: TEZ-2384-1.patch Attach the trivial patch. > Add warning message in the case of prewarn under non-session mode > - > > Key: TEZ-2384 > URL: https://issues.apache.org/jira/browse/TEZ-2384 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Minor > Attachments: TEZ-2384-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2271) OutputFormat is not recognized correctly
[
https://issues.apache.org/jira/browse/TEZ-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang resolved TEZ-2271.
-
Resolution: Duplicate
Resolved in TEZ-2305
> OutputFormat is not recognized correctly
> -
>
> Key: TEZ-2271
> URL: https://issues.apache.org/jira/browse/TEZ-2271
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Critical
>
> TEZ-2162 didn't resolve it completely. Paste the comment from TEZ-2162
> bq. But there's one exceptional case that if user specify OutputFormat
> through conf, then even MROutput is on mapper side, user still need to use
> property MRJobConfig.NEW_API_REDUCER_CONFIG rather than
> NEW_API_MAPPER_CONFIG. Or I think we can take useNewAPI as true as any of
> NEW_API_MAPPER_CONFIG, NEW_API_REDUCER_CONFIG is set true (patch didn't
> include this now)
> This issue happens when translating from Mapper-only MR job to tez.
> {code}
> 12:50:07,425 - Thread( main) - (Job.java:1386) - Job job_1427876879456_0010
> failed with state FAILED due to: Vertex failed, vertexName=initialmap,
> vertexId=vertex_1427876879456_0010_1_00, diagnostics=[Task failed,
> taskId=task_1427876879456_0010_1_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.lang.ClassCastException:
> org.apache.hadoop.mapred.FileOutputCommitter cannot be cast to
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getWorkOutputPath(FileOutputFormat.java:222)
> at
> org.apache.sqoop.mapreduce.TextImportMapper.setup(TextImportMapper.java:48)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at
> org.apache.tez.mapreduce.processor.map.MapProcessor.runNewMapper(MapProcessor.java:237)
> at
> org.apache.tez.mapreduce.processor.map.MapProcessor.run(MapProcessor.java:124)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:330)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2383) release sort buffers on close
[ https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2383: -- Attachment: TEZ-2383.3.patch Added null checks. > release sort buffers on close > - > > Key: TEZ-2383 > URL: https://issues.apache.org/jira/browse/TEZ-2383 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2383.1.patch, TEZ-2383.2.patch, TEZ-2383.3.patch > > > Currently they get released when sorter object gets GC-ed, but it might be > good to explicitly release them on close as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2305) MR compatibility sleep job fails with IOException: Undefined job output-path
[
https://issues.apache.org/jira/browse/TEZ-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518767#comment-14518767
]
TezQA commented on TEZ-2305:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729029/TEZ-2305-5.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/570//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/570//console
This message is automatically generated.
> MR compatibility sleep job fails with IOException: Undefined job output-path
>
>
> Key: TEZ-2305
> URL: https://issues.apache.org/jira/browse/TEZ-2305
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Tassapol Athiapinya
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: TEZ-2305-3.patch, TEZ-2305-4.patch, TEZ-2305-5.patch,
> TEZ-2305.1.patch, TEZ-2305.2.patch
>
>
> Running MR sleep job has an IOException.
> {code}
> 15/04/09 20:52:25 INFO mapreduce.Job: Job job_1428612196442_0002 failed with
> state FAILED due to: Vertex failed, vertexName=initialmap,
> vertexId=vertex_1428612196442_0002_1_00, diagnostics=[Task failed,
> taskId=task_1428612196442_0002_1_00_01, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.io.IOException:
> Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 2 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> o
[jira] [Updated] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2084: -- Attachment: Screen Shot 2015-04-29 at 2.10.20 PM.png TEZ-2084.1.patch [~zjffdu] attaching the patch and the screenshot. [~Sreenath] please review. > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519093#comment-14519093 ] Jeff Zhang commented on TEZ-1521: - Attach patch, [~hitesh] Please help review > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Critical > Attachments: TEZ-1521-1.patch > > > The TezEvents may be added to pendingTaskEvents and route again later when > task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent > will been logged twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1564) State machine error: Invalid event: T_SCHEDULE at SCHEDULED
[ https://issues.apache.org/jira/browse/TEZ-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519099#comment-14519099 ] Rajesh Balamohan commented on TEZ-1564: --- [~zjffdu] - Haven't tried checking with other jobs (mainly happens when jobs are killed abruptly. Will update asap. > State machine error: Invalid event: T_SCHEDULE at SCHEDULED > --- > > Key: TEZ-1564 > URL: https://issues.apache.org/jira/browse/TEZ-1564 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Critical > Attachments: applogs.txt.tar.gz, dag.dot > > > ERROR [AsyncDispatcher event handler] > org.apache.tez.dag.app.dag.impl.TaskImpl: Can't handle this event at current > state for task_1409722953518_0162_1_07_00 > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > T_SCHEDULE at SCHEDULED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:827) > at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:95) > at > org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1604) > at > org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1590) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) > at java.lang.Thread.run(Thread.java:724) > I will attach the dag + app logs soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2384) Add warning message in the case of prewarn under non-session mode
[
https://issues.apache.org/jira/browse/TEZ-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518896#comment-14518896
]
TezQA commented on TEZ-2384:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729070/TEZ-2384-1.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/574//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/574//console
This message is automatically generated.
> Add warning message in the case of prewarn under non-session mode
> -
>
> Key: TEZ-2384
> URL: https://issues.apache.org/jira/browse/TEZ-2384
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Minor
> Attachments: TEZ-2384-1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1564) State machine error: Invalid event: T_SCHEDULE at SCHEDULED
[
https://issues.apache.org/jira/browse/TEZ-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518772#comment-14518772
]
Jeff Zhang commented on TEZ-1564:
-
Check it again, vertex (Map 5) is scheduled 2 times. And both are in
onSourceTaskCompleted, and in the same thread (AsyncDispatcher event handler),
not sure why it is scheduled 2 times. [~rajesh.balamohan] Have you see this
error again ?
{noformat}
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_05_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,743 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
2014-09-09 21:47:16,755 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Source task attempt completed for
vertex: vertex_1409722953518_0162_1_07 attempt:
attempt_1409722953518_0162_1_06_06_0 with state: SUCCEEDED vertexState:
RUNNING
2014-09-09 21:47:16,756 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager: Starting 1 in Map 5
{noformat}
> State machine error: Invalid event: T_SCHEDULE at SCHEDULED
> ---
>
> Key: TEZ-1564
> URL: https://issues.apache.org/jira/browse/TEZ-1564
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Critical
> Attachments: applogs.txt.tar.gz, dag.dot
>
>
> ERROR [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.TaskImpl: Can't handle this event at current
> state for task_1409722953518_0162_1_07_00
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> T_SCHEDULE at SCHEDULED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:827)
> at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:95)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1604)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1590)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:724)
> I will attach the dag + app logs soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-2084: - Assignee: Prakash Ramachandran > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2377) RandomWriter ends up using TextOutputFormat instead of SequenceFileOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518785#comment-14518785
]
Jeff Zhang edited comment on TEZ-2377 at 4/29/15 6:01 AM:
--
This issue only happens when translating MR job to Tez. MROutput may be
associated with either mapper or reducer while MRInput can only be associated
with mapper. So MRInput don't have this kind of issue.
was (Author: zjffdu):
This issue only happen when translating MR job to Tez. MROutput may be
associated with either mapper or reducer while MRInput can only be associated
with mapper. So MRInput don't have this kind of issue.
> RandomWriter ends up using TextOutputFormat instead of
> SequenceFileOutputFormat
> ---
>
> Key: TEZ-2377
> URL: https://issues.apache.org/jira/browse/TEZ-2377
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2377.1.patch
>
>
> {code}
> yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar randomwriter
> "-Dmapreduce.randomwriter.totalbytes=10737418" /tmp/test1
> {code}
> This ends up generating TextOutputFormat.
> {code}
> yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar sort
> "-Dmapreduce.framework.name=yarn-tez" -r 5 /tmp/test1 /tmp/test_sorted
> {code}
> This ends up throwing error
> {noformat}
> Failure while running task:java.io.IOException:
> hdfs://tez-vm:56565/tmp/test1/part-0 not a SequenceFile
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1851)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1811)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1760)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1774)
> at
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapReduce.setupNewRecordReader(MRReaderMapReduce.java:149)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapReduce.(MRReaderMapReduce.java:78)
> at
> org.apache.tez.mapreduce.input.MRInput.initializeInternal(MRInput.java:475)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2384) Add warning message in the case of prewarn under non-session mode
[ https://issues.apache.org/jira/browse/TEZ-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2384: Priority: Minor (was: Major) > Add warning message in the case of prewarn under non-session mode > - > > Key: TEZ-2384 > URL: https://issues.apache.org/jira/browse/TEZ-2384 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2305) MR compatibility sleep job fails with IOException: Undefined job output-path
[
https://issues.apache.org/jira/browse/TEZ-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519104#comment-14519104
]
Jeff Zhang commented on TEZ-2305:
-
Thanks [~hitesh] Committed to 0.5, 0.6 & master
> MR compatibility sleep job fails with IOException: Undefined job output-path
>
>
> Key: TEZ-2305
> URL: https://issues.apache.org/jira/browse/TEZ-2305
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Tassapol Athiapinya
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: TEZ-2305-3.patch, TEZ-2305-4.patch, TEZ-2305-5.patch,
> TEZ-2305.1.patch, TEZ-2305.2.patch
>
>
> Running MR sleep job has an IOException.
> {code}
> 15/04/09 20:52:25 INFO mapreduce.Job: Job job_1428612196442_0002 failed with
> state FAILED due to: Vertex failed, vertexName=initialmap,
> vertexId=vertex_1428612196442_0002_1_00, diagnostics=[Task failed,
> taskId=task_1428612196442_0002_1_00_01, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.io.IOException:
> Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 2 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hado
[jira] [Updated] (TEZ-2384) Add warning message in the case of prewarn under non-session mode
[ https://issues.apache.org/jira/browse/TEZ-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2384: Issue Type: Improvement (was: Bug) > Add warning message in the case of prewarn under non-session mode > - > > Key: TEZ-2384 > URL: https://issues.apache.org/jira/browse/TEZ-2384 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1521: Attachment: TEZ-1521-1.patch > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Critical > Attachments: TEZ-1521-1.patch > > > The TezEvents may be added to pendingTaskEvents and route again later when > task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent > will been logged twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2360) per-io counters flag should generate both overall and per-edge counters
[
https://issues.apache.org/jira/browse/TEZ-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518849#comment-14518849
]
TezQA commented on TEZ-2360:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729055/TEZ-2360.3.patch
against master revision 5b2f011.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/572//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/572//console
This message is automatically generated.
> per-io counters flag should generate both overall and per-edge counters
>
>
> Key: TEZ-2360
> URL: https://issues.apache.org/jira/browse/TEZ-2360
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Prakash Ramachandran
> Attachments: TEZ-2360.1.patch, TEZ-2360.2.patch, TEZ-2360.3.patch
>
>
> Currently, the per-io flag disables overall per task counters and retains
> only per edge counters. It would be useful to have both overall and per edge
> counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519134#comment-14519134 ] Sreenath Somarajapuram commented on TEZ-2084: - Just adding the style white-space: pre; into div.indent must do the trick. > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents
[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519181#comment-14519181
]
Rajesh Balamohan commented on TEZ-776:
--
- In Edge, "routeInputSourceTaskFailedEventToDestination" is checked for
enabling onDemandRouting. Would it an issue if the custom edgeManagers do not
have routeInputSourceTaskFailedEventToDestination or prepareForRouting() etc
missed out?
- In Edge.java, would be the following code be an issue if we have bigger
PartitionRange (corner case) in ShuffleVertexManager's
CustomShuffleEdgeManager?
{noformat}
if (routeMeta.getNumEvents() + listToAdd.size() > listMaxSize) {
return false;
}
{noformat}
- Common code in the
ShuffleVertexManager.CustomShuffleEdgeManager.prepareRouting() can be moved to
separate method "private int[] createIndex(int taskIndex, int partitionRange)"?
That can possibly reduce the code in prepareRouting and
createTargetIndicesForRemainder might not be needed in that case
- In OneToOneEdgeManager,commonRouteMeta can be init-ed in prepareForRouting()
similar to other managers (mainly for consistency)
- Should we throw exceptions in routeDataMovementEventToDestination in
OneToOneEdgeManager when sourceTaskIndex != destinationTaskIndex?. Or there can
be instances when null is acceptable?
- eventIndicesCreated, sourceIndicesRemainder can be removed in
ShuffleVertexManager
- BroadcastEdgeManager - cachedEventsLock, cachedEvents may not be relevant
- EdgeManagerPlugin - unwanted imports can be removed
> Reduce AM mem usage caused by storing TezEvents
> ---
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
> TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
> TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch,
> TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch,
> TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2383 PreCommit Build #575
Jira: https://issues.apache.org/jira/browse/TEZ-2383
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/575/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2776 lines...]
[INFO] Final Memory: 78M/1030M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729093/TEZ-2383.3.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/575//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/575//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
2df10484b417fd388d2e07e985bc656ecbae02c6 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #573
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2493156 bytes
Compression is 9.5%
Took 2.1 sec
Description set: TEZ-2383
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2383) release sort buffers on close
[
https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519210#comment-14519210
]
TezQA commented on TEZ-2383:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729093/TEZ-2383.3.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/575//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/575//console
This message is automatically generated.
> release sort buffers on close
> -
>
> Key: TEZ-2383
> URL: https://issues.apache.org/jira/browse/TEZ-2383
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2383.1.patch, TEZ-2383.2.patch, TEZ-2383.3.patch
>
>
> Currently they get released when sorter object gets GC-ed, but it might be
> good to explicitly release them on close as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-2385) branch-0.6 compile failure
Jeff Zhang created TEZ-2385: --- Summary: branch-0.6 compile failure Key: TEZ-2385 URL: https://issues.apache.org/jira/browse/TEZ-2385 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
Prakash Ramachandran created TEZ-2386: - Summary: Tez UI: Inconsistent usage of icon colors Key: TEZ-2386 URL: https://issues.apache.org/jira/browse/TEZ-2386 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Prakash Ramachandran Assignee: Prakash Ramachandran if there's failed attempts in a DAG, and it succeeds - an orange icon shows up on the DAG page. This is very useful to identify DAGs which may need some debugging. However, the color is Green for Vertex / Task views after this - so it's difficult to know which one actually had problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2383) release sort buffers on close
[
https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519290#comment-14519290
]
TezQA commented on TEZ-2383:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729093/TEZ-2383.3.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/576//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/576//console
This message is automatically generated.
> release sort buffers on close
> -
>
> Key: TEZ-2383
> URL: https://issues.apache.org/jira/browse/TEZ-2383
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2383.1.patch, TEZ-2383.2.patch, TEZ-2383.3.patch
>
>
> Currently they get released when sorter object gets GC-ed, but it might be
> good to explicitly release them on close as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2383 PreCommit Build #576
Jira: https://issues.apache.org/jira/browse/TEZ-2383
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/576/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2776 lines...]
[INFO] Final Memory: 71M/967M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729093/TEZ-2383.3.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/576//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/576//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
07aee87627b0991f5b22ebed4ef838a86cf5b595 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #575
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2488615 bytes
Compression is 9.5%
Took 1.5 sec
Description set: TEZ-2383
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2378) In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size
[
https://issues.apache.org/jira/browse/TEZ-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519294#comment-14519294
]
Rajesh Balamohan commented on TEZ-2378:
---
These are normal exceptions when fetcher is unable to get the data from local
system, but having these exceptions in the logs might lead to distractions when
debugging some other issue.
> In case Fetcher (unordered) fails to do local fetch, log in debug mode to
> reduce log size
> -
>
> Key: TEZ-2378
> URL: https://issues.apache.org/jira/browse/TEZ-2378
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> Following can be logged as debug mode as opposed to WARN level. May be
> counters can be added later to track the number of times it failed to do
> local-fetch.
> {noformat}
> 2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to
> shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier
> [inputIndex=81], attemptNumber=0,
> pathComponent=attempt_1429683757595_0485_1_03_81_0_10003,
> fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from
> cn047-10.l42scl.hortonworks.com(local fetch)
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429683757595_0485_1_03_81_0_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Description: due to TEZ-2226 > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang > > due to TEZ-2226 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Priority: Critical (was: Major) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Affects Version/s: 0.6.1 > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Target Version/s: 0.6.1 > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Description: due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6 (was: due to TEZ-2226. DAG/Vertex level configuration is not available in 00.6) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Description: due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may need to use the AM configuration. (was: due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Description: due to TEZ-2226. DAG/Vertex level configuration is not available in 00.6 (was: due to TEZ-2226) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226. DAG/Vertex level configuration is not available in 00.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519327#comment-14519327 ] Prakash Ramachandran commented on TEZ-2084: --- hi [~Sreenath] using pre it was difficult to control the spacing, hence used the margin. > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
[ https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2386: -- Attachment: TEZ-2386.wip.1.patch Hi [~Sreenath] can you have a look? one issue i faced was the icons were not getting positioned properly (due to position absolute in task-status class). similar issue with the graphical view. > Tez UI: Inconsistent usage of icon colors > - > > Key: TEZ-2386 > URL: https://issues.apache.org/jira/browse/TEZ-2386 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prakash Ramachandran >Assignee: Prakash Ramachandran > Attachments: TEZ-2386.wip.1.patch > > > if there's failed attempts in a DAG, and it succeeds - an orange icon shows > up on the DAG page. This is very useful to identify DAGs which may need some > debugging. > However, the color is Green for Vertex / Task views after this - so it's > difficult to know which one actually had problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2385: Description: due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may need to use the AM configuration or either back port TEZ-1233 to 0.6 (was: due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may need to use the AM configuration.) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Priority: Critical > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
[ https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2386: -- Attachment: TEZ-2386.1.patch changes * fixed the warning icon to be consistent * fixed few issues in position of icon in index/graphical view page * fixed links to failed tasks/attempts [~Sreenath] can you review? * added failed tasks count to history event [~hitesh]/[~zjffdu] can you review the java changes? > Tez UI: Inconsistent usage of icon colors > - > > Key: TEZ-2386 > URL: https://issues.apache.org/jira/browse/TEZ-2386 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prakash Ramachandran >Assignee: Prakash Ramachandran > Attachments: TEZ-2386.1.patch, TEZ-2386.wip.1.patch > > > if there's failed attempts in a DAG, and it succeeds - an orange icon shows > up on the DAG page. This is very useful to identify DAGs which may need some > debugging. > However, the color is Green for Vertex / Task views after this - so it's > difficult to know which one actually had problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2360) per-io counters flag should generate both overall and per-edge counters
[ https://issues.apache.org/jira/browse/TEZ-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519421#comment-14519421 ] Prakash Ramachandran commented on TEZ-2360: --- [~bikassaha]/[~sseth]/[~hitesh] can you review? > per-io counters flag should generate both overall and per-edge counters > > > Key: TEZ-2360 > URL: https://issues.apache.org/jira/browse/TEZ-2360 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Prakash Ramachandran > Attachments: TEZ-2360.1.patch, TEZ-2360.2.patch, TEZ-2360.3.patch > > > Currently, the per-io flag disables overall per task counters and retains > only per edge counters. It would be useful to have both overall and per edge > counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519427#comment-14519427 ] Sreenath Somarajapuram commented on TEZ-2084: - Then cool. +1 LGTM. > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2380) Disable fall back to reading from timeline if timeline disabled
[ https://issues.apache.org/jira/browse/TEZ-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519461#comment-14519461 ] Prakash Ramachandran commented on TEZ-2380: --- +1 LGTM > Disable fall back to reading from timeline if timeline disabled > > > Key: TEZ-2380 > URL: https://issues.apache.org/jira/browse/TEZ-2380 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Chang Li > Attachments: TEZ-2380.1.patch > > > As part of TEZ-2226, timeline logging can get disabled in cases where domain > creation fails. In this scenario, the dag client shoudl also not fall back to > reading from timeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2386 PreCommit Build #577
Jira: https://issues.apache.org/jira/browse/TEZ-2386
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/577/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2157 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729178/TEZ-2386.1.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.TestFetcher
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/577//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/577//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7a5dfa6a1d8a1d68744a7c3b1aad05823fd09dcc logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #576
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2512398 bytes
Compression is 7.3%
Took 4.5 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.tez.runtime.library.common.shuffle.TestFetcher.testSetupLocalDiskFetch
Error Message:
test timed out after 3000 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 3000 milliseconds
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493)
at java.lang.Class.getConstructor0(Class.java:2803)
at java.lang.Class.newInstance(Class.java:345)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2563)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:341)
at
org.apache.tez.runtime.library.common.shuffle.LocalDiskFetchedInput.(LocalDiskFetchedInput.java:50)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:538)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.setupLocalDiskFetch(Fetcher.java:517)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher$$EnhancerByMockitoWithCGLIB$$c02f6577.CGLIB$setupLocalDiskFetch$0()
at
org.apache.tez.runtime.library.common.shuffle.Fetcher$$EnhancerByMockitoWithCGLIB$$c02f6577$$FastClassByMockitoWithCGLIB$$5c87c82f.invoke()
at org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
at
org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
at
org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
at
org.mockito.internal.invocation.realmethod.FilteredC
[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
[
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519474#comment-14519474
]
TezQA commented on TEZ-2386:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729178/TEZ-2386.1.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.TestFetcher
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/577//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/577//console
This message is automatically generated.
> Tez UI: Inconsistent usage of icon colors
> -
>
> Key: TEZ-2386
> URL: https://issues.apache.org/jira/browse/TEZ-2386
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Prakash Ramachandran
>Assignee: Prakash Ramachandran
> Attachments: TEZ-2386.1.patch, TEZ-2386.wip.1.patch
>
>
> if there's failed attempts in a DAG, and it succeeds - an orange icon shows
> up on the DAG page. This is very useful to identify DAGs which may need some
> debugging.
> However, the color is Green for Vertex / Task views after this - so it's
> difficult to know which one actually had problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519476#comment-14519476 ] Hitesh Shah commented on TEZ-1521: -- Aren't events put in pending queue only on the destination vertex? Or are there some events also now being queued on the source vertex? For source vertex, we log events to recovery only when they are generated by the input initializer, tasks, etc. Could you explain what you observed with the events being logged twice? i.e. which events are pushed to the pending queue and end up getting logged twice? It will be good to understand which kind of events are being queued up and how we should handle them. > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Critical > Attachments: TEZ-1521-1.patch > > > The TezEvents may be added to pendingTaskEvents and route again later when > task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent > will been logged twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2387) TEZ-2226 broke branch 0.6
Hitesh Shah created TEZ-2387: Summary: TEZ-2226 broke branch 0.6 Key: TEZ-2387 URL: https://issues.apache.org/jira/browse/TEZ-2387 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned TEZ-2385: Assignee: Hitesh Shah > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2387) TEZ-2226 broke branch 0.6
[ https://issues.apache.org/jira/browse/TEZ-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved TEZ-2387. -- Resolution: Duplicate > TEZ-2226 broke branch 0.6 > -- > > Key: TEZ-2387 > URL: https://issues.apache.org/jira/browse/TEZ-2387 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-1776) TA_CONTAINER_TERMINATING event should not always fail the task attempt
[ https://issues.apache.org/jira/browse/TEZ-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-1776: - Assignee: Prakash Ramachandran > TA_CONTAINER_TERMINATING event should not always fail the task attempt > -- > > Key: TEZ-1776 > URL: https://issues.apache.org/jira/browse/TEZ-1776 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Prakash Ramachandran >Priority: Critical > > This is sometime sent when the node fails or other non-task related container > failures. For those cases the attempt should transition to killed instead of > failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
[ https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2386: -- Attachment: TEZ-2386.2.patch reattaching the patch as UT failed due to unrelated reason (timeout) > Tez UI: Inconsistent usage of icon colors > - > > Key: TEZ-2386 > URL: https://issues.apache.org/jira/browse/TEZ-2386 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prakash Ramachandran >Assignee: Prakash Ramachandran > Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.wip.1.patch > > > if there's failed attempts in a DAG, and it succeeds - an orange icon shows > up on the DAG page. This is very useful to identify DAGs which may need some > debugging. > However, the color is Green for Vertex / Task views after this - so it's > difficult to know which one actually had problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2388) Send dag identifier as part of the fetcher request string
Siddharth Seth created TEZ-2388: --- Summary: Send dag identifier as part of the fetcher request string Key: TEZ-2388 URL: https://issues.apache.org/jira/browse/TEZ-2388 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2388) Send dag identifier as part of the fetcher request string
[ https://issues.apache.org/jira/browse/TEZ-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2388: Attachment: TEZ-2388.1.txt > Send dag identifier as part of the fetcher request string > - > > Key: TEZ-2388 > URL: https://issues.apache.org/jira/browse/TEZ-2388 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-2388.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2388) Send dag identifier as part of the fetcher request string
[ https://issues.apache.org/jira/browse/TEZ-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved TEZ-2388. - Resolution: Fixed Fix Version/s: TEZ-2003 > Send dag identifier as part of the fetcher request string > - > > Key: TEZ-2388 > URL: https://issues.apache.org/jira/browse/TEZ-2388 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: TEZ-2003 > > Attachments: TEZ-2388.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1529) ATS and TezClient integration in secure kerberos enabled cluster
[
https://issues.apache.org/jira/browse/TEZ-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519545#comment-14519545
]
Steve Loughran commented on TEZ-1529:
-
oh, this is exactly what I'm looking for. I've been trying to work out how to
use the YARN {{TimelineClient}} to do the token renewal for the read side of
things
> ATS and TezClient integration in secure kerberos enabled cluster
> -
>
> Key: TEZ-1529
> URL: https://issues.apache.org/jira/browse/TEZ-1529
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Prakash Ramachandran
>Assignee: Prakash Ramachandran
>Priority: Blocker
> Attachments: TEZ-1529.1.patch
>
>
> This is a follow up for TEZ-1495 which address ATS - TezClient integration.
> however it does not enable it in secure kerberos enabled cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2386 PreCommit Build #578
Jira: https://issues.apache.org/jira/browse/TEZ-2386
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/578/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2791 lines...]
[INFO] Final Memory: 75M/1226M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729184/TEZ-2386.2.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/578//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/578//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
ac23e6e2e41a71f50840d5bba2006badeac22e77 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #576
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2550621 bytes
Compression is 7.2%
Took 2.2 sec
Description set: TEZ-2386
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors
[
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519598#comment-14519598
]
TezQA commented on TEZ-2386:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729184/TEZ-2386.2.patch
against master revision 9ba4b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/578//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/578//console
This message is automatically generated.
> Tez UI: Inconsistent usage of icon colors
> -
>
> Key: TEZ-2386
> URL: https://issues.apache.org/jira/browse/TEZ-2386
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Prakash Ramachandran
>Assignee: Prakash Ramachandran
> Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.wip.1.patch
>
>
> if there's failed attempts in a DAG, and it succeeds - an orange icon shows
> up on the DAG page. This is very useful to identify DAGs which may need some
> debugging.
> However, the color is Green for Vertex / Task views after this - so it's
> difficult to know which one actually had problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2084) TEZ-UI Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519676#comment-14519676 ] Prakash Ramachandran commented on TEZ-2084: --- [~jeagles] should this be checked into 0.6.1 too? > TEZ-UI Stacktrace format info is lost in diagnostics > > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2084) Tez UI: Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2084: -- Summary: Tez UI: Stacktrace format info is lost in diagnostics (was: TEZ-UI Stacktrace format info is lost in diagnostics) > Tez UI: Stacktrace format info is lost in diagnostics > - > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2084 PreCommit Build #579
Jira: https://issues.apache.org/jira/browse/TEZ-2084
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/579/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 25 lines...]
==
Testing patch for TEZ-2084.
==
==
HEAD is now at c76698e TEZ-2084. Tez UI: Stacktrace format info is lost in
diagnostics (pramachandran)
Previous HEAD position was c76698e... TEZ-2084. Tez UI: Stacktrace format info
is lost in diagnostics (pramachandran)
Switched to branch 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to c76698ea337bd8e95e31a5b953db72b56b614dd7.
TEZ-2084 patch is being downloaded at Wed Apr 29 17:40:27 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12729106/Screen%20Shot%202015-04-29%20at%202.10.20%20PM.png
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729106/Screen%20Shot%202015-04-29%20at%202.10.20%20PM.png
against master revision c76698e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/579//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6bb562aefc93a6f53d55b677e3b61c798bd69d0b logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-2084) Tez UI: Stacktrace format info is lost in diagnostics
[
https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519803#comment-14519803
]
TezQA commented on TEZ-2084:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729106/Screen%20Shot%202015-04-29%20at%202.10.20%20PM.png
against master revision c76698e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/579//console
This message is automatically generated.
> Tez UI: Stacktrace format info is lost in diagnostics
> -
>
> Key: TEZ-2084
> URL: https://issues.apache.org/jira/browse/TEZ-2084
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Prakash Ramachandran
> Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png,
> TEZ-2084.1.patch, am_log.png, diag_format.png, task_attemp_log.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2084) Tez UI: Stacktrace format info is lost in diagnostics
[ https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2084: -- Attachment: TEZ-2084.2.patch > Tez UI: Stacktrace format info is lost in diagnostics > - > > Key: TEZ-2084 > URL: https://issues.apache.org/jira/browse/TEZ-2084 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jeff Zhang >Assignee: Prakash Ramachandran > Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png, > TEZ-2084.1.patch, TEZ-2084.2.patch, am_log.png, diag_format.png, > task_attemp_log.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2378) In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size
[
https://issues.apache.org/jira/browse/TEZ-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519864#comment-14519864
]
Hitesh Shah commented on TEZ-2378:
--
bq. These are normal exceptions when fetcher is unable to get the data from
local system
I don't understand. Why is the fetcher failing to read off the local
filesystem? That should be a cause for concern I would assume. A distraction
yes when you know that there is something else wrong but a problem regardless
if this is happening in the first place.
Also, if the local fetch fails, do we error out as falling back to the http
fetch would likely hit the same error?
bq. 2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed
to shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier
[inputIndex=81], attemptNumber=0,
pathComponent=attempt_1429683757595_0485_1_03_81_0_10003,
fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from
cn047-10.l42scl.hortonworks.com(local fetch)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/attempt_1429683757595_0485_1_03_81_0_10003/file.out.index in any of
the configured local directories
Also, I am assuming that we are not making calls to the local fetcher when the
data is remote or if the output size is 0? So, the above error should only be
seen when the map output somehow disappeared off the local disk?
> In case Fetcher (unordered) fails to do local fetch, log in debug mode to
> reduce log size
> -
>
> Key: TEZ-2378
> URL: https://issues.apache.org/jira/browse/TEZ-2378
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> Following can be logged as debug mode as opposed to WARN level. May be
> counters can be added later to track the number of times it failed to do
> local-fetch.
> {noformat}
> 2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to
> shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier
> [inputIndex=81], attemptNumber=0,
> pathComponent=attempt_1429683757595_0485_1_03_81_0_10003,
> fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from
> cn047-10.l42scl.hortonworks.com(local fetch)
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429683757595_0485_1_03_81_0_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2380) Disable fall back to reading from timeline if timeline disabled
[ https://issues.apache.org/jira/browse/TEZ-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519877#comment-14519877 ] Prakash Ramachandran commented on TEZ-2380: --- [~lichangleo] will wait for the 0.6.1 build failure fix before committing. > Disable fall back to reading from timeline if timeline disabled > > > Key: TEZ-2380 > URL: https://issues.apache.org/jira/browse/TEZ-2380 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Chang Li > Attachments: TEZ-2380.1.patch > > > As part of TEZ-2226, timeline logging can get disabled in cases where domain > creation fails. In this scenario, the dag client shoudl also not fall back to > reading from timeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2084 PreCommit Build #580
Jira: https://issues.apache.org/jira/browse/TEZ-2084
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/580/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 19 lines...]
No emails were triggered.
[PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson7352122362743621841.sh
Running in Jenkins mode
==
==
Testing patch for TEZ-2084.
==
==
HEAD is now at c76698e TEZ-2084. Tez UI: Stacktrace format info is lost in
diagnostics (pramachandran)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Current branch master is up to date.
TEZ-2084 patch is being downloaded at Wed Apr 29 18:23:53 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12729241/TEZ-2084.2.patch
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729241/TEZ-2084.2.patch
against master revision c76698e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/580//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1b234baf2b00f0f60ca41e1d106961e968d76ae0 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-2084) Tez UI: Stacktrace format info is lost in diagnostics
[
https://issues.apache.org/jira/browse/TEZ-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519881#comment-14519881
]
TezQA commented on TEZ-2084:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729241/TEZ-2084.2.patch
against master revision c76698e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/580//console
This message is automatically generated.
> Tez UI: Stacktrace format info is lost in diagnostics
> -
>
> Key: TEZ-2084
> URL: https://issues.apache.org/jira/browse/TEZ-2084
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jeff Zhang
>Assignee: Prakash Ramachandran
> Attachments: Screen Shot 2015-04-29 at 2.10.20 PM.png,
> TEZ-2084.1.patch, TEZ-2084.2.patch, am_log.png, diag_format.png,
> task_attemp_log.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-2389) Tez UI: Sort by attempt-no is incorrect in attempts pages.
Sreenath Somarajapuram created TEZ-2389: --- Summary: Tez UI: Sort by attempt-no is incorrect in attempts pages. Key: TEZ-2389 URL: https://issues.apache.org/jira/browse/TEZ-2389 Project: Apache Tez Issue Type: Bug Reporter: Sreenath Somarajapuram -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2389) Tez UI: Sort by attempt-no is incorrect in attempts pages.
[ https://issues.apache.org/jira/browse/TEZ-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram reassigned TEZ-2389: --- Assignee: Sreenath Somarajapuram > Tez UI: Sort by attempt-no is incorrect in attempts pages. > -- > > Key: TEZ-2389 > URL: https://issues.apache.org/jira/browse/TEZ-2389 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2389) Tez UI: Sort by attempt-no is incorrect in attempts pages.
[ https://issues.apache.org/jira/browse/TEZ-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2389: Attachment: TEZ-2389.1.patch Cause: Getter to return the value to be sorted was missing for the column. [~pramachandran] Please review the patch. > Tez UI: Sort by attempt-no is incorrect in attempts pages. > -- > > Key: TEZ-2389 > URL: https://issues.apache.org/jira/browse/TEZ-2389 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2389.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2385: - Attachment: TEZ-2385.1.patch [~pramachandran] [~zjffdu] [~lichangleo] Please review. > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520077#comment-14520077 ] Chang Li commented on TEZ-2385: --- [~hitesh], thanks for the patch, it's a nice fix. Have tested the patch against branch-0.6 and confirmed that there is no more compile failure. +1 (non-binding) > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2366) Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
[
https://issues.apache.org/jira/browse/TEZ-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520082#comment-14520082
]
Siddharth Seth commented on TEZ-2366:
-
Relying on the shuffle port from config won't work because the config may not
actually be available to the Input.
This information can be fetched from the context - via the following.
{code}
ByteBuffer shuffleMetadata = context
.getServiceProviderMetaData(ShuffleUtils.SHUFFLE_HANDLER_SERVICE_ID);
int shufflePort =
ShuffleUtils.deserializeShuffleProviderMetaData(shuffleMetadata);
{code}
The incoming event shuffle port could be matched against this.
[~hitesh] - any thoughts on getting the NodeId ? I don't believe it's exposed
to containers - so would have to be sent via the context. The event doesn't
have this information either.
> Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
>
>
> Key: TEZ-2366
> URL: https://issues.apache.org/jira/browse/TEZ-2366
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Daniel Dai
>Priority: Critical
> Attachments: TEZ-2366.test.txt, TEZ-2366.wip.1.patch
>
>
> There are around 20 unit tests (out of around 2000) fail intermittently after
> TEZ-2333. Here is a stack:
> {code}
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429899954360_0001_1_01_00_1_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:611)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:591)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:536)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupLocalDiskFetch(Fetcher.java:517)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:190)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> To reproduce that in Pig test, using the following commands:
> svn co http://svn.apache.org/repos/asf/pig/trunk
> ant -Dhadoopversion=23 -Dtest.exec.type=tez -Dtestcase=TestTezAutoParallelism
> test
> Note in Pig codebase, we already set TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to
> "true"
> (http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java?view=markup).
> I tried changing TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to "false" in Pig and does
> not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2366) Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
[
https://issues.apache.org/jira/browse/TEZ-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520111#comment-14520111
]
Hitesh Shah commented on TEZ-2366:
--
Maybe an initial fix would be to just disable this for mini-cluster.
For a longer term fix: Exposing nodeId or some host identifier requires the
datamovement event to be updated at the source and then compared against the
hostIdentifier for the fetcher container.
The yarn node id should ideally be something that YARN injects into the env -
for now, we can add it to the env when we launch the container.
> Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
>
>
> Key: TEZ-2366
> URL: https://issues.apache.org/jira/browse/TEZ-2366
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Daniel Dai
>Priority: Critical
> Attachments: TEZ-2366.test.txt, TEZ-2366.wip.1.patch
>
>
> There are around 20 unit tests (out of around 2000) fail intermittently after
> TEZ-2333. Here is a stack:
> {code}
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429899954360_0001_1_01_00_1_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:611)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:591)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:536)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupLocalDiskFetch(Fetcher.java:517)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:190)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> To reproduce that in Pig test, using the following commands:
> svn co http://svn.apache.org/repos/asf/pig/trunk
> ant -Dhadoopversion=23 -Dtest.exec.type=tez -Dtestcase=TestTezAutoParallelism
> test
> Note in Pig codebase, we already set TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to
> "true"
> (http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java?view=markup).
> I tried changing TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to "false" in Pig and does
> not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2360) per-io counters flag should generate both overall and per-edge counters
[ https://issues.apache.org/jira/browse/TEZ-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520133#comment-14520133 ] Bikas Saha commented on TEZ-2360: - More descriptive names than counter1 and counter2 would be better. Should getUnderlyingCounter() return "this"? In the test, for one of the non-zero value counters can you please check that per edge counters have been correctly aggregated at the vertex level (e.g. shuffle_bytes at the vertex == sum of shuffle_bytes on the per edge counters). That will verify that the main objective of the jira has been achieved - ie - continuing to compatibly show these aggregated results on the vertex level (which would have disappeared without this change). > per-io counters flag should generate both overall and per-edge counters > > > Key: TEZ-2360 > URL: https://issues.apache.org/jira/browse/TEZ-2360 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Prakash Ramachandran > Attachments: TEZ-2360.1.patch, TEZ-2360.2.patch, TEZ-2360.3.patch > > > Currently, the per-io flag disables overall per task counters and retains > only per edge counters. It would be useful to have both overall and per edge > counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2360) per-io counters flag should generate both overall and per-edge counters
[ https://issues.apache.org/jira/browse/TEZ-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520160#comment-14520160 ] Siddharth Seth commented on TEZ-2360: - Nothing else to add. Looks good, after Bikas' comments are addressed. I'm not sure why we have a getUnderlyingCounters method. > per-io counters flag should generate both overall and per-edge counters > > > Key: TEZ-2360 > URL: https://issues.apache.org/jira/browse/TEZ-2360 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Prakash Ramachandran > Attachments: TEZ-2360.1.patch, TEZ-2360.2.patch, TEZ-2360.3.patch > > > Currently, the per-io flag disables overall per task counters and retains > only per edge counters. It would be useful to have both overall and per edge > counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2389) Tez UI: Sort by attempt-no is incorrect in attempts pages.
[
https://issues.apache.org/jira/browse/TEZ-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520171#comment-14520171
]
TezQA commented on TEZ-2389:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729252/TEZ-2389.1.patch
against master revision c76698e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/581//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/581//console
This message is automatically generated.
> Tez UI: Sort by attempt-no is incorrect in attempts pages.
> --
>
> Key: TEZ-2389
> URL: https://issues.apache.org/jira/browse/TEZ-2389
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-2389.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2389 PreCommit Build #581
Jira: https://issues.apache.org/jira/browse/TEZ-2389
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/581/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2776 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12729252/TEZ-2389.1.patch
against master revision c76698e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/581//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/581//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
33e4552d104677691939eabf9f4ddc817266b9dc logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #578
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2581228 bytes
Compression is 7.1%
Took 0.66 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2348) EOF exception during UnorderedKVReader.next()
[
https://issues.apache.org/jira/browse/TEZ-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520212#comment-14520212
]
Siddharth Seth commented on TEZ-2348:
-
[~gopalv] - I see the point of throwing an exception if this is accessed
incorrectly. Fail the query fast with a specific Exception rather than the app
potentially going into a loop - which can be really difficult to debug -
especially if there's no logging / large clusters.
Putting debugging aside, from an API perspective, I think an iterator like
interface is a lot cleaner. Issues should ideally be found in smaller scale
testing. In this particular case, this manifests as an exception from IFiles
which causes unnecessary confusion.
Moving all the next() invocations to throw an exception is theoretically an
incompatible change. However, it's highly unlikely that anyone goes past a
next() invocation returning false so the impact may not be huge (unlikely to be
used in different places, and not safe for multiple threads). If we're making
this change, it should be for all readers and as early as possible.
A terrible option would be to have the behaviour configurable :)... Exception
when we think an issue may be caused by incorrect usage, loop otherwise.
> EOF exception during UnorderedKVReader.next()
> -
>
> Key: TEZ-2348
> URL: https://issues.apache.org/jira/browse/TEZ-2348
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.2
>Reporter: Jason Dere
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2348.1.patch, TEZ-2348.2.patch, TEZ-2348.3.patch,
> _tez_session_dir.tgz
>
>
> {noformat}
> Caused by: java.lang.RuntimeException: java.io.IOException: Reached EOF.
> Completed reading 516605
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:184)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
> ... 13 more
> Caused by: java.io.IOException: Reached EOF. Completed reading 516605
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile.checkState(IFile.java:817)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.positionToNextRecord(IFile.java:698)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readRawKey(IFile.java:731)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:727)
> at
> org.apache.tez.runtime.library.common.readers.UnorderedKVReader.readNextFromCurrentReader(UnorderedKVReader.java:151)
> at
> org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:112)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$KeyValuesFromKeyValue.next(ReduceRecordSource.java:439)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents
[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520228#comment-14520228
]
Bikas Saha commented on TEZ-776:
bq. In Edge, "routeInputSourceTaskFailedEventToDestination" is checked for
enabling onDemandRouting
It uses the new signature of "routeCompositeDataMovementEventToDestination".
The logic is that if the plugin has not explicitly used the new API then it
must be using legacy API (since they were abstract). In some sense its a
reverse check to check if the plugin is a legacy API plugin. Of course the
check is not exhaustive but its not meant to be. This is to enabled older
versions of hive to run without change with newer versions of tez and limit the
legacy routing to only those vertices that use legacy plugins.
bq. if (routeMeta.getNumEvents() + listToAdd.size() > listMaxSize) {
Wil fix. I somehow missed it (I thought I had left a TODO there but clearly I
did not and forgot about it :P)
bq. Common code in the
ShuffleVertexManager.CustomShuffleEdgeManager.prepareRouting()
Will check if thats possible.
bq. In OneToOneEdgeManager,commonRouteMeta can be init-ed in prepareForRouting()
I did that so that I could make it final and allow compiler optimizations. I
forgot to add the final :P Added the final now.
bq. Should we throw exceptions in routeDataMovementEventToDestination in
OneToOneEdgeManager when sourceTaskIndex != destinationTaskIndex?. Or there can
be instances when null is acceptable?
It is by design. Not all consumers get events from every producer. Similar to
ScatterGather#routeDataMovementEventToDestination
bq. eventIndicesCreated, sourceIndicesRemainder can be removed in
ShuffleVertexManager
bq. BroadcastEdgeManager - cachedEventsLock, cachedEvents may not be relevant
bq. EdgeManagerPlugin - unwanted imports can be removed
Done.
> Reduce AM mem usage caused by storing TezEvents
> ---
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Bikas Saha
> Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
> TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
> TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch,
> TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch,
> TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
> With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
> events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
> without_patch_jmc_output_of_AM.png
>
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-2390) tez-tools swimlane tool fails to parse large jobs >8K containers
Jonathan Eagles created TEZ-2390: Summary: tez-tools swimlane tool fails to parse large jobs >8K containers Key: TEZ-2390 URL: https://issues.apache.org/jira/browse/TEZ-2390 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2382) Publish counters to timeline for failed/killed attempts
[ https://issues.apache.org/jira/browse/TEZ-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved TEZ-2382. -- Resolution: Cannot Reproduce Counters are already published to ats for failed attempts. Aggregation at the task and vertex level is missing which is tracked in a different jira > Publish counters to timeline for failed/killed attempts > > > Key: TEZ-2382 > URL: https://issues.apache.org/jira/browse/TEZ-2382 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > > This will be useful for debugging if attempts fail due to OOM or if an > attempt was killed as its speculated task ran faster, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2390) tez-tools swimlane tool fails to parse large jobs >8K containers
[ https://issues.apache.org/jira/browse/TEZ-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-2390: - Attachment: TEZ-2390.1.patch > tez-tools swimlane tool fails to parse large jobs >8K containers > > > Key: TEZ-2390 > URL: https://issues.apache.org/jira/browse/TEZ-2390 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-2390.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2390) tez-tools swimlane tool fails to parse large jobs >8K containers
[ https://issues.apache.org/jira/browse/TEZ-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520240#comment-14520240 ] Jonathan Eagles commented on TEZ-2390: -- Cleanup the python warnings and better of parsing of incomplete history in addition to allowing the container to id mapping extend beyond 8K. [~gopalv], can you take a look? > tez-tools swimlane tool fails to parse large jobs >8K containers > > > Key: TEZ-2390 > URL: https://issues.apache.org/jira/browse/TEZ-2390 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-2390.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2348) EOF exception during UnorderedKVReader.next()
[
https://issues.apache.org/jira/browse/TEZ-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520244#comment-14520244
]
Hitesh Shah commented on TEZ-2348:
--
bq. A terrible option would be to have the behaviour configurable
+1 - please do not make this configurable. A throw seems like the best option.
Handling bad user code should be part of the framework to the extent possible.
> EOF exception during UnorderedKVReader.next()
> -
>
> Key: TEZ-2348
> URL: https://issues.apache.org/jira/browse/TEZ-2348
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.2
>Reporter: Jason Dere
>Assignee: Rajesh Balamohan
> Attachments: TEZ-2348.1.patch, TEZ-2348.2.patch, TEZ-2348.3.patch,
> _tez_session_dir.tgz
>
>
> {noformat}
> Caused by: java.lang.RuntimeException: java.io.IOException: Reached EOF.
> Completed reading 516605
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:184)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
> ... 13 more
> Caused by: java.io.IOException: Reached EOF. Completed reading 516605
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile.checkState(IFile.java:817)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.positionToNextRecord(IFile.java:698)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readRawKey(IFile.java:731)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:727)
> at
> org.apache.tez.runtime.library.common.readers.UnorderedKVReader.readNextFromCurrentReader(UnorderedKVReader.java:151)
> at
> org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:112)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$KeyValuesFromKeyValue.next(ReduceRecordSource.java:439)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520263#comment-14520263 ] Hitesh Shah commented on TEZ-2385: -- Thanks [~lichangleo]. \cc [~jeagles] for review. > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2366) Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
[
https://issues.apache.org/jira/browse/TEZ-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520264#comment-14520264
]
Siddharth Seth commented on TEZ-2366:
-
Shuffle port should be sufficient though, right ? Won't have two Shuffle
listeners on the same port if there's multiple NodeManagers.
The NodeId would be host + RPC port ? unless YARN has added an actual
identifier which is unique across restarts of the same service.
> Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
>
>
> Key: TEZ-2366
> URL: https://issues.apache.org/jira/browse/TEZ-2366
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Daniel Dai
>Priority: Critical
> Attachments: TEZ-2366.test.txt, TEZ-2366.wip.1.patch
>
>
> There are around 20 unit tests (out of around 2000) fail intermittently after
> TEZ-2333. Here is a stack:
> {code}
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429899954360_0001_1_01_00_1_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:611)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:591)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:536)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.setupLocalDiskFetch(Fetcher.java:517)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:190)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> To reproduce that in Pig test, using the following commands:
> svn co http://svn.apache.org/repos/asf/pig/trunk
> ant -Dhadoopversion=23 -Dtest.exec.type=tez -Dtestcase=TestTezAutoParallelism
> test
> Note in Pig codebase, we already set TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to
> "true"
> (http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java?view=markup).
> I tried changing TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH to "false" in Pig and does
> not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2383) release sort buffers on close
[ https://issues.apache.org/jira/browse/TEZ-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520280#comment-14520280 ] Siddharth Seth commented on TEZ-2383: - The patch looks good. The intent is to allow GC of the Inputs / Outputs even if the user code holds on to a context object after it completes ? We should probably do the same for the Input and OutputContexts. > release sort buffers on close > - > > Key: TEZ-2383 > URL: https://issues.apache.org/jira/browse/TEZ-2383 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-2383.1.patch, TEZ-2383.2.patch, TEZ-2383.3.patch > > > Currently they get released when sorter object gets GC-ed, but it might be > good to explicitly release them on close as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2385) branch-0.6 compile failure
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520281#comment-14520281 ] Jonathan Eagles commented on TEZ-2385: -- +1. looks good, [~hitesh]. This is the correct approach for the 0.6 line. > branch-0.6 compile failure > -- > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2385) branch-0.6 compile failure caused by TEZ-2226
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2385: - Summary: branch-0.6 compile failure caused by TEZ-2226 (was: branch-0.6 compile failure) > branch-0.6 compile failure caused by TEZ-2226 > - > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2385) branch-0.6 compile failure caused by TEZ-2226
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520313#comment-14520313 ] Hitesh Shah commented on TEZ-2385: -- Thanks [~jeagles] for the review. Committing shortly. > branch-0.6 compile failure caused by TEZ-2226 > - > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2385) branch-0.6 compile failure caused by TEZ-2226
[ https://issues.apache.org/jira/browse/TEZ-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved TEZ-2385. -- Resolution: Fixed Fix Version/s: 0.6.1 Committed to branch 0.6 > branch-0.6 compile failure caused by TEZ-2226 > - > > Key: TEZ-2385 > URL: https://issues.apache.org/jira/browse/TEZ-2385 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.1 >Reporter: Jeff Zhang >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.6.1 > > Attachments: TEZ-2385.1.patch > > > due to TEZ-2226. DAG/Vertex level configuration is not available in 0.6, may > need to use the AM configuration or either back port TEZ-1233 to 0.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2380) Disable fall back to reading from timeline if timeline disabled
[ https://issues.apache.org/jira/browse/TEZ-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520316#comment-14520316 ] Hitesh Shah commented on TEZ-2380: -- Committing shortly. TEZ-2385 just committed. > Disable fall back to reading from timeline if timeline disabled > > > Key: TEZ-2380 > URL: https://issues.apache.org/jira/browse/TEZ-2380 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Chang Li > Attachments: TEZ-2380.1.patch > > > As part of TEZ-2226, timeline logging can get disabled in cases where domain > creation fails. In this scenario, the dag client shoudl also not fall back to > reading from timeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1560) Invalid state machine transition in recovery
[
https://issues.apache.org/jira/browse/TEZ-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-1560:
-
Target Version/s: 0.7.0, 0.5.4, 0.6.1 (was: 0.7.0)
> Invalid state machine transition in recovery
>
>
> Key: TEZ-1560
> URL: https://issues.apache.org/jira/browse/TEZ-1560
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: TEZ-1560-1.patch, failed_tez_job.txt.gz
>
>
> {code}
> 2014-09-04 16:08:25,504 INFO [main] org.apache.tez.dag.app.dag.impl.DAGImpl:
> dag_1409818083015_0001_1 transitioned from NEW to RUNNING
> 2014-09-04 16:08:25,504 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Recovered Vertex State,
> vertexId=vertex_1409818083015_0001_1_00 [v1], state=NEW,
> numInitedSourceVertices=0, numStartedSourceVertices=0,
> numRecoveredSourceVertices=0, recoveredEvents=0, tasksIsNull=false, numTasks=0
> 2014-09-04 16:08:25,505 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Root Inputs exist for Vertex: v1
> : {Input={InputName=Input},
> {Descriptor=ClassName=org.apache.tez.test.dag.MultiAttemptDAG$NoOpInput,
> hasPayload=false},
> {ControllerDescriptor=ClassName=org.apache.tez.test.dag.MultiAttemptDAG$TestRootInputInitializer,
> hasPayload=false}}
> 2014-09-04 16:08:25,505 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Starting root input initializer
> for input: Input, with class:
> [org.apache.tez.test.dag.MultiAttemptDAG$TestRootInputInitializer]
> 2014-09-04 16:08:25,506 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Setting user vertex manager
> plugin:
> org.apache.tez.test.dag.MultiAttemptDAG$FailOnAttemptVertexManagerPlugin on
> vertex: v1
> 2014-09-04 16:08:25,508 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Creating 2 for vertex:
> vertex_1409818083015_0001_1_00 [v1]
> 2014-09-04 16:08:25,518 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Starting root input initializers:
> 1
> 2014-09-04 16:08:25,520 INFO [InputInitializer [v1] #0]
> org.apache.tez.dag.app.dag.RootInputInitializerManager: Starting
> InputInitializer for Input: Input on vertex vertex_1409818083015_0001_1_00
> [v1]
> 2014-09-04 16:08:25,522 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.RootInputInitializerManager: Succeeded
> InputInitializer for Input: Input on vertex vertex_1409818083015_0001_1_00
> [v1]
> 2014-09-04 16:08:25,523 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: vertex_1409818083015_0001_1_00
> [v1] transitioned from NEW to INITIALIZING due to event V_INIT
> 2014-09-04 16:08:25,523 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Recovered Vertex State,
> vertexId=vertex_1409818083015_0001_1_01 [v2], state=NEW,
> numInitedSourceVertices0, numStartedSourceVertices=0,
> numRecoveredSourceVertices=1, tasksIsNull=false, numTasks=0
> 2014-09-04 16:08:25,523 ERROR [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.VertexImpl: Can't handle Invalid event
> V_SOURCE_VERTEX_RECOVERED on vertex v2 with vertexId
> vertex_1409818083015_0001_1_01 at current state NEW
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> V_SOURCE_VERTEX_RECOVERED at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1344)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1641)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-04 16:08:25,524 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
