[jira] [Created] (TEZ-2457) Improve Documentation to explicitly list all valid Tez configuration variables
Hari Sekhon created TEZ-2457: Summary: Improve Documentation to explicitly list all valid Tez configuration variables Key: TEZ-2457 URL: https://issues.apache.org/jira/browse/TEZ-2457 Project: Apache Tez Issue Type: Improvement Affects Versions: 0.5.2 Environment: HDP 2.2 Reporter: Hari Sekhon Request to improve Tez documentation by adding a page showing all valid Tez configuration variables with their defaults and description as well as which MapReduce variables Tez respects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-2244 PreCommit Build #688
Jira: https://issues.apache.org/jira/browse/TEZ-2244
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/688/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2840 lines...]
[INFO] Final Memory: 71M/932M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733447/TEZ-2244.1.patch
against master revision cea05e3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/688//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/688//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
3527164ef466e674c8342bcd28f5b6749ab2dc97 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #687
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2643370 bytes
Compression is 4.7%
Took 2.1 sec
Description set: TEZ-2244
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2244) PipelinedSorter: Progressive allocation for sort-buffers
[
https://issues.apache.org/jira/browse/TEZ-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547577#comment-14547577
]
TezQA commented on TEZ-2244:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733447/TEZ-2244.1.patch
against master revision cea05e3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/688//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/688//console
This message is automatically generated.
PipelinedSorter: Progressive allocation for sort-buffers
Key: TEZ-2244
URL: https://issues.apache.org/jira/browse/TEZ-2244
Project: Apache Tez
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Gopal V
Assignee: Rajesh Balamohan
Attachments: TEZ-2244.1.patch, TEZ-2244.WIP.patch
Currently, the sort buffers are allocated pessimistically for all tasks so
that the largest task's spill stays within memory.
After the chained buffer implementation inside PipelinedSorter, it brings up
the possibility of only allocating the first chunk of the sort buffer when
the sorter starts up.
This allows for the tasks which do not heavily use the sort buffer (like a
grouping aggregation) to use the sort-space only when the map-aggregation
turns itself off.
Not reserving memory on startup hurts the worst-case scenario for the
pipelined sorter, but improves the average case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2457) Improve Documentation to explicitly list all valid Tez configuration variables
[ https://issues.apache.org/jira/browse/TEZ-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548472#comment-14548472 ] Hitesh Shah commented on TEZ-2457: -- TEZ-2294 might be a good starting point for you to look. It has configs generated from close to the 0.7 release but most should be applicable to your query. For all general purposes, Tez does *not* support any MapReduce variables. However, there are some helper functions that Hive/Pig use to convert MR configs to Tez configs. Good point on the MR configs though - I will look to see if there is a way to document this mapping more clearly. Improve Documentation to explicitly list all valid Tez configuration variables -- Key: TEZ-2457 URL: https://issues.apache.org/jira/browse/TEZ-2457 Project: Apache Tez Issue Type: Improvement Affects Versions: 0.5.2 Environment: HDP 2.2 Reporter: Hari Sekhon Request to improve Tez documentation by adding a page showing all valid Tez configuration variables with their defaults and description as well as which MapReduce variables Tez respects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2458) Tez UI: killed vertices display diagnostics 'killed/failed due to:null'
Prakash Ramachandran created TEZ-2458: - Summary: Tez UI: killed vertices display diagnostics 'killed/failed due to:null' Key: TEZ-2458 URL: https://issues.apache.org/jira/browse/TEZ-2458 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Prakash Ramachandran Assignee: Prakash Ramachandran When you run a hive query and kills it in the middle of running it with cmd+c, you will see 'Vertex killed due to user-initiated job kill.' in the console. However, when you go to Tez UI, select that dag name, and choose Map/Reducer vertex under View, Diagnostics shows 'killed/failed due to:null' as shown in the attached screenshot. reported by [~taksaito] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2458) Tez UI: killed vertices display diagnostics 'killed/failed due to:null'
[ https://issues.apache.org/jira/browse/TEZ-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2458: -- Attachment: TEZ-2458.1.patch Tez UI: killed vertices display diagnostics 'killed/failed due to:null' --- Key: TEZ-2458 URL: https://issues.apache.org/jira/browse/TEZ-2458 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Prakash Ramachandran Assignee: Prakash Ramachandran Attachments: Screen Shot 2015-05-19 at 12.26.01 AM.png, TEZ-2458.1.patch When you run a hive query and kills it in the middle of running it with cmd+c, you will see 'Vertex killed due to user-initiated job kill.' in the console. However, when you go to Tez UI, select that dag name, and choose Map/Reducer vertex under View, Diagnostics shows 'killed/failed due to:null' as shown in the attached screenshot. reported by [~taksaito] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2458 PreCommit Build #689
Jira: https://issues.apache.org/jira/browse/TEZ-2458
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/689/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2867 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733600/TEZ-2458.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/689//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/689//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
106b5773b84bcc6b3736c920ddb76d4818cc1c0a logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #688
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2677547 bytes
Compression is 4.7%
Took 1.3 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2076) Tez framework to extract/analyze data stored in ATS for specific dag
[
https://issues.apache.org/jira/browse/TEZ-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549387#comment-14549387
]
Hitesh Shah commented on TEZ-2076:
--
Minor comments:
{code}
String help = LINE_SEPARATOR
385 + java -jar tez-history-parser-x.y.z-jar-with-dependencies.jar
386 + LINE_SEPARATOR
387 + OR
388 + LINE_SEPARATOR
389 + java -cp tez-history-parser-x.y.z-jar-with-dependencies.jar
org.apache.tez.history.ATSImportTool
390 + LINE_SEPARATOR
391 + OR
392 + LINE_SEPARATOR
393 +
HADOOP_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$HADOOP_CLASSPATH hadoop jar
394 + tez-history-parser-x.y.z.jar +
ATSImportTool.class.getName()
395 + LINE_SEPARATOR;
396 formatter.printHelp(240, help, Options,
{code}
- Not sure why there are repetitive options ( with/without class name ) - can
this be reduced to 2 i.e. one via java -cp and the other via hadoop jar?
Also, [~gopalv] raised this point offline too. There needs to be a way to warn
the user if the import tool is used on an in-progress dag - maybe this tool
should just error out for an in-progress dag and only work if there is a
special --allow-incomplete-data or similar flag set. Could be done in a
follow-up jira. \cc [~pramachandran] as the same is applicable to the UI
download tool.
Tez framework to extract/analyze data stored in ATS for specific dag
Key: TEZ-2076
URL: https://issues.apache.org/jira/browse/TEZ-2076
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: TEZ-2076.1.patch, TEZ-2076.10.patch, TEZ-2076.11.patch,
TEZ-2076.12.patch, TEZ-2076.13.patch, TEZ-2076.14.patch, TEZ-2076.15.patch,
TEZ-2076.2.patch, TEZ-2076.3.patch, TEZ-2076.4.patch, TEZ-2076.5.patch,
TEZ-2076.6.patch, TEZ-2076.7.patch, TEZ-2076.8.patch, TEZ-2076.9.patch,
TEZ-2076.WIP.2.patch, TEZ-2076.WIP.3.patch, TEZ-2076.WIP.patch
- Users should be able to download ATS data pertaining to a DAG from Tez-UI
(more like a zip file containing DAG/Vertex/Task/TaskAttempt info).
- This can be plugged to an analyzer which parses the data, adds semantics
and provides an in-memory representation for further analysis.
- This will enable to write different analyzer rules, which can be run on top
of this in-memory representation to come up with analysis on the DAG.
- Results of this analyzer rules can be rendered on to UI (standalone webapp)
later point in time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2409) Allow different edges to have different routing plugins
[
https://issues.apache.org/jira/browse/TEZ-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549273#comment-14549273
]
TezQA commented on TEZ-2409:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733617/TEZ-2409.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/691//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/691//console
This message is automatically generated.
Allow different edges to have different routing plugins
---
Key: TEZ-2409
URL: https://issues.apache.org/jira/browse/TEZ-2409
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Priority: Critical
Attachments: TEZ-2409.1.patch
It may be useful to allow different edge manager plugin types based on
different requirements. In order to support this, we would need to support
different plugins per edge for routing the events on that edge. A motivating
scenario is when a custom plugin from an older release of a downstream
project is using older API's while the latest release of that project has
moved on to newer API's. This would allow both old and new releases to work
with the latest Tez framework as optimally as possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2076) Tez framework to extract/analyze data stored in ATS for specific dag
[ https://issues.apache.org/jira/browse/TEZ-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549441#comment-14549441 ] Hitesh Shah edited comment on TEZ-2076 at 5/18/15 10:51 PM: +1 once above usage help and related docs are fixed. was (Author: hitesh): +1 once above usage docs are fixed. Tez framework to extract/analyze data stored in ATS for specific dag Key: TEZ-2076 URL: https://issues.apache.org/jira/browse/TEZ-2076 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2076.1.patch, TEZ-2076.10.patch, TEZ-2076.11.patch, TEZ-2076.12.patch, TEZ-2076.13.patch, TEZ-2076.14.patch, TEZ-2076.15.patch, TEZ-2076.2.patch, TEZ-2076.3.patch, TEZ-2076.4.patch, TEZ-2076.5.patch, TEZ-2076.6.patch, TEZ-2076.7.patch, TEZ-2076.8.patch, TEZ-2076.9.patch, TEZ-2076.WIP.2.patch, TEZ-2076.WIP.3.patch, TEZ-2076.WIP.patch - Users should be able to download ATS data pertaining to a DAG from Tez-UI (more like a zip file containing DAG/Vertex/Task/TaskAttempt info). - This can be plugged to an analyzer which parses the data, adds semantics and provides an in-memory representation for further analysis. - This will enable to write different analyzer rules, which can be run on top of this in-memory representation to come up with analysis on the DAG. - Results of this analyzer rules can be rendered on to UI (standalone webapp) later point in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2456) Refactor recovery event logging to ensure it meet the recovery event spec
[ https://issues.apache.org/jira/browse/TEZ-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549198#comment-14549198 ] Hitesh Shah commented on TEZ-2456: -- bq. DAGFinishedEvent may be logged multiple times. ( DAG move from SUCCEEDED from ERROR ? Should we ignore this ? ) No - as this affects final state. bq. VertexFinishedEvent should be logged before DAGFinishedEvent There should never be cases where a vertex can change state after a dag has finished. We should make sure that the state machine ensures that this scenario can never occur. \cc [~bikassaha]. bq. RootInputDataInformation must be logged before VertexInitializedEvent This depends on what criteria we choose to detect whether the root input initializer has completed running. If that flag is the vertex init event, that it fine. bq. DataMovement must be logged before TaskFinishedEvent Re=phrase this to all events generated from all task attempts of a given task should be logged before the task finished event. This becomes tricky for retrospective failures. Needs to be looked at what happens when we have events followed by task finished, then task re-run, more events and a final finished event. There could be crashes at any stage in the list. bq. TaskFinishedEvent must be logged before VertexFinishedEvent Task re-runs? Retrospective failures? bq. VertexParallelismUpdatedEvent must be logged before TaskStartedEvent This depends on the state machine. If the framework supports it, then recovery also should. bq. TaskAttemptFinishedEvent should be logged before TaskFinishedEvent Retrospective failures? Also, nothing in the list related to speculated attempts. How should those be handled? bq. should only be logged once. Should not log again when it’s recovered. There are multiple places where this is called out. Is there an issue if it gets logged twice? Will something break? Should there be checks to ensure it is logged only once or can the recovery handle it if the event is logged twice? What kind of problems do you see if it happens twice? Refactor recovery event logging to ensure it meet the recovery event spec - Key: TEZ-2456 URL: https://issues.apache.org/jira/browse/TEZ-2456 Project: Apache Tez Issue Type: Sub-task Reporter: Jeff Zhang Assignee: Jeff Zhang Currently we don't have spec for the recovery event logging. Recovery would be fragile to code change. This jira try to define the spec and refactor the recovery event logging to ensure it meet the spec. [~hitesh] Please help review the following spec I drafted. *DAG* * DAGSubmitted/DAGInitializedEvent/DAGStartedEvent must been logged once, Should not log it again when it’s recovered. * DAGFinishedEvent may be logged multiple times. ( DAG move from SUCCEEDED from ERROR ? Should we ignore this ? ) * VertexFinishedEvent should be logged before DAGFinishedEvent *Vertex* * RootInputDataInformation must be logged before VertexInitializedEvent * DataMovement must be logged before TaskFinishedEvent * TaskFinishedEvent must be logged before VertexFinishedEvent * VertexInitializedEvent / VertexStartedEvent should only be logged once, should not log again when it’s recovered. * VertexFinishedEvent may be logged multiple times. (e.g. Vertex move from SUCCEEDED to FAILED) * VertexParallelismUpdatedEvent must be logged before TaskStartedEvent * TaskFinishedEvent should be logged before VertexFinishedEvent *Task* * If there’s no TaskStartedEvent, TaskFinishedEvent may still be logged (e.g. Task is killed in NEW ) Current’s behavior is that TaskFinishedEvent won’t be logged if there’s no TaskStartedEvent. * TaskStartedEvent should only be logged once. Should not log again when it’s recovered. * TaskFinishedEvent may be logged multiple times (e.g. Task move from SUCCEEDED to FAILED) * TaskAttemptFinishedEvent should be logged before TaskFinishedEvent *TaskAttempt* * If there’s no TaskAttemptStartedEvent, TaskAttemptFinishedEvent may still be logged ( e.g. TaskAttempt is killed in NEW ) Current’s behavior is that TaskAttemptFinishedEvent won’t be logged if there’s no TaskAttemptStartedEvent * TaskAttemptStartedEvent should only be logged once. Should not log again when it’s recovered. * TaskAttemptFinishedEvent may be logged multiple times. (e.g. TaskAttempt move from SUCCEEDED to FAILED) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2458) Tez UI: killed vertices display diagnostics 'killed/failed due to:null'
[
https://issues.apache.org/jira/browse/TEZ-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549173#comment-14549173
]
TezQA commented on TEZ-2458:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733600/TEZ-2458.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/689//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/689//console
This message is automatically generated.
Tez UI: killed vertices display diagnostics 'killed/failed due to:null'
---
Key: TEZ-2458
URL: https://issues.apache.org/jira/browse/TEZ-2458
Project: Apache Tez
Issue Type: Bug
Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
Attachments: Screen Shot 2015-05-19 at 12.26.01 AM.png,
TEZ-2458.1.patch
When you run a hive query and kills it in the middle of running it with
cmd+c, you will see 'Vertex killed due to user-initiated job kill.' in the
console.
However, when you go to Tez UI, select that dag name, and choose Map/Reducer
vertex under View, Diagnostics shows 'killed/failed due to:null' as shown in
the attached screenshot.
reported by [~taksaito]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2409) Allow different edges to have different routing plugins
[ https://issues.apache.org/jira/browse/TEZ-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2409: Attachment: TEZ-2409.1.patch Allow different edges to have different routing plugins --- Key: TEZ-2409 URL: https://issues.apache.org/jira/browse/TEZ-2409 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Priority: Critical Attachments: TEZ-2409.1.patch It may be useful to allow different edge manager plugin types based on different requirements. In order to support this, we would need to support different plugins per edge for routing the events on that edge. A motivating scenario is when a custom plugin from an older release of a downstream project is using older API's while the latest release of that project has moved on to newer API's. This would allow both old and new releases to work with the latest Tez framework as optimally as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2409) Allow different edges to have different routing plugins
[ https://issues.apache.org/jira/browse/TEZ-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2409: Attachment: (was: TEZ-2409.1.patch) Allow different edges to have different routing plugins --- Key: TEZ-2409 URL: https://issues.apache.org/jira/browse/TEZ-2409 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Priority: Critical It may be useful to allow different edge manager plugin types based on different requirements. In order to support this, we would need to support different plugins per edge for routing the events on that edge. A motivating scenario is when a custom plugin from an older release of a downstream project is using older API's while the latest release of that project has moved on to newer API's. This would allow both old and new releases to work with the latest Tez framework as optimally as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2409) Allow different edges to have different routing plugins
[ https://issues.apache.org/jira/browse/TEZ-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-2409: Attachment: TEZ-2409.1.patch Allow different edges to have different routing plugins --- Key: TEZ-2409 URL: https://issues.apache.org/jira/browse/TEZ-2409 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Priority: Critical Attachments: TEZ-2409.1.patch It may be useful to allow different edge manager plugin types based on different requirements. In order to support this, we would need to support different plugins per edge for routing the events on that edge. A motivating scenario is when a custom plugin from an older release of a downstream project is using older API's while the latest release of that project has moved on to newer API's. This would allow both old and new releases to work with the latest Tez framework as optimally as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2454) Change FetcherOrderedGroup to work as Callables instead of blocking threads
[
https://issues.apache.org/jira/browse/TEZ-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549248#comment-14549248
]
TezQA commented on TEZ-2454:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733611/TEZ-2454.3.txt
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/690//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/690//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/690//console
This message is automatically generated.
Change FetcherOrderedGroup to work as Callables instead of blocking threads
---
Key: TEZ-2454
URL: https://issues.apache.org/jira/browse/TEZ-2454
Project: Apache Tez
Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: TEZ-2454.1.wip.txt, TEZ-2454.2.txt, TEZ-2454.3.txt
The Fetcher threads for Ordered Input currently run and block till merge
completes, which makes it difficult to use them via ThreadPools.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2409 PreCommit Build #691
Jira: https://issues.apache.org/jira/browse/TEZ-2409
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/691/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2881 lines...]
[INFO] Final Memory: 71M/925M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733617/TEZ-2409.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/691//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/691//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
37a8963f2705e32e7444048be50f016557b4b759 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #688
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2679576 bytes
Compression is 4.7%
Took 1 sec
Description set: TEZ-2409
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Assigned] (TEZ-2461) tez-history-parser compile fails with hadoop-2.4
[ https://issues.apache.org/jira/browse/TEZ-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned TEZ-2461: - Assignee: Rajesh Balamohan tez-history-parser compile fails with hadoop-2.4 Key: TEZ-2461 URL: https://issues.apache.org/jira/browse/TEZ-2461 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Rajesh Balamohan Attachments: TEZ-2461.1.patch https://builds.apache.org/job/Tez-Build-Hadoop-2.4/98/console \cc [~rajesh.balamohan] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2460) Temporary solution for issue due to YARN-2560
[
https://issues.apache.org/jira/browse/TEZ-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549723#comment-14549723
]
Jeff Zhang commented on TEZ-2460:
-
[~pramachandran] [~bikassaha] [~hitesh] Please help review it. ( no test
included, manually verify the version incompatible case )
Temporary solution for issue due to YARN-2560
-
Key: TEZ-2460
URL: https://issues.apache.org/jira/browse/TEZ-2460
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2460-1.patch
Due to YARN-2560, DAGClient can't get the correct diagnostics, this jira is
to make a temporary solution for it before YARN-2560 is resolved. Here's one
example that dag is failed due to version incompatible.
{noformat}
15/05/18 17:34:18 INFO client.TezClient: Tez Client Version: [
component=tez-api, version=0.7.0.2.3.0.0-1986,
revision=7a048b526519e53de5d6464493697e92b15718fc,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=20150514-0857 ]
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:19 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:19 INFO client.TezClient: Using
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage
Timeline ACLs
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:21 INFO examples.OrderedWordCount: Running OrderedWordCount
15/05/18 17:34:21 INFO client.TezClient: Submitting DAG application with id:
application_1431967480765_0017
15/05/18 17:34:21 INFO client.TezClientUtils: Using tez.lib.uris value from
configuration: /hdp/apps/2.3.0.0-1986/tez/tez.tar.gz
15/05/18 17:34:21 INFO client.TezClient: Tez system stage directory
hdfs://c1ha/tmp/ambari-qa/staging/.tez/application_1431967480765_0017 doesn't
exist and is created
15/05/18 17:34:21 INFO acls.ATSHistoryACLPolicyManager: Created Timeline
Domain for History ACLs, domainId=Tez_ATS_application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1431967480765_0017, dagName=OrderedWordCount
15/05/18 17:34:22 INFO impl.YarnClientImpl: Submitted application
application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: The url to track the Tez AM:
http://c6402.ambari.apache.org:8088/proxy/application_1431967480765_0017/
15/05/18 17:34:22 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:22 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:22 INFO client.DAGClientImpl: Waiting for DAG to start running
15/05/18 17:34:34 INFO client.DAGClientImpl: DAG initialized:
CurrentState=Running
15/05/18 17:34:38 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
15/05/18 17:34:38 INFO examples.OrderedWordCount: DAG diagnostics: []
{noformat}
{noformat}
2015-05-18 16:57:46,807 INFO [main] app.DAGAppMaster: Created DAGAppMaster
for application appattempt_1431967480765_0008_01, versionInfo=[
component=tez-dag, version=0.5.2.2.2.3.0-2611,
revision=2d3c6b639d5b1048bd20aad5736823a35edd2485,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTIme=20150314-1805 ]
2015-05-18 16:57:47,931 INFO [main] app.DAGAppMaster: Comparing client
version with AM version, clientVersion=0.7.0.2.3.0.0-1986,
AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:47,932 FATAL [main] app.DAGAppMaster: Incompatible versions
found, clientVersion=0.7.0.2.3.0.0-1986, AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:49,348 INFO [main] app.DAGAppMaster: Adding session token to
jobTokenSecretManager for application
2015-05-18 16:57:49,354 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.container.AMContainerEventType for class
org.apache.tez.dag.app.rm.container.AMContainerMap
2015-05-18 16:57:49,355 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.node.AMNodeEventType for class
org.apache.tez.dag.app.rm.node.AMNodeTracker
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class
org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGEventType for class
org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2015-05-18 16:57:49,357 INFO [main] event.AsyncDispatcher: Registering class
[jira] [Issue Comment Deleted] (TEZ-2461) tez-history-parser compile fails with hadoop-2.4
[
https://issues.apache.org/jira/browse/TEZ-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-2461:
-
Comment: was deleted
(was: {code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project tez-history-parser: Compilation failure: Compilation failure:
[ERROR]
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/ws/tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:[44,37]
package org.apache.directory.api.util does not exist
[ERROR]
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/ws/tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/utils/Utils.java:[21,37]
package org.apache.directory.api.util does not exist
{code}
Saw the directory api issue too. )
tez-history-parser compile fails with hadoop-2.4
Key: TEZ-2461
URL: https://issues.apache.org/jira/browse/TEZ-2461
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Rajesh Balamohan
Fix For: 0.8.0
Attachments: TEZ-2461.1.patch
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/98/console
\cc [~rajesh.balamohan]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2454) Change FetcherOrderedGroup to work as Callables instead of blocking threads
[
https://issues.apache.org/jira/browse/TEZ-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549715#comment-14549715
]
TezQA commented on TEZ-2454:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733686/TEZ-2454.4.txt
against master revision 11aa17e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/694//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/694//console
This message is automatically generated.
Change FetcherOrderedGroup to work as Callables instead of blocking threads
---
Key: TEZ-2454
URL: https://issues.apache.org/jira/browse/TEZ-2454
Project: Apache Tez
Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: TEZ-2454.1.wip.txt, TEZ-2454.2.txt, TEZ-2454.3.txt,
TEZ-2454.4.txt
The Fetcher threads for Ordered Input currently run and block till merge
completes, which makes it difficult to use them via ThreadPools.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2460 PreCommit Build #695
Jira: https://issues.apache.org/jira/browse/TEZ-2460
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/695/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3008 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733689/TEZ-2460-1.patch
against master revision 11aa17e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/695//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/695//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
97aedef2d25aed1edfc413165d616fc6077e6f09 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #694
Archived 47 artifacts
Archive block size is 32768
Received 6 blocks and 2691232 bytes
Compression is 6.8%
Took 0.81 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2456) Refactor recovery event logging to ensure it meet the recovery event spec
[ https://issues.apache.org/jira/browse/TEZ-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549781#comment-14549781 ] Jeff Zhang commented on TEZ-2456: - [~hitesh] Thanks for the review. bq. TaskFinishedEvent must be logged before VertexFinishedEvent ( Retrospective failures? ) Rephase it to: For VertexFinishedEvent (SUCCEEDED), there must be at least n TaskFinishedEvent (SUCCEEDED) bq. TaskAttemptFinishedEvent should be logged before TaskFinishedEvent ( Retrospective failures? ) Rephase it to: For TaskFinishedEvent (SUCCEEDED), there must be at least one TaskAttemptFinishedEvent (SUCCEEDED) bq. Also, nothing in the list related to speculated attempts. How should those be handled? After TEZ-2249, all the task attempts should be finished before task is finished. bq. There are multiple places where this is called out. Is there an issue if it gets logged twice? Will something break? Should there be checks to ensure it is logged only once or can the recovery handle it if the event is logged twice? What kind of problems do you see if it happens twice? Two issues will be caused by multiple logging for the same event. * The metrics will be incorrect, specially for the start_time finished_time * If the AM is killed again, the next recovery will handle the same recovery event multiple times, may cause some potential issue. I think the restoreFromEvent method assume every event is logged once. Refactor recovery event logging to ensure it meet the recovery event spec - Key: TEZ-2456 URL: https://issues.apache.org/jira/browse/TEZ-2456 Project: Apache Tez Issue Type: Sub-task Reporter: Jeff Zhang Assignee: Jeff Zhang Currently we don't have spec for the recovery event logging. Recovery would be fragile to code change. This jira try to define the spec and refactor the recovery event logging to ensure it meet the spec. [~hitesh] Please help review the following spec I drafted. *DAG* * DAGSubmitted/DAGInitializedEvent/DAGStartedEvent must been logged once, Should not log it again when it’s recovered. * DAGFinishedEvent may be logged multiple times. ( DAG move from SUCCEEDED from ERROR ? Should we ignore this ? ) * VertexFinishedEvent should be logged before DAGFinishedEvent *Vertex* * RootInputDataInformation must be logged before VertexInitializedEvent * DataMovement must be logged before TaskFinishedEvent * TaskFinishedEvent must be logged before VertexFinishedEvent * VertexInitializedEvent / VertexStartedEvent should only be logged once, should not log again when it’s recovered. * VertexFinishedEvent may be logged multiple times. (e.g. Vertex move from SUCCEEDED to FAILED) * VertexParallelismUpdatedEvent must be logged before TaskStartedEvent * TaskFinishedEvent should be logged before VertexFinishedEvent *Task* * If there’s no TaskStartedEvent, TaskFinishedEvent may still be logged (e.g. Task is killed in NEW ) Current’s behavior is that TaskFinishedEvent won’t be logged if there’s no TaskStartedEvent. * TaskStartedEvent should only be logged once. Should not log again when it’s recovered. * TaskFinishedEvent may be logged multiple times (e.g. Task move from SUCCEEDED to FAILED) * TaskAttemptFinishedEvent should be logged before TaskFinishedEvent *TaskAttempt* * If there’s no TaskAttemptStartedEvent, TaskAttemptFinishedEvent may still be logged ( e.g. TaskAttempt is killed in NEW ) Current’s behavior is that TaskAttemptFinishedEvent won’t be logged if there’s no TaskAttemptStartedEvent * TaskAttemptStartedEvent should only be logged once. Should not log again when it’s recovered. * TaskAttemptFinishedEvent may be logged multiple times. (e.g. TaskAttempt move from SUCCEEDED to FAILED) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2456) Refactor recovery event logging to ensure it meet the recovery event spec
[ https://issues.apache.org/jira/browse/TEZ-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549781#comment-14549781 ] Jeff Zhang edited comment on TEZ-2456 at 5/19/15 4:51 AM: -- [~hitesh] Thanks for the review. bq. TaskFinishedEvent must be logged before VertexFinishedEvent ( Retrospective failures? ) Rephase it to: For VertexFinishedEvent (SUCCEEDED), before it there must be at least n TaskFinishedEvent (SUCCEEDED) bq. TaskAttemptFinishedEvent should be logged before TaskFinishedEvent ( Retrospective failures? ) Rephase it to: For TaskFinishedEvent (SUCCEEDED), before it there must be at least one TaskAttemptFinishedEvent (SUCCEEDED) bq. Also, nothing in the list related to speculated attempts. How should those be handled? After TEZ-2249, all the task attempts should be finished before task is finished. bq. There are multiple places where this is called out. Is there an issue if it gets logged twice? Will something break? Should there be checks to ensure it is logged only once or can the recovery handle it if the event is logged twice? What kind of problems do you see if it happens twice? Two issues will be caused by multiple logging for the same event. * The metrics will be incorrect, specially for the start_time finished_time * If the AM is killed again, the next recovery will handle the same recovery event multiple times, may cause some potential issue. I think the restoreFromEvent method assume every event is logged once. was (Author: zjffdu): [~hitesh] Thanks for the review. bq. TaskFinishedEvent must be logged before VertexFinishedEvent ( Retrospective failures? ) Rephase it to: For VertexFinishedEvent (SUCCEEDED), there must be at least n TaskFinishedEvent (SUCCEEDED) bq. TaskAttemptFinishedEvent should be logged before TaskFinishedEvent ( Retrospective failures? ) Rephase it to: For TaskFinishedEvent (SUCCEEDED), there must be at least one TaskAttemptFinishedEvent (SUCCEEDED) bq. Also, nothing in the list related to speculated attempts. How should those be handled? After TEZ-2249, all the task attempts should be finished before task is finished. bq. There are multiple places where this is called out. Is there an issue if it gets logged twice? Will something break? Should there be checks to ensure it is logged only once or can the recovery handle it if the event is logged twice? What kind of problems do you see if it happens twice? Two issues will be caused by multiple logging for the same event. * The metrics will be incorrect, specially for the start_time finished_time * If the AM is killed again, the next recovery will handle the same recovery event multiple times, may cause some potential issue. I think the restoreFromEvent method assume every event is logged once. Refactor recovery event logging to ensure it meet the recovery event spec - Key: TEZ-2456 URL: https://issues.apache.org/jira/browse/TEZ-2456 Project: Apache Tez Issue Type: Sub-task Reporter: Jeff Zhang Assignee: Jeff Zhang Currently we don't have spec for the recovery event logging. Recovery would be fragile to code change. This jira try to define the spec and refactor the recovery event logging to ensure it meet the spec. [~hitesh] Please help review the following spec I drafted. *DAG* * DAGSubmitted/DAGInitializedEvent/DAGStartedEvent must been logged once, Should not log it again when it’s recovered. * DAGFinishedEvent may be logged multiple times. ( DAG move from SUCCEEDED from ERROR ? Should we ignore this ? ) * VertexFinishedEvent should be logged before DAGFinishedEvent *Vertex* * RootInputDataInformation must be logged before VertexInitializedEvent * DataMovement must be logged before TaskFinishedEvent * TaskFinishedEvent must be logged before VertexFinishedEvent * VertexInitializedEvent / VertexStartedEvent should only be logged once, should not log again when it’s recovered. * VertexFinishedEvent may be logged multiple times. (e.g. Vertex move from SUCCEEDED to FAILED) * VertexParallelismUpdatedEvent must be logged before TaskStartedEvent * TaskFinishedEvent should be logged before VertexFinishedEvent *Task* * If there’s no TaskStartedEvent, TaskFinishedEvent may still be logged (e.g. Task is killed in NEW ) Current’s behavior is that TaskFinishedEvent won’t be logged if there’s no TaskStartedEvent. * TaskStartedEvent should only be logged once. Should not log again when it’s recovered. * TaskFinishedEvent may be logged multiple times (e.g. Task move from SUCCEEDED to FAILED) * TaskAttemptFinishedEvent should be logged before TaskFinishedEvent *TaskAttempt* * If there’s no TaskAttemptStartedEvent, TaskAttemptFinishedEvent may still be logged (
[jira] [Commented] (TEZ-2461) tez-history-parser compile fails with hadoop-2.4
[
https://issues.apache.org/jira/browse/TEZ-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549738#comment-14549738
]
TezQA commented on TEZ-2461:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733697/TEZ-2461.1.patch
against master revision 11aa17e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/696//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/696//console
This message is automatically generated.
tez-history-parser compile fails with hadoop-2.4
Key: TEZ-2461
URL: https://issues.apache.org/jira/browse/TEZ-2461
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Rajesh Balamohan
Attachments: TEZ-2461.1.patch
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/98/console
\cc [~rajesh.balamohan]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2461) tez-history-parser compile fails with hadoop-2.4
[
https://issues.apache.org/jira/browse/TEZ-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549791#comment-14549791
]
Hitesh Shah commented on TEZ-2461:
--
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project tez-history-parser: Compilation failure: Compilation failure:
[ERROR]
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/ws/tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:[44,37]
package org.apache.directory.api.util does not exist
[ERROR]
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/ws/tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/parser/utils/Utils.java:[21,37]
package org.apache.directory.api.util does not exist
{code}
Saw the directory api issue too.
tez-history-parser compile fails with hadoop-2.4
Key: TEZ-2461
URL: https://issues.apache.org/jira/browse/TEZ-2461
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Rajesh Balamohan
Fix For: 0.8.0
Attachments: TEZ-2461.1.patch
https://builds.apache.org/job/Tez-Build-Hadoop-2.4/98/console
\cc [~rajesh.balamohan]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2460) Temporary solution for issue due to YARN-2560
[
https://issues.apache.org/jira/browse/TEZ-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549803#comment-14549803
]
Bikas Saha commented on TEZ-2460:
-
This workaround may need to be around longer than temporary depending on how
much effort it is to resolve the root cause in YARN. The patch looks fine
though we should add a private configuration to determine how long the timeout
should be. Rests looks good.
Temporary solution for issue due to YARN-2560
-
Key: TEZ-2460
URL: https://issues.apache.org/jira/browse/TEZ-2460
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2460-1.patch
Due to YARN-2560, DAGClient can't get the correct diagnostics, this jira is
to make a temporary solution for it before YARN-2560 is resolved. Here's one
example that dag is failed due to version incompatible.
{noformat}
15/05/18 17:34:18 INFO client.TezClient: Tez Client Version: [
component=tez-api, version=0.7.0.2.3.0.0-1986,
revision=7a048b526519e53de5d6464493697e92b15718fc,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=20150514-0857 ]
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:19 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:19 INFO client.TezClient: Using
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage
Timeline ACLs
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:21 INFO examples.OrderedWordCount: Running OrderedWordCount
15/05/18 17:34:21 INFO client.TezClient: Submitting DAG application with id:
application_1431967480765_0017
15/05/18 17:34:21 INFO client.TezClientUtils: Using tez.lib.uris value from
configuration: /hdp/apps/2.3.0.0-1986/tez/tez.tar.gz
15/05/18 17:34:21 INFO client.TezClient: Tez system stage directory
hdfs://c1ha/tmp/ambari-qa/staging/.tez/application_1431967480765_0017 doesn't
exist and is created
15/05/18 17:34:21 INFO acls.ATSHistoryACLPolicyManager: Created Timeline
Domain for History ACLs, domainId=Tez_ATS_application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1431967480765_0017, dagName=OrderedWordCount
15/05/18 17:34:22 INFO impl.YarnClientImpl: Submitted application
application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: The url to track the Tez AM:
http://c6402.ambari.apache.org:8088/proxy/application_1431967480765_0017/
15/05/18 17:34:22 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:22 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:22 INFO client.DAGClientImpl: Waiting for DAG to start running
15/05/18 17:34:34 INFO client.DAGClientImpl: DAG initialized:
CurrentState=Running
15/05/18 17:34:38 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
15/05/18 17:34:38 INFO examples.OrderedWordCount: DAG diagnostics: []
{noformat}
{noformat}
2015-05-18 16:57:46,807 INFO [main] app.DAGAppMaster: Created DAGAppMaster
for application appattempt_1431967480765_0008_01, versionInfo=[
component=tez-dag, version=0.5.2.2.2.3.0-2611,
revision=2d3c6b639d5b1048bd20aad5736823a35edd2485,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTIme=20150314-1805 ]
2015-05-18 16:57:47,931 INFO [main] app.DAGAppMaster: Comparing client
version with AM version, clientVersion=0.7.0.2.3.0.0-1986,
AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:47,932 FATAL [main] app.DAGAppMaster: Incompatible versions
found, clientVersion=0.7.0.2.3.0.0-1986, AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:49,348 INFO [main] app.DAGAppMaster: Adding session token to
jobTokenSecretManager for application
2015-05-18 16:57:49,354 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.container.AMContainerEventType for class
org.apache.tez.dag.app.rm.container.AMContainerMap
2015-05-18 16:57:49,355 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.node.AMNodeEventType for class
org.apache.tez.dag.app.rm.node.AMNodeTracker
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class
org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGEventType for class
Failed: TEZ-2461 PreCommit Build #696
Jira: https://issues.apache.org/jira/browse/TEZ-2461
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/696/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3009 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733697/TEZ-2461.1.patch
against master revision 11aa17e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/696//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/696//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a5aff1cc25b5559b0f2d6be8d21bff33b2423fb1 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #694
Archived 47 artifacts
Archive block size is 32768
Received 6 blocks and 2689005 bytes
Compression is 6.8%
Took 1.6 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2461) tez-history-parser compile fails with hadoop-2.4
[ https://issues.apache.org/jira/browse/TEZ-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549774#comment-14549774 ] Jeff Zhang commented on TEZ-2461: - +1 tez-history-parser compile fails with hadoop-2.4 Key: TEZ-2461 URL: https://issues.apache.org/jira/browse/TEZ-2461 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Rajesh Balamohan Attachments: TEZ-2461.1.patch https://builds.apache.org/job/Tez-Build-Hadoop-2.4/98/console \cc [~rajesh.balamohan] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2076) Tez framework to extract/analyze data stored in ATS for specific dag
[ https://issues.apache.org/jira/browse/TEZ-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2076: -- Attachment: TEZ-2076.16.patch Reduced the options to 2 and fixed javadoc to reflect this. Thanks [~hitesh]. Will wait for precommit build and commit to master. Will create a followup jira for --allow-incomplete-data flag Tez framework to extract/analyze data stored in ATS for specific dag Key: TEZ-2076 URL: https://issues.apache.org/jira/browse/TEZ-2076 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2076.1.patch, TEZ-2076.10.patch, TEZ-2076.11.patch, TEZ-2076.12.patch, TEZ-2076.13.patch, TEZ-2076.14.patch, TEZ-2076.15.patch, TEZ-2076.16.patch, TEZ-2076.2.patch, TEZ-2076.3.patch, TEZ-2076.4.patch, TEZ-2076.5.patch, TEZ-2076.6.patch, TEZ-2076.7.patch, TEZ-2076.8.patch, TEZ-2076.9.patch, TEZ-2076.WIP.2.patch, TEZ-2076.WIP.3.patch, TEZ-2076.WIP.patch - Users should be able to download ATS data pertaining to a DAG from Tez-UI (more like a zip file containing DAG/Vertex/Task/TaskAttempt info). - This can be plugged to an analyzer which parses the data, adds semantics and provides an in-memory representation for further analysis. - This will enable to write different analyzer rules, which can be run on top of this in-memory representation to come up with analysis on the DAG. - Results of this analyzer rules can be rendered on to UI (standalone webapp) later point in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2454) Change FetcherOrderedGroup to work as Callables instead of blocking threads
[ https://issues.apache.org/jira/browse/TEZ-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2454: Attachment: TEZ-2454.4.txt findbugs ignored properly this time. Change FetcherOrderedGroup to work as Callables instead of blocking threads --- Key: TEZ-2454 URL: https://issues.apache.org/jira/browse/TEZ-2454 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-2454.1.wip.txt, TEZ-2454.2.txt, TEZ-2454.3.txt, TEZ-2454.4.txt The Fetcher threads for Ordered Input currently run and block till merge completes, which makes it difficult to use them via ThreadPools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2454 PreCommit Build #690
Jira: https://issues.apache.org/jira/browse/TEZ-2454
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/690/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2881 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733611/TEZ-2454.3.txt
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/690//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/690//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/690//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
f2606419b5a31bab23cce0d96ba1b66745001750 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #688
Archived 44 artifacts
Archive block size is 32768
Received 20 blocks and 2135483 bytes
Compression is 23.5%
Took 1.7 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Success: TEZ-2076 PreCommit Build #693
Jira: https://issues.apache.org/jira/browse/TEZ-2076
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/693/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3021 lines...]
[INFO] Final Memory: 87M/1470M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733662/TEZ-2076.16.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/693//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/693//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
3dd8d5ce1c11fdeef9f1a2bbbcbb64c36ae54460 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #691
Archived 47 artifacts
Archive block size is 32768
Received 4 blocks and 2736290 bytes
Compression is 4.6%
Took 0.76 sec
Description set: TEZ-2076
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Failed: TEZ-2409 PreCommit Build #692
Jira: https://issues.apache.org/jira/browse/TEZ-2409
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/692/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2474 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733655/TEZ-2409.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.dag.impl.TestVertexImpl
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/692//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/692//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1b8cc91d36a0437e0ad6f9073c3dc9cd7920fcf5 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #691
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2625496 bytes
Compression is 4.8%
Took 0.84 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2409) Allow different edges to have different routing plugins
[
https://issues.apache.org/jira/browse/TEZ-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549501#comment-14549501
]
TezQA commented on TEZ-2409:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733655/TEZ-2409.1.patch
against master revision 12fc2c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.dag.impl.TestVertexImpl
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/692//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/692//console
This message is automatically generated.
Allow different edges to have different routing plugins
---
Key: TEZ-2409
URL: https://issues.apache.org/jira/browse/TEZ-2409
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Priority: Critical
Attachments: TEZ-2409.1.patch
It may be useful to allow different edge manager plugin types based on
different requirements. In order to support this, we would need to support
different plugins per edge for routing the events on that edge. A motivating
scenario is when a custom plugin from an older release of a downstream
project is using older API's while the latest release of that project has
moved on to newer API's. This would allow both old and new releases to work
with the latest Tez framework as optimally as possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-2459) Add --allow-incomplete-data flag to ATSImportTool
Rajesh Balamohan created TEZ-2459: - Summary: Add --allow-incomplete-data flag to ATSImportTool Key: TEZ-2459 URL: https://issues.apache.org/jira/browse/TEZ-2459 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Provide a way to warn the user if the import tool is used on an in-progress dag - maybe this tool should just error out for an in-progress dag and only work if there is a special --allow-incomplete-data or similar flag set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2460) Temporary solution for issue due to YARN-2560
[
https://issues.apache.org/jira/browse/TEZ-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated TEZ-2460:
Attachment: TEZ-2460-1.patch
Temporary solution for issue due to YARN-2560
-
Key: TEZ-2460
URL: https://issues.apache.org/jira/browse/TEZ-2460
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2460-1.patch
Due to YARN-2560, DAGClient can't get the correct diagnostics, this jira is
to make a temporary solution for it before YARN-2560 is resolved. Here's one
example that dag is failed due to version incompatible.
{noformat}
15/05/18 17:34:18 INFO client.TezClient: Tez Client Version: [
component=tez-api, version=0.7.0.2.3.0.0-1986,
revision=7a048b526519e53de5d6464493697e92b15718fc,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=20150514-0857 ]
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:19 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:19 INFO client.TezClient: Using
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage
Timeline ACLs
15/05/18 17:34:19 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:21 INFO examples.OrderedWordCount: Running OrderedWordCount
15/05/18 17:34:21 INFO client.TezClient: Submitting DAG application with id:
application_1431967480765_0017
15/05/18 17:34:21 INFO client.TezClientUtils: Using tez.lib.uris value from
configuration: /hdp/apps/2.3.0.0-1986/tez/tez.tar.gz
15/05/18 17:34:21 INFO client.TezClient: Tez system stage directory
hdfs://c1ha/tmp/ambari-qa/staging/.tez/application_1431967480765_0017 doesn't
exist and is created
15/05/18 17:34:21 INFO acls.ATSHistoryACLPolicyManager: Created Timeline
Domain for History ACLs, domainId=Tez_ATS_application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1431967480765_0017, dagName=OrderedWordCount
15/05/18 17:34:22 INFO impl.YarnClientImpl: Submitted application
application_1431967480765_0017
15/05/18 17:34:22 INFO client.TezClient: The url to track the Tez AM:
http://c6402.ambari.apache.org:8088/proxy/application_1431967480765_0017/
15/05/18 17:34:22 INFO impl.TimelineClientImpl: Timeline service address:
http://c6402.ambari.apache.org:8188/ws/v1/timeline/
15/05/18 17:34:22 INFO client.RMProxy: Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
15/05/18 17:34:22 INFO client.DAGClientImpl: Waiting for DAG to start running
15/05/18 17:34:34 INFO client.DAGClientImpl: DAG initialized:
CurrentState=Running
15/05/18 17:34:38 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
15/05/18 17:34:38 INFO examples.OrderedWordCount: DAG diagnostics: []
{noformat}
{noformat}
2015-05-18 16:57:46,807 INFO [main] app.DAGAppMaster: Created DAGAppMaster
for application appattempt_1431967480765_0008_01, versionInfo=[
component=tez-dag, version=0.5.2.2.2.3.0-2611,
revision=2d3c6b639d5b1048bd20aad5736823a35edd2485,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTIme=20150314-1805 ]
2015-05-18 16:57:47,931 INFO [main] app.DAGAppMaster: Comparing client
version with AM version, clientVersion=0.7.0.2.3.0.0-1986,
AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:47,932 FATAL [main] app.DAGAppMaster: Incompatible versions
found, clientVersion=0.7.0.2.3.0.0-1986, AMVersion=0.5.2.2.2.3.0-2611
2015-05-18 16:57:49,348 INFO [main] app.DAGAppMaster: Adding session token to
jobTokenSecretManager for application
2015-05-18 16:57:49,354 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.container.AMContainerEventType for class
org.apache.tez.dag.app.rm.container.AMContainerMap
2015-05-18 16:57:49,355 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.node.AMNodeEventType for class
org.apache.tez.dag.app.rm.node.AMNodeTracker
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class
org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2015-05-18 16:57:49,356 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGEventType for class
org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2015-05-18 16:57:49,357 INFO [main] event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.VertexEventType for class
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2015-05-18 16:57:49,358 INFO [main]
[jira] [Commented] (TEZ-2450) support async http clients in ordered unordered inputs
[ https://issues.apache.org/jira/browse/TEZ-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547618#comment-14547618 ] Siddharth Seth commented on TEZ-2450: - Minor stuff - Default value for TEZ_RUNTIME_SHUFFLE_USE_ASYNC_HTTP in TezConfiguration - ShuffleUtils.getHttpConnection - Avoid reading from config on each request. - TestPipelinedShuffle - conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_USE_ASYNC_HTTP, true); and later set to true again in the test. Think all tests may end up running with the AsyncClient More important - CompletionHandler - this throws exceptions from the various callbacks. In case of errors (Connection Validation), this probably needs to inform the ShuffleManager / ShuffleScheduler - like the Fetcher does. Not sure how the library handles exceptions thrown by the callback. - response = httpAsyncClient.executeRequest(request, new CompletionHandler()).get(); - Will this only return after all data has been fetched, or will it return right after the Connection establishment ? Will this need to go to disk / memory based on the response size - using the onPartReceived callback ? Given the async nature and callbacks when data is received - I'm guessing the requirement for N fetcher threads probably goes away in the future, to be replaced by thread management done in the library itself ? support async http clients in ordered unordered inputs Key: TEZ-2450 URL: https://issues.apache.org/jira/browse/TEZ-2450 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2450.1.patch, TEZ-2450.WIP.patch It will be helpful to switch between JDK other async http impls. For LLAP scenarios, it would be useful to make http clients interruptible which is supported in async libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2450) support async http clients in ordered unordered inputs
[ https://issues.apache.org/jira/browse/TEZ-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547773#comment-14547773 ] Rajesh Balamohan commented on TEZ-2450: --- Thanks for the review [~sseth]. Canceling the patch as onBodyPartReceived() implementation would need more thinking, as it would impact the way MapOutputs are generated. It is possible to stream the contents of onBodyPartReceived to file. But even for that decision (file/mem), we need to read ShuffleHeader from inputStream which does not look like a straight forward approach using onBodyPartReceived. It appears that response.getResponseBodyAsStream() is wrapping the bytes in inputstream, which wouldn't be much useful for large downloads. support async http clients in ordered unordered inputs Key: TEZ-2450 URL: https://issues.apache.org/jira/browse/TEZ-2450 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2450.1.patch, TEZ-2450.WIP.patch It will be helpful to switch between JDK other async http impls. For LLAP scenarios, it would be useful to make http clients interruptible which is supported in async libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2455) Tez UI: Dag view caching, error handling and minor layout changes
[ https://issues.apache.org/jira/browse/TEZ-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2455: -- Summary: Tez UI: Dag view caching, error handling and minor layout changes (was: Tez UI: Dag view changes and data load changes) Tez UI: Dag view caching, error handling and minor layout changes - Key: TEZ-2455 URL: https://issues.apache.org/jira/browse/TEZ-2455 Project: Apache Tez Issue Type: Bug Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Attachments: TEZ-2455.1.patch, TEZ-2455.2.patch, TEZ-2455.3.patch # Enable caching in Dag View. # View throws error when all vertex data are not loaded # Display single outputs directly below the vertex # Vetex Input nodes at the same level slightly overlaps at times # Entities under a DAG not loading in IE # Prevent 'All DAGs' page from hitting ATS with a huge limit when rowCount is manually edited -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2455) Tez UI: Dag view caching, error handling and minor layout changes
[ https://issues.apache.org/jira/browse/TEZ-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547864#comment-14547864 ] Prakash Ramachandran commented on TEZ-2455: --- +1 LGTM. committing. Tez UI: Dag view caching, error handling and minor layout changes - Key: TEZ-2455 URL: https://issues.apache.org/jira/browse/TEZ-2455 Project: Apache Tez Issue Type: Bug Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Attachments: TEZ-2455.1.patch, TEZ-2455.2.patch, TEZ-2455.3.patch # Enable caching in Dag View. # View throws error when all vertex data are not loaded # Display single outputs directly below the vertex # Vetex Input nodes at the same level slightly overlaps at times # Entities under a DAG not loading in IE # Prevent 'All DAGs' page from hitting ATS with a huge limit when rowCount is manually edited -- This message was sent by Atlassian JIRA (v6.3.4#6332)
