[
https://issues.apache.org/jira/browse/TEZ-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035798#comment-14035798
]
Rohini Palaniswamy commented on TEZ-1117:
-----------------------------------------
[~bikassaha],
We are not looking at a feature to terminate the session app when DAG fails.
We would never be using it as we want to continue run and finish other DAGs
even if one DAG fails as that is the default pig option unless specified. To
the user it is still a failure and we don't report success. If user has
specified to terminate on partial failure, we go kill the remaining hadoop jobs
(or session in tez) ourselves.
In MR world user will go look at the failed MR jobs on partial failure or full
failure and debug. That is why we want the Session App to show up as FAILED.
This is more of user perception and usability. Most of the end users go and
look at the hadoop jobs. They are not going to care/understand that the hadoop
job shows successful as it was a session and may comprise of successful and
failed dags as that is all internal details. To them it is confusing to see
something that FAILED execution with status as SUCCEEDED and they think it is a
bug and reported as such. I tried to explain why it is so, gave up and just
ended up saying that it is a known issue and will be fixed later. The intent is
just to try keep it simpler to user where something that failed shows up as
failed, and something that was killed shows up as killed.
bq. Beyond this, we can make more fine grained rules etc but I would like to
avoid coding Pig exit status logic into Tez. If further combinations are
needed, then we could think of allowing the user to specify final status in the
session.shutdown() call.
We would be ok with calling shutdown with some status. But I think that you
will find this a common usecase (hive as well) that you should make it a
default behavior. Also calling shutdown with status will still not handle the
case of reporting failures if session times out.
bq. when to shutdown a session? i.e. when a DAG fails or only when a user
invokes the shutdown?
We are looking at shutting down the session ourselves or on timeout as is
the current behavior.
bq. how is the final session status determined? If any dag fails, mark yarn app
as failed? or mark yarn app as failed iff the session shutdown as a result of
the last dag failing or some other fatal error? i.e a user initiated shutdown
will never mark a session as failed?
Looking for a user initiated shutdown, timeout or fatal error marking yarn
app failed when one of the submitted dag is failed. Fatal error to me should be
a failure even if there are only successful DAGs.
> Option to make YARN application failed on dag failure
> -----------------------------------------------------
>
> Key: TEZ-1117
> URL: https://issues.apache.org/jira/browse/TEZ-1117
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
>
> Can we have an configuration to make the Application status FAILED on
> termination if one of the DAGs fail? It is very confusing for users to see
> the application SUCCEEDED.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
