[
https://issues.apache.org/jira/browse/LIVY-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay P C updated LIVY-713:
----------------------------
Summary: Yarn Resource Held up after task fails. (was: Multiple execution
of dags puts the execution of applications on livy to idle state)
> Yarn Resource Held up after task fails.
> ---------------------------------------
>
> Key: LIVY-713
> URL: https://issues.apache.org/jira/browse/LIVY-713
> Project: Livy
> Issue Type: Bug
> Components: API
> Affects Versions: 0.5.0
> Environment: AWS EMR-5.21.0
> Reporter: Sanjay P C
> Priority: Major
> Labels: EMR, LIVY
> Attachments: airflow.PNG, livy.PNG, yarn-resource.PNG
>
>
> Overview :
> Here we are using airflow as scheduler to run task and Livy is used as
> interface to submit spark jobs from airflow to AWS EMR cluster.
>
> Issue :
> Here assuming airflow as initiated a job/task which inturn sends message to
> livy to submit spark jobs. Now YARN has allocated resources for the submitted
> task. If a task is failed due to a bad record / missing file / any exception,
> then airflow marks task as Failed but task submitted by livy is stated as
> idle and YARN states it has UNKNOWN. Due to this the resources are being held
> up and any other task submitted to yarn will be in UNKNOWN state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
