[
https://issues.apache.org/jira/browse/LIVY-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay P C updated LIVY-713:
----------------------------
Description:
Overview :
Here we are using airflow as scheduler to run task and Livy is used as
interface to submit spark jobs from airflow to AWS EMR cluster.
Issue :
Here assuming airflow as initiated a job/task which inturn sends message to
livy to submit spark jobs. Now YARN has allocated resources for the submitted
task. If a task is failed due to a bad record / missing file / any exception,
then airflow marks task as Failed but task submitted by livy is stated as idle
and YARN states it has UNKNOWN. Due to this the resources are being held up and
any other task submitted to yarn will be in UNKNOWN state.
was:
Overview :
Here we are using airflow as scheduler to run task and Livy is used as
interface to submit spark jobs from airflow to AWS EMR cluster.
Issue :
Here assuming airflow as initiated a job/task which inturn sends message to
livy to submit spark jobs. Now YARN has allocated resources for the submitted
task. If a task is failed due to a bad record / missing file / any exception,
then airflow marks task as Failed but task submitted by livy is stated as idle
and YARN states it has UNKNOWN. Due to this the resources are being held up and
any other task submitted to yarn will in a UNKNOWN state.
> Yarn Resource Held up after task fails.
> ---------------------------------------
>
> Key: LIVY-713
> URL: https://issues.apache.org/jira/browse/LIVY-713
> Project: Livy
> Issue Type: Bug
> Components: API
> Affects Versions: 0.5.0
> Reporter: Sanjay P C
> Priority: Major
> Attachments: airflow.PNG, livy.PNG, yarn-resource.PNG
>
>
> Overview :
> Here we are using airflow as scheduler to run task and Livy is used as
> interface to submit spark jobs from airflow to AWS EMR cluster.
>
> Issue :
> Here assuming airflow as initiated a job/task which inturn sends message to
> livy to submit spark jobs. Now YARN has allocated resources for the submitted
> task. If a task is failed due to a bad record / missing file / any exception,
> then airflow marks task as Failed but task submitted by livy is stated as
> idle and YARN states it has UNKNOWN. Due to this the resources are being held
> up and any other task submitted to yarn will be in UNKNOWN state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
