[ https://issues.apache.org/jira/browse/LIVY-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay P C updated LIVY-713: ---------------------------- Environment: AWS EMR-5.21.0 > Yarn Resource Held up after task fails. > --------------------------------------- > > Key: LIVY-713 > URL: https://issues.apache.org/jira/browse/LIVY-713 > Project: Livy > Issue Type: Bug > Components: API > Affects Versions: 0.5.0 > Environment: AWS EMR-5.21.0 > Reporter: Sanjay P C > Priority: Major > Labels: EMR, LIVY > Attachments: airflow.PNG, livy.PNG, yarn-resource.PNG > > > Overview : > Here we are using airflow as scheduler to run task and Livy is used as > interface to submit spark jobs from airflow to AWS EMR cluster. > > Issue : > Here assuming airflow as initiated a job/task which inturn sends message to > livy to submit spark jobs. Now YARN has allocated resources for the submitted > task. If a task is failed due to a bad record / missing file / any exception, > then airflow marks task as Failed but task submitted by livy is stated as > idle and YARN states it has UNKNOWN. Due to this the resources are being held > up and any other task submitted to yarn will be in UNKNOWN state. -- This message was sent by Atlassian Jira (v8.3.4#803005)