[
https://issues.apache.org/jira/browse/AIRFLOW-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman updated AIRFLOW-849:
------------------------------------
Description:
As of now, Airflow shows that a task is "running" in Airflow when it could be
listed as "WAITING" in the Spark UI (and can be listed as "WAITING" for hours
when the Airflow UI might indicate it is running if I forget to check the Spark
UI).
Would be awesome once ticket 802 for spark integration is complete to add this
enhancement of more granular detail about the task.
Note (Daniel 3/29/20): We should add an "external" field to task state to store
task-specific data like spark cluster state
was:
As of now, Airflow shows that a task is "running" in Airflow when it could be
listed as "WAITING" in the Spark UI (and can be listed as "WAITING" for hours
when the Airflow UI might indicate it is running if I forget to check the Spark
UI).
Would be awesome once ticket 802 for spark integration is complete to add this
enhancement of more granular detail about the task.
> Show whether spark job is "running" or "waiting"
> ------------------------------------------------
>
> Key: AIRFLOW-849
> URL: https://issues.apache.org/jira/browse/AIRFLOW-849
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ui
> Reporter: Daniel Blazevski
> Priority: Minor
>
> As of now, Airflow shows that a task is "running" in Airflow when it could be
> listed as "WAITING" in the Spark UI (and can be listed as "WAITING" for hours
> when the Airflow UI might indicate it is running if I forget to check the
> Spark UI).
> Would be awesome once ticket 802 for spark integration is complete to add
> this enhancement of more granular detail about the task.
> Note (Daniel 3/29/20): We should add an "external" field to task state to
> store task-specific data like spark cluster state
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
