Casey created AIRFLOW-2167: ------------------------------ Summary: Scheduler's clear_nonexistent_import_errors function should be called on first iteration Key: AIRFLOW-2167 URL: https://issues.apache.org/jira/browse/AIRFLOW-2167 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 1.9.0 Reporter: Casey Assignee: Casey Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png
In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not called until the amount of seconds defined by `dag_dir_list_interval` has elapsed. If the scheduler is not alive for the duration of `dag_dir_list_interval` (300 seconds) this cleanup never occurs. In some environments this could result in error messages displaying on the UI permanently, even if the DAG has been removed from the environment. It was previously an Airflow best practice to have the scheduler run N runtimes and terminate. Then, the scheduler would started again by an auxiliary process like Docker or Supervisor. This situation is what brought the bug to my attention. My suggested fix is to tweak jobs.py to run the import error cleanup on the first iteration and periodically as defined by `dag_dir_list_interval`. This way, a scheduler setup with a small number of runs will still have old errors cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)