spider-man-tm commented on PR #68678: URL: https://github.com/apache/airflow/pull/68678#issuecomment-4752252295
I think you're right that the stale detection mechanism in DagProcessorManager._scan_stale_dags already handles this case: if hoge is removed from file_a.py, its last_parsed_time stops being updated, and after stale_dag_threshold seconds it gets marked stale — so the next parse of file_b.py will succeed without collision. So the current implementation is already eventually consistent in periodic-parsing environments like Cloud Composer. One alternative (Option A) would be to proactively mark disappeared DAGs as stale at parse time — reducing the error window from stale_dag_threshold to just the next parse of the original file. But that adds complexity. Happy to implement Option A if you think the transient import error is unacceptable, or we can leave the current behavior and document it. What do you prefer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
