knab-analytics opened a new issue, #29531: URL: https://github.com/apache/airflow/issues/29531
### Apache Airflow version 2.5.1 ### What happened Same problem as https://github.com/apache/airflow/issues/28296, but seems to happen nondeterministically, and still happens when ignoring `depends_on_past=True`. I've got a task that retrieves some filenames, which then creates dynamically mapped tasks to move the files, one per task. I'm using a similar task across multiple DAGs. However, task mapping fails on some DAG runs: it inconsistently happens per DAG run, and some DAGs do not seem to be affected at all. These seem to be the DAGs where no task was ever mapped, so that the mapped task instance ended up in a Skipped state. What happens is that multiple files will be found, but only a single dynamically mapped task will be created. This task never starts and has map_index of -1. It can be found under the "List instances, all runs" menu, but says "No Data found." under the "Mapped Tasks" tab.   When I press the "Run" button when the mapped task is selected, the following error appears: ``` Could not queue task instance for execution, dependencies not met: Previous Dagrun State: depends_on_past is true for this task's DAG, but the previous task instance has not run yet., Task has been mapped: The task has yet to be mapped! ``` The previous task _has_ run however. No errors appeared in my Airflow logs. When I try to run the task with **Ignore All Deps** enabled, I get the error: ``` Could not queue task instance for execution, dependencies not met: Previous Dagrun State: depends_on_past is true for this task's DAG, but the previous task instance has not run yet., Task has been mapped: The task has yet to be mapped! ``` This last bit is a contradiction, the task cannot be mapped and not mapped simultaneously. If the amount of mapped tasks is 0 while in this erroneous state, the mapped tasks will not be marked as skipped as expected. ### What you think should happen instead The mapped tasks should not get stuck with "no status". The mapped tasks should be created and ran successfully, or in the case of a 0-length list output of the upstream task they should be skipped. ### How to reproduce Run the below DAG, if it runs successfully clear several tasks out of order. This may not immediately reproduce the bug, but after some task clearing, for me it always ends up in the faulty state described above. ``` from airflow import DAG from airflow.decorators import task import datetime as dt from airflow.operators.python import PythonOperator import random @task def get_filenames_kwargs(): return [ {"file_name": i} for i in range(random.randint(0, 2)) ] def print_filename(file_name): print(file_name) with DAG( dag_id="dtm_test_2", start_date=dt.datetime(2023, 2, 10), default_args={ "owner": "airflow", "depends_on_past": True, }, schedule="@daily", ) as dag: get_filenames_task = get_filenames_kwargs.override(task_id="get_filenames_task")() print_filename_task = PythonOperator.partial( task_id="print_filename_task", python_callable=print_filename, ).expand(op_kwargs=get_filenames_task) ``` ### Operating System Amazon Linux v2 ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
