knab-analytics opened a new issue, #29531:
URL: https://github.com/apache/airflow/issues/29531

   ### Apache Airflow version
   
   2.5.1
   
   ### What happened
   
   Same problem as https://github.com/apache/airflow/issues/28296, but seems to 
happen nondeterministically, and still happens when ignoring 
`depends_on_past=True`.
   
   I've got a task that retrieves some filenames, which then creates 
dynamically mapped tasks to move the files, one per task.
   I'm using a similar task across multiple DAGs. However, task mapping fails 
on some DAG runs: it inconsistently happens per DAG run, and some DAGs do not 
seem to be affected at all. These seem to be the DAGs where no task was ever 
mapped, so that the mapped task instance ended up in a Skipped state.
   
   What happens is that multiple files will be found, but only a single 
dynamically mapped task will be created. This task never starts and has 
map_index of -1. It can be found under the "List instances, all runs" menu, but 
says "No Data found." under the "Mapped Tasks" tab.
   
   ![Screenshot 2023-02-14 at 13 29 
15](https://user-images.githubusercontent.com/64646000/218742434-c132d3c1-8013-446f-8fd0-9b485506f43e.png)
   
   ![Screenshot 2023-02-14 at 13 29 
25](https://user-images.githubusercontent.com/64646000/218742461-fb0114f6-6366-403b-841e-03b0657e3561.png)
   
   When I press the "Run" button when the mapped task is selected, the 
following error appears:
   
   ```
   Could not queue task instance for execution, dependencies not met: Previous 
Dagrun State: depends_on_past is true for this task's DAG, but the previous 
task instance has not run yet., Task has been mapped: The task has yet to be 
mapped!
   ```
   
   The previous task _has_ run however. No errors appeared in my Airflow logs.
   
   When I try to run the task with **Ignore All Deps** enabled, I get the error:
   
   ```
   Could not queue task instance for execution, dependencies not met: Previous 
Dagrun State: depends_on_past is true for this task's DAG, but the previous 
task instance has not run yet., Task has been mapped: The task has yet to be 
mapped!
   ```
   
   This last bit is a contradiction, the task cannot be mapped and not mapped 
simultaneously.
   
   If the amount of mapped tasks is 0 while in this erroneous state, the mapped 
tasks will not be marked as skipped as expected.
   
   ### What you think should happen instead
   
   The mapped tasks should not get stuck with "no status".
   
   The mapped tasks should be created and ran successfully, or in the case of a 
0-length list output of the upstream task they should be skipped.
   
   ### How to reproduce
   
   Run the below DAG, if it runs successfully clear several tasks out of order. 
This may not immediately reproduce the bug, but after some task clearing, for 
me it always ends up in the faulty state described above.
   
   ```
   from airflow import DAG
   from airflow.decorators import task
   import datetime as dt
   
   from airflow.operators.python import PythonOperator
   import random
   
   
   @task
   def get_filenames_kwargs():
       return [
           {"file_name": i}
           for i in range(random.randint(0, 2))
       ]
   
   def print_filename(file_name):
       print(file_name)
   
   with DAG(
           dag_id="dtm_test_2",
           start_date=dt.datetime(2023, 2, 10),
           default_args={
               "owner": "airflow",
               "depends_on_past": True,
           },
           schedule="@daily",
   ) as dag:
       get_filenames_task = 
get_filenames_kwargs.override(task_id="get_filenames_task")()
   
       print_filename_task = PythonOperator.partial(
           task_id="print_filename_task",
           python_callable=print_filename,
       ).expand(op_kwargs=get_filenames_task)
   ```
   
   ### Operating System
   
   Amazon Linux v2
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to