JosephABC opened a new issue, #37154:
URL: https://github.com/apache/airflow/issues/37154

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.7.3
   
   ### What happened?
   
   ### **Dag Information**
   
   - `generate_records_task` will generate a list of integers based on attempt
     - 1 less record per attempt (4 -> 3 -> 2)
   
   - The list of records returned is dynamic and will serve as input to the 
mapped task operator to generate the intended number of mapped tasks to run on 
each record
   
   - `depends_on_past` is true for this DAG
   
   - We are not using task flow decorators
   - Business uses for this pattern : Identify cities/regions for which we need 
to run pipeline dynamically. The source of truth for this can change in real 
time.
   
   <img width="500" alt="dag_graph" 
src="https://github.com/apache/airflow/assets/25327813/1e2c2a61-2525-4265-bfc2-3aefe03fbfcc";>
   
   
   
   ### **Previous Dag Run**
   
   - When we perform actions like clear dag or clear task and downstream, 
`generate_records_task` task will generate different number of records for each 
attempt
   - When a mapped task like `mapped_task_instance` was executed the second 
time for the same dag run, it would mark one of the mapped tasks as `removed` 
state. Whole mapped task was marked as success
   
   
![previous_dag_run_attempt_1](https://github.com/apache/airflow/assets/25327813/7d5ca12a-3fb6-4eaf-8bf2-13b0c698f4d7)
   
   
![previous_dag_run_attempt_2](https://github.com/apache/airflow/assets/25327813/2570346b-98ad-421f-9568-7d07b56dc4b7)
   
   
![previous_dag_run](https://github.com/apache/airflow/assets/25327813/13ae030d-b7aa-4171-92fe-e761add72cc1)
   
   ### **Subsequent Dag Run**
   
   - We tried 2 scenarios for subsequent Dag Run
     - Clear Dag Run
     - Delete Dag Run (similar to having a new Dag Run)
   - In both senarios, the `mapped_task_instance` task did not schedule and dag 
run was stuck in `running` state
     - For Clear Dag Run: We can see that there are 3 mapped with no_status and 
1 with removed. 
     - For Delete Dag Run: No mapped tasks were generated
   
   
![subsequent_dag_run_clear](https://github.com/apache/airflow/assets/25327813/b3d85005-c1ef-4d05-a8ae-c12050d990a6)
   
     - For Delete Dag Run: No mapped tasks were generated
   
   
![subsequent_dag_run_delete](https://github.com/apache/airflow/assets/25327813/ce1368f9-e790-48b3-ba78-deacfef631a3)
   
   ### **Temporary Resolution**
   When we delete the `removed` mapped task for `mapped_task_instance` in the 
previous dag run, scheduling resumed for `mapped_task_instance` for current dag 
run
   
   
   
   ### What you think should happen instead?
   
   When `depends_on_past` is set to True, mapped task in subsequent Dag Run 
should only depend on the overall status of the mapped task in the previous Dag 
Run.
   
   Mapped task in removed state may be an intended outcome from operational 
changes leading to rerun of the previous Dag Run.
   
   ### How to reproduce
   
   DAG Code for reproducing error in this word document
   
   [Uploading Removed Mapped Task issue in Airflow 2.7.3 (Airflow).docx…]()
   
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to