Question on dynamic tasks in a DAG and wait_for_downstream --- wait_on_downstream waits forever for dynamically added tasks in later dag runs.

Hello Team, Currently we are using the airflow version - 1.10.10 to data ingest. In our DAG, we create tasks dynamically based on data volume , i.e if data volume is high, number of parallel tasks increases and if the data volume is less the number of parallel tasks reduces in the next run or

Re: Question on dynamic tasks in a DAG and wait_for_downstream

Hi Sibarata - I discussed your use case with some of our data engineers, we might have some recommendations on how to better execute this - let us know if you want to chat. -Ry Airflow Committer + Founder/CTO of Astronomer On Mon, Sep 7, 2020 at 4:30 AM Sibabrata Pattanaik (spattana) wrote: >

Question on dynamic tasks in a DAG and wait_for_downstream

Hello Team, Currently we are using airflow version - 1.10.10 to data ingest. In our DAG, we create tasks dynamically based on data volume , i.e if data volume is high, number of parallel tasks increases and if the data volume is less number of parallel tasks reduces in the next run or vice vers