I have a DAG with 3 parallel branches. Each branch processes a file (multiple steps). Each branch is triggered by the arrival of one of three files which can happen at any time. After a file is processed, we have to trigger a database update (we use dbt).
I had a dbt operator at the end of each branch but this may result in two tasks running too close to each other. We need to wait at least 1h between updates. I'd like change the DAG so that each branch starts another DAG that will update the DB but I want this DAG to run with concurrency = 1 and wait at least 1 hour between successive runs. This way if two files arrive around the same time, the first one will trigger a database update and the second will trigger the second update but not immediately after the first one. The reason we need to wait 1 hour is that we have a process feeding data back every 30 minutes and need to wait for this data to be fed back before we run another update. Do you have a different idea to implement this? How do you suggest we implement the second DAG so that it won't run more than one DagRun at the same time and so that it will wait at least one hour after its prior execution before it runs again. Thanks! Pedro
