Yes! We use tables to generate some of our DAGs and it works well. What's most important is to avoid block-copying code among DAGs.
On Sat, Aug 27, 2016 at 12:05 PM, Vincent Poulain < [email protected]> wrote: > Hello, > First, thank you for your great job. Airflow made my job easier :) > > Is a dynamic generation of DAG a good practice ? As WePay explained there : > https://wecode.wepay.com/posts/airflow-wepay. > > In part of a DAG, I have *n* ECS tasks need to be launched daily, each one > with specific *params*. *n & params* depending on configuration which is > daily updated. > For now, I have just one task fetching this configuration and internally > broadcast ECS task. > I am wondering if Airflow would be suitable to generate DAG dynamically > (depending on configuration) in order to split this task into those > multiple ECS task (using ECSOperator, for instance). > > Thank you for your tips, > -- Lance Norskog [email protected] Redwood City, CA
