I find them to be very useful. I think it is an easy way to group a set of tasks together that have a one to many to one dependency structure. I find using a subdag to group the many into a single task makes for a much cleaner dag and makes it easier to see the status of the dag. I read many of the same “warnings” when first implementing airflow but decided to give it a try. I rarely see any issues, but I also have a pretty small footprint running only about 20 dags. I do use the Celery Executor as well and assign all my sub dags to their own pool to try and make sure the sub dags don’t take up too many worker slots. I’m not sure how much of a difference that makes but it’s there.
James Coder > On Dec 29, 2019, at 12:48 PM, David Muñoz <[email protected]> wrote: > > Hi all, > > Apologies if this topic has already been treated. > > I want to create a solution for a data pipeline and subdags are perfect due > to it allows me to group the phases / tasks on functional meaning. Reading > documentation and other experiences in internet, strongly recommend to > avoid them, what do you think? > > Thanks in advance. > > Kind regards. > > David.
