Hi David,
A few reasons why SubDagOperator was strongly discouraged in the past:
- No concurrency control, e.g. you cannot control the number of parallel
tasks in the subdag via pool or DAG concurrency. SubDagOperator used to
rely on the backfill scheduler which did not have any concurre
I find them to be very useful. I think it is an easy way to group a set of
tasks together that have a
one to many to one dependency structure. I find using a subdag to group the
many into a single task makes for a much cleaner dag and makes it easier to see
the status of the dag.
I read many o
Hi all,
Apologies if this topic has already been treated.
I want to create a solution for a data pipeline and subdags are perfect due
to it allows me to group the phases / tasks on functional meaning. Reading
documentation and other experiences in internet, strongly recommend to
avoid them, what
