I agree with Gerard on all fronts. SubDags are difficult, slow, and can cause a lot of strange edge cases. The difficulties in SubDags is a sticking point for Airflow competitors that I want to remove as quickly as possible. I think that 2.0 is a perfect time to introduce TaskGroups as people are already prepared mentally for breaking changes/new functionality.
I don’t think we need to give a specific “this will be around until 3.0” message either. We can just say it WILL be taken out in a future version and we can decide later whether that means 2.1 or 3.0. I really don’t like the idea of setting a feature for deprecation in a minor release. I also think that even if TaskGroups ends up not being the answer, it’s very likely that it will be a different feature, rather than a reversion to SubDags. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2] On Fri, Sep 18, 2020 at 2:16 PM, Gerard Casas Saez <gcasass...@twitter.com.invalid> wrote: Internally, we have blocked people from using SubDags due to its slowness and sometimes instability. I would vote for adding TaskGroups earliest as possible + remove documentation for SubDags (users can go to old versions of docs to find that) + add a notice of deprecation on new TaskGroup documentation page + Add deprecation warning on import/usage SubDag. I would vote for not having to wait until 3.X for deprecating it as it's difficult to use and adds complexity that may not be needed. Instead schedule removal at 2.1 or so. This is the take from our side that we don't have SubDags, would be curious on people usages of SubDags and to check if there's any use case that does not fit TaskGroups. Gerard Casas Saez Twitter | Cortex | @casassaez <http://twitter.com/casassaez> On Fri, Sep 18, 2020 at 6:34 AM Kaxil Naik <kaxiln...@apache.org> wrote: > Hi all, > > One of the things we discussed on Monday's Airflow 2.0 Dev call was around > *TaskGroups* - a new concept introduced by > https://github.com/apache/airflow/pull/10153 (AIP-34 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > >) > from > Qian Yu. > > There was a point raised in the meeting that we should deprecate SubDags in > favour of TaskGroups since TaskGroups are UI-only concept and do not impact > scheduling decisions or have known limitations around Concurrency / Pools > limits like SubDags. > > Having thought. a little bit more after the meeting, I think we should > hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see > how users feel about TaskGroup and we, the developers would also know more > about limitations of it (if any). > > We can then issue a Deprecation Warning and remove SubDags eventually in > Airflow 3.0. Until then both can live in the codebase and we document the > TaskGroups better. > > What do you all think? > > Meeting Notes from that call are at: > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020> > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020 > > > > Regards, > Kaxil >
