One thing only here. While I am all for removing Sub-dags eventually, We either remove them in 2.0 or 3.0. We can't remove any functionality or change Airflow in a backwards-incompatible way in 2.X ( https://semver.org/#spec-item-8)
We already agreed I believe, that we are using Semver <https://semver.org/> and that basically means that there are no intentional backwards-incompatible changes when Minor version is increased. I think we ourselves experienced that quite a number of times when our dependencies did not keep to their semver promise, so let's not do that to others :). Just a thought - I wonder if deprecating SubDags now and removing it in 2.0 is a possibility? I think the more we can remove from Airflow when getting to 2.0 - the better. We still have a chance to ADD stuff in 2.1 and beyond. It can then be better thought, more integrated with the future of Airflow as we see it. If we will get information from the users that they miss it after we removed it in 2.0 and they cannot live without it and cannot migrate to 2.0 without it, we can always add it. We have a viable TaskGroup replacement now. And maybe we can even get some semi-automated conversions SubDags -> TaskGroups. J. On Fri, Sep 18, 2020 at 11:30 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > I agree with Gerard on all fronts. > > SubDags are difficult, slow, and can cause a lot of strange edge cases. > The difficulties in SubDags is a sticking point for Airflow competitors > that I want to remove as quickly as possible. I think that 2.0 is a perfect > time to introduce TaskGroups as people are already prepared mentally for > breaking changes/new functionality. > > I don’t think we need to give a specific “this will be around until 3.0” > message either. We can just say it WILL be taken out in a future version > and we can decide later whether that means 2.1 or 3.0. I really don’t like > the idea of setting a feature for deprecation in a minor release. I also > think that even if TaskGroups ends up not being the answer, it’s very > likely that it will be a different feature, rather than a reversion to > SubDags. > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2 > ] > On Fri, Sep 18, 2020 at 2:16 PM, Gerard Casas Saez > <gcasass...@twitter.com.invalid> > wrote: > Internally, we have blocked people from using SubDags due to its slowness > and sometimes instability. I would vote for adding TaskGroups earliest as > possible + remove documentation for SubDags (users can go to old versions > of docs to find that) + add a notice of deprecation on new TaskGroup > documentation page + Add deprecation warning on import/usage SubDag. > > I would vote for not having to wait until 3.X for deprecating it as it's > difficult to use and adds complexity that may not be needed. Instead > schedule removal at 2.1 or so. > > This is the take from our side that we don't have SubDags, would be curious > on people usages of SubDags and to check if there's any use case that does > not fit TaskGroups. > > Gerard Casas Saez > Twitter | Cortex | @casassaez <http://twitter.com/casassaez> > > > On Fri, Sep 18, 2020 at 6:34 AM Kaxil Naik <kaxiln...@apache.org> wrote: > > > Hi all, > > > > One of the things we discussed on Monday's Airflow 2.0 Dev call was > around > > *TaskGroups* - a new concept introduced by > > https://github.com/apache/airflow/pull/10153 (AIP-34 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > > >) > > from > > Qian Yu. > > > > There was a point raised in the meeting that we should deprecate SubDags > in > > favour of TaskGroups since TaskGroups are UI-only concept and do not > impact > > scheduling decisions or have known limitations around Concurrency / Pools > > limits like SubDags. > > > > Having thought. a little bit more after the meeting, I think we should > > hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see > > how users feel about TaskGroup and we, the developers would also know > more > > about limitations of it (if any). > > > > We can then issue a Deprecation Warning and remove SubDags eventually in > > Airflow 3.0. Until then both can live in the codebase and we document the > > TaskGroups better. > > > > What do you all think? > > > > Meeting Notes from that call are at: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020> > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020 > > > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020 > > > > > > > Regards, > > Kaxil > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
