I added auto cherry-picking of dev and noticed that there are conflicts because we already missed some dev / ci change so I am also cherry-picking remaining changes in https://github.com/apache/airflow/pull/52293 - this way future auto-cherry-picks will be far easier to merge.
J. On Wed, Jun 25, 2025 at 11:23 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Nice, good discussion! > > Thanks & Regards, > Amogh Desai > > > On Wed, Jun 25, 2025 at 12:57 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > PR here to add backport label automatically for dev tools changes (and to > > maintain it with minor version upgrades) > > https://github.com/apache/airflow/pull/52189 > > > > On Tue, Jun 24, 2025 at 9:14 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Might be a good idea, yes. > > > > > > On Tue, Jun 24, 2025 at 9:09 PM Jens Scheffler > > <j_scheff...@gmx.de.invalid> > > > wrote: > > > > > >> Hi Jarek, > > >> > > >> one thought came into my mind... as you said it is a lot of trouble to > > >> find which "dev" PR might be missing on a mainteance branch... should > we > > >> cherry-pick dev tooling PRs by default as well to active maintenance > > >> branches? I mean... automating this in the CI like if `backport > to`label > > >> is set? > > >> > > >> Jens > > >> > > >> On 24.06.25 11:38, Jarek Potiuk wrote: > > >> > https://github.com/apache/airflow/pull/52148 -> Proposed PR. Let's > > >> iterate > > >> > there if there are comments. > > >> > > > >> > On Tue, Jun 24, 2025 at 11:25 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > >> > > > >> >> Yeah. I think what I am really after is expressed by Wei: +0.5 > vote > > - > > >> >> i.e. we should be fine with **some** selected cherry-picks that are > > not > > >> >> bug-fixes that we proactively cherry-pick and do not "complain" > that > > >> this > > >> >> is wrong because technically it's not a bugfix. I think It would be > > >> great > > >> >> if in every such cherry-pick we rationalise why we are doing it: > > >> >> > > >> >> "I know it's not a bugfix, but because this is an actively > developed > > >> part > > >> >> that is cherry-picked heavily, let me cherry-pick that one to make > it > > >> >> easier for others to cherry-pick bugfixes". > > >> >> > > >> >> If we are all on-board with that (seems so) then I can update the > > >> >> cherry-pick description to add it as possible exception. > > >> >> > > >> >> J. > > >> >> > > >> >> On Tue, Jun 24, 2025 at 10:14 AM Kaxil Naik <kaxiln...@gmail.com> > > >> wrote: > > >> >> > > >> >>> It looks like everyone is mostly on the same page based on all the > > >> emails > > >> >>> - > > >> >>> so no comments :) . > > >> >>> > > >> >>> On Tue, 24 Jun 2025 at 07:45, Wei Lee <weilee...@gmail.com> > wrote: > > >> >>> > > >> >>>> I’m +1 for 1-3 (assuming the doc changes relate to the backported > > >> >>> version). > > >> >>>> +0.5 for 4. I hope that changes not related to new features will > be > > >> >>>> backported when feasible; however, we can skip them if the > required > > >> >>> effort > > >> >>>> is substantial. This is because failing to backport these items > > could > > >> >>>> potentially lead to future conflicts with points 1 or 3. > > >> >>>> > > >> >>>> Best, > > >> >>>> Wei > > >> >>>> > > >> >>>>> On Jun 24, 2025, at 5:32 AM, Jens Scheffler > > >> >>> <j_scheff...@gmx.de.INVALID> > > >> >>>> wrote: > > >> >>>>> I am +1 for (1) to (3) [also assuming that (2) is mostly like a > > doc > > >> >>>> bugfix!] > > >> >>>>> For (4) I am hesitant and would rather be conservative. Every > > >> >>>> cherry-pick has a risk to break something in old codebase. As > > >> branches > > >> >>>> change over time and backport PRs are clearly less cautious > > reviewed > > >> it > > >> >>>> might lead to introduced inconsistencies which might not be > covered > > >> in > > >> >>>> tests. I would also not back-port just for the sake of easier > later > > >> >>>> maintenance for other cherry-picks. But for (4) the rule might > not > > be > > >> >>> too > > >> >>>> strict and every rule is made for exceptions which in (4) might > be > > >> >>> likely. > > >> >>>> But I would not advertise to backport refactorings if no clear > > >> benefit. > > >> >>>>> On 23.06.25 15:33, Pavankumar Gopidesu wrote: > > >> >>>>>> Thanks Jarek, for starting this discussion, > > >> >>>>>> > > >> >>>>>> I agree with all the points. > > >> >>>>>> > > >> >>>>>> The real intention behind to backport > > >> >>>>>> https://github.com/apache/airflow/pull/51992 is , this area > has > > a > > >> >>> lot > > >> >>>> of > > >> >>>>>> ongoing development going and I felt it's worth porting to > > >> v3-0-test. > > >> >>>>>> I myself faced situations where I tried to backport some doc > > >> changes > > >> >>> , I > > >> >>>>>> failed because of conflicts that previous changes were not > ported > > >> >>> back > > >> >>>> on > > >> >>>>>> the same file etc or conflicts with some lines. > > >> >>>>>> So I am very strong on this if there are any areas with heavy > > >> >>>> development > > >> >>>>>> activity going. I feel it's worth backporting the changes and > > IMHO > > >> i > > >> >>>>>> don't see any problem. > > >> >>>>>> > > >> >>>>>> Pavan > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> On Mon, Jun 23, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com > > > > >> >>> wrote: > > >> >>>>>>>> I think We are approaching it from the same point of view, > just > > >> >>> that > > >> >>>> we > > >> >>>>>>> have different conclusions. > > >> >>>>>>> > > >> >>>>>>> I agree. I think that simply there is one special case that we > > >> >>> should > > >> >>>>>>> "allow". Details below. > > >> >>>>>>> > > >> >>>>>>>> For 4, I do not have strong opinions on either front, but > > >> defining > > >> >>>> what > > >> >>>>>>> to > > >> >>>>>>>> do and what not should > > >> >>>>>>>> probably be that if that change makes it easy to backport > > >> something > > >> >>>> else, > > >> >>>>>>>> maybe have it? > > >> >>>>>>> Yep. That's **exactly** my proposal. I do not want to backport > > >> >>> **all** > > >> >>>>>>> refactorings. That would be stupid. In my definition of 4. I > > want > > >> to > > >> >>>>>>> cherry-pick those pre-commit when it's small, and automated > and > > >> when > > >> >>>> we can > > >> >>>>>>> easily anticipate it will make it quite likely someone (soon) > > will > > >> >>> do > > >> >>>>>>> another cherry-pick that will be conflicting. > > >> >>>>>>> > > >> >>>>>>> Yes. It's not "0/1" and quite a bit personal judgment, and yes > > it > > >> >>>> **might > > >> >>>>>>> not** result in conflict (so we might end up with YAGNI), but > I > > >> >>> think > > >> >>>> we > > >> >>>>>>> should define (and trust committers judgment) when they do it, > > >> >>> because > > >> >>>> they > > >> >>>>>>> want to prevent "others" to have problems. > > >> >>>>>>> > > >> >>>>>>> For example in case of > > >> https://github.com/apache/airflow/pull/51992 > > >> >>>> -> the > > >> >>>>>>> only reason why I think it makes sense to cherry-pick it is > > >> because > > >> >>> it > > >> >>>> is > > >> >>>>>>> in the area of task-sdk that has been recently bug-fixed > quite a > > >> >>>> number of > > >> >>>>>>> times and there are open related issues to this area that make > > it > > >> >>>> (IMHO) > > >> >>>>>>> quite likely to result in conflict. I think there are other > > areas > > >> >>> like > > >> >>>> that > > >> >>>>>>> (UI for example) where we clearly cherry-pick a number of > > changes, > > >> >>>> because > > >> >>>>>>> there are some pretty active "bug-fixing" in this area as 3.0 > > had > > >> >>>> still a > > >> >>>>>>> number of low priority but either known or anticipated bugs > that > > >> are > > >> >>>> likely > > >> >>>>>>> to be fixed in 3.0.3, 3.0.4 > > >> >>>>>>> > > >> >>>>>>> So just to clarify - I do not want to cherry-pick "all" > > >> refactoring, > > >> >>>> but > > >> >>>>>>> leave to the judgment of the commiter merging the request (and > > >> >>> author) > > >> >>>> to > > >> >>>>>>> decide to cherry-pick such change anticipating it will make > life > > >> >>>> easier for > > >> >>>>>>> others. > > >> >>>>>>> > > >> >>>>>>> J > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> On Mon, Jun 23, 2025 at 2:57 PM Amogh Desai < > > >> >>> amoghdesai....@gmail.com> > > >> >>>>>>> wrote: > > >> >>>>>>> > > >> >>>>>>>> Agree with point 1 - 3 definitely. > > >> >>>>>>>> > > >> >>>>>>>> For 4, I do not have strong opinions on either front, but > > >> defining > > >> >>>> what > > >> >>>>>>> to > > >> >>>>>>>> do and what not should > > >> >>>>>>>> probably be that if that change makes it easy to backport > > >> something > > >> >>>> else, > > >> >>>>>>>> maybe have it? > > >> >>>>>>>> > > >> >>>>>>>> For ex: > > >> >>>>>>>> *PR 1 changes file1.py and file2.py* > > >> >>>>>>>> *PR 2 changes some lines in file2.py* > > >> >>>>>>>> > > >> >>>>>>>> *We backport PR 1 as it's a bug fix and do not for PR 2 as > its > > >> some > > >> >>>>>>>> refactoring.* > > >> >>>>>>>> > > >> >>>>>>>> *Now while trying to backport PR 3 (bugfix), it conflicts and > > >> needs > > >> >>>> PR 2 > > >> >>>>>>> to > > >> >>>>>>>> be picked* > > >> >>>>>>>> *to land PR 3.* > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> Thanks & Regards, > > >> >>>>>>>> Amogh Desai > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> On Mon, Jun 23, 2025 at 5:02 PM Ash Berlin-Taylor < > > >> a...@apache.org> > > >> >>>>>>> wrote: > > >> >>>>>>>>> I think We are approaching it from the same point of view, > > just > > >> >>> that > > >> >>>> we > > >> >>>>>>>>> have different conclusions. > > >> >>>>>>>>> > > >> >>>>>>>>> Points 1-3 I agree with. > > >> >>>>>>>>> > > >> >>>>>>>>> We do already have this written up > > >> >>>>>>>>> > > >> >>> > > >> > > > https://github.com/apache/airflow/blob/130e9600443e06c08acc1b28c69a62c858d6e6a2/dev/README_RELEASE_AIRFLOW.md?plain=1#L116-L129 > > >> >>>>>>>>> On 4 I think of it in terms that “every change carries some > > >> risk” > > >> >>> — > > >> >>>> in > > >> >>>>>>>> the > > >> >>>>>>>>> linked example of 51992 the risk is almost zero, but > > generally: > > >> >>> 3.0.x > > >> >>>>>>> are > > >> >>>>>>>>> meant to be bug fix releases on top of 3.0.0, and if it’s > not > > >> >>> fixing > > >> >>>> a > > >> >>>>>>>> bug > > >> >>>>>>>>> we don’t back port it. The one exception I have to this is > if > > >> the > > >> >>>>>>> change > > >> >>>>>>>> is > > >> >>>>>>>>> needed to make it easy to backport a change that is a bug > > >> without > > >> >>>>>>>> conflicts. > > >> >>>>>>>>> I think our default approach has to be we don’t back port a > > >> change > > >> >>>>>>> unless > > >> >>>>>>>>> it is fixing a bug, otherwise the risk of “oh I’ll just fix > > >> this” > > >> >>>> ends > > >> >>>>>>> up > > >> >>>>>>>>> introducing more bugs than we fix. Stability of a Minor > > release > > >> >>>> series > > >> >>>>>>> is > > >> >>>>>>>>> my primary desire, and not changing things more than we have > > to > > >> is > > >> >>>> the > > >> >>>>>>>> best > > >> >>>>>>>>> way I know of doing that. > > >> >>>>>>>>> > > >> >>>>>>>>> Things are slightly different now that we have automated > > >> >>> cherry-picks > > >> >>>>>>> but > > >> >>>>>>>>> I still don’t think it is worth porting refactoring > > >> automatically. > > >> >>>> It’s > > >> >>>>>>>>> extra change and risk for almost zero benefit to users is my > > >> view. > > >> >>>>>>>>> > > >> >>>>>>>>> -ash > > >> >>>>>>>>> > > >> >>>>>>>>>> On 23 Jun 2025, at 11:43, Jarek Potiuk <ja...@potiuk.com> > > >> wrote: > > >> >>>>>>>>>> > > >> >>>>>>>>>> BTW. I'd be happy to capture result of this discussion if > we > > >> can > > >> >>> get > > >> >>>>>>>> to a > > >> >>>>>>>>>> consensus or vote eventually in the "cherry-picking" > > >> guidelines. > > >> >>>>>>>>>> > > >> >>>>>>>>>> On Mon, Jun 23, 2025 at 12:42 PM Jarek Potiuk < > > >> ja...@potiuk.com> > > >> >>>>>>>> wrote: > > >> >>>>>>>>>>> I wanted to start a discussion on "things that we > > cherry-pick" > > >> >>> (to > > >> >>>>>>>> vX_Z > > >> >>>>>>>>>>> branch). > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I think there are different opinions on what kind of > changes > > >> >>> should > > >> >>>>>>> be > > >> >>>>>>>>>>> cherry-picked and it might be a good idea to agree on a > > common > > >> >>>>>>>> approach. > > >> >>>>>>>>>>> I think (following the comment of Ash here) > > >> >>>>>>>>>>> > > >> >>> > > https://github.com/apache/airflow/pull/51992#issuecomment-2995632849 > > >> >>>>>>>>> that > > >> >>>>>>>>>>> we can use a very simplistic and (I'd say) dogmatic > approach > > >> >>> "only > > >> >>>>>>>>>>> cherry-pick bug fixes. Full stop". But I believe (and past > > >> >>>>>>> experience > > >> >>>>>>>>> from > > >> >>>>>>>>>>> a lot of cherry-picking that I've been doing - multiple > > times > > >> >>>>>>> helping > > >> >>>>>>>> to > > >> >>>>>>>>>>> bring past branches to be green and spending countless > hours > > >> on > > >> >>> it, > > >> >>>>>>>>> that it > > >> >>>>>>>>>>> should be a bit more nuanced. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I would love to see what others think, but from my > > experience > > >> >>> those > > >> >>>>>>>> are > > >> >>>>>>>>>>> the things that we **should** cherry-pick: > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> 1) bug-fixes (of course) > > >> >>>>>>>>>>> 2) doc changes (when they are improvements or filling > gaps) > > >> >>>>>>>>>>> 3) dev tool changes (every time we did not, it resulted in > > >> >>> hours of > > >> >>>>>>> my > > >> >>>>>>>>>>> time when things were breaking and I tried to reconcile > it) > > >> >>>>>>>>>>> 4) results of automated refactorings that have very low > > risks > > >> >>> (in > > >> >>>>>>> the > > >> >>>>>>>>>>> areas that are likely to have cherry-picks) > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> t) - is non-controversial I think > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> 2) - is also relatively non-controversial and very low > risk > > >> and > > >> >>>>>>> gives > > >> >>>>>>>>> our > > >> >>>>>>>>>>> users a chance to get better docs earlier (even today for > > >> >>> example I > > >> >>>>>>>>> cherry > > >> >>>>>>>>>>> picked this one: > > https://github.com/apache/airflow/pull/52068 > > >> - > > >> >>>>>>>> because > > >> >>>>>>>>>>> one of my friends who tries to learn Airflow 3 pinged me > > that > > >> >>>>>>>>>>> "ConfiuguringRuff" link that we have in 3.0.2 leads to 404 > > NOT > > >> >>>> found > > >> >>>>>>>>>>> 3) - it had always bitten us if we stopped cherry-picking > > dev > > >> >>> tool > > >> >>>>>>>>>>> changes. The thing is that external dependencies change > all > > >> the > > >> >>>> time > > >> >>>>>>>>> and we > > >> >>>>>>>>>>> are continuously catching up with those, also we improve, > > >> speed > > >> >>> up > > >> >>>>>>> and > > >> >>>>>>>>>>> simplify the tooling - and often things that worked when > > >> branch > > >> >>> was > > >> >>>>>>>> cut, > > >> >>>>>>>>>>> does not work today - countless, countless hours lost in > one > > >> or > > >> >>> two > > >> >>>>>>>>>>> branches when we stopped doing it - I think even once or > > >> twice I > > >> >>>> had > > >> >>>>>>>> to > > >> >>>>>>>>>>> just copy over most (but not all) the code from main to > the > > >> >>> branch > > >> >>>>>>> and > > >> >>>>>>>>>>> commit one single "catch-up dev tooling with main" big > > change > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> 4) Is likely most controversial - example here: > > >> >>>>>>>>>>> https://github.com/apache/airflow/pull/51992/ - those are > > the > > >> >>> kind > > >> >>>>>>> of > > >> >>>>>>>>>>> (really small) changes that are done in "active" area > (i.e. > > >> area > > >> >>>>>>> that > > >> >>>>>>>>> had > > >> >>>>>>>>>>> and will have a lot of cherry-picks anyway, but they are > > done > > >> >>> with > > >> >>>>>>>>>>> automated refactoring - like renaming variables and such. > > This > > >> >>>>>>>>> introduces > > >> >>>>>>>>>>> clarity and readability, so this is good we are doing > them. > > >> But > > >> >>> if > > >> >>>>>>> we > > >> >>>>>>>> do > > >> >>>>>>>>>>> not cherry-pick them and then we cherry-pick any change > that > > >> >>>> touches > > >> >>>>>>>> the > > >> >>>>>>>>>>> same code, this lead to a conflict. Conflicts are > > frustrating, > > >> >>>>>>>>> especially > > >> >>>>>>>>>>> those kinds - you never know what you should do - should > you > > >> >>>> "merge" > > >> >>>>>>>>> this > > >> >>>>>>>>>>> naming change with your change? or should you leave the > > >> original > > >> >>>>>>>>> namiing, > > >> >>>>>>>>>>> or should you try to find the past commit that changed it > > and > > >> >>>>>>>>> cherry-pick > > >> >>>>>>>>>>> it as well? This paired with the fact that we are using > > >> >>>>>>> cherry-picker > > >> >>>>>>>>> that > > >> >>>>>>>>>>> allows to cherry-pick stuff very quickly, automatically > and > > >> >>>>>>> painlessly > > >> >>>>>>>>> when > > >> >>>>>>>>>>> there are no conflicts, make me think that yes - we should > > >> >>> cherry > > >> >>>>>>>> -pick > > >> >>>>>>>>>>> those changes proactively as a service to those > contributors > > >> who > > >> >>>>>>> will > > >> >>>>>>>>>>> follow up with their cherry-picking. It's just "good > > service" > > >> >>> and > > >> >>>>>>>>> helping > > >> >>>>>>>>>>> others who will come after you. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> That's how my definition of "what we should cherry-pick" > > is... > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I wonder what others think about it ? > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> J. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>> > > >> >>> > > --------------------------------------------------------------------- > > >> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> >>>>>>>>> For additional commands, e-mail: > dev-h...@airflow.apache.org > > >> >>>>>>>>> > > >> >>>>>>>>> > > >> >>>>> > > >> --------------------------------------------------------------------- > > >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >> >>>>> > > >> >>>> > > >> >>>> > > --------------------------------------------------------------------- > > >> >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >> >>>> > > >> >>>> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > > >> > > >
