PR here to add backport label automatically for dev tools changes (and to maintain it with minor version upgrades) https://github.com/apache/airflow/pull/52189
On Tue, Jun 24, 2025 at 9:14 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Might be a good idea, yes. > > On Tue, Jun 24, 2025 at 9:09 PM Jens Scheffler <j_scheff...@gmx.de.invalid> > wrote: > >> Hi Jarek, >> >> one thought came into my mind... as you said it is a lot of trouble to >> find which "dev" PR might be missing on a mainteance branch... should we >> cherry-pick dev tooling PRs by default as well to active maintenance >> branches? I mean... automating this in the CI like if `backport to`label >> is set? >> >> Jens >> >> On 24.06.25 11:38, Jarek Potiuk wrote: >> > https://github.com/apache/airflow/pull/52148 -> Proposed PR. Let's >> iterate >> > there if there are comments. >> > >> > On Tue, Jun 24, 2025 at 11:25 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> >> Yeah. I think what I am really after is expressed by Wei: +0.5 vote - >> >> i.e. we should be fine with **some** selected cherry-picks that are not >> >> bug-fixes that we proactively cherry-pick and do not "complain" that >> this >> >> is wrong because technically it's not a bugfix. I think It would be >> great >> >> if in every such cherry-pick we rationalise why we are doing it: >> >> >> >> "I know it's not a bugfix, but because this is an actively developed >> part >> >> that is cherry-picked heavily, let me cherry-pick that one to make it >> >> easier for others to cherry-pick bugfixes". >> >> >> >> If we are all on-board with that (seems so) then I can update the >> >> cherry-pick description to add it as possible exception. >> >> >> >> J. >> >> >> >> On Tue, Jun 24, 2025 at 10:14 AM Kaxil Naik <kaxiln...@gmail.com> >> wrote: >> >> >> >>> It looks like everyone is mostly on the same page based on all the >> emails >> >>> - >> >>> so no comments :) . >> >>> >> >>> On Tue, 24 Jun 2025 at 07:45, Wei Lee <weilee...@gmail.com> wrote: >> >>> >> >>>> I’m +1 for 1-3 (assuming the doc changes relate to the backported >> >>> version). >> >>>> +0.5 for 4. I hope that changes not related to new features will be >> >>>> backported when feasible; however, we can skip them if the required >> >>> effort >> >>>> is substantial. This is because failing to backport these items could >> >>>> potentially lead to future conflicts with points 1 or 3. >> >>>> >> >>>> Best, >> >>>> Wei >> >>>> >> >>>>> On Jun 24, 2025, at 5:32 AM, Jens Scheffler >> >>> <j_scheff...@gmx.de.INVALID> >> >>>> wrote: >> >>>>> I am +1 for (1) to (3) [also assuming that (2) is mostly like a doc >> >>>> bugfix!] >> >>>>> For (4) I am hesitant and would rather be conservative. Every >> >>>> cherry-pick has a risk to break something in old codebase. As >> branches >> >>>> change over time and backport PRs are clearly less cautious reviewed >> it >> >>>> might lead to introduced inconsistencies which might not be covered >> in >> >>>> tests. I would also not back-port just for the sake of easier later >> >>>> maintenance for other cherry-picks. But for (4) the rule might not be >> >>> too >> >>>> strict and every rule is made for exceptions which in (4) might be >> >>> likely. >> >>>> But I would not advertise to backport refactorings if no clear >> benefit. >> >>>>> On 23.06.25 15:33, Pavankumar Gopidesu wrote: >> >>>>>> Thanks Jarek, for starting this discussion, >> >>>>>> >> >>>>>> I agree with all the points. >> >>>>>> >> >>>>>> The real intention behind to backport >> >>>>>> https://github.com/apache/airflow/pull/51992 is , this area has a >> >>> lot >> >>>> of >> >>>>>> ongoing development going and I felt it's worth porting to >> v3-0-test. >> >>>>>> I myself faced situations where I tried to backport some doc >> changes >> >>> , I >> >>>>>> failed because of conflicts that previous changes were not ported >> >>> back >> >>>> on >> >>>>>> the same file etc or conflicts with some lines. >> >>>>>> So I am very strong on this if there are any areas with heavy >> >>>> development >> >>>>>> activity going. I feel it's worth backporting the changes and IMHO >> i >> >>>>>> don't see any problem. >> >>>>>> >> >>>>>> Pavan >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Jun 23, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> >> >>> wrote: >> >>>>>>>> I think We are approaching it from the same point of view, just >> >>> that >> >>>> we >> >>>>>>> have different conclusions. >> >>>>>>> >> >>>>>>> I agree. I think that simply there is one special case that we >> >>> should >> >>>>>>> "allow". Details below. >> >>>>>>> >> >>>>>>>> For 4, I do not have strong opinions on either front, but >> defining >> >>>> what >> >>>>>>> to >> >>>>>>>> do and what not should >> >>>>>>>> probably be that if that change makes it easy to backport >> something >> >>>> else, >> >>>>>>>> maybe have it? >> >>>>>>> Yep. That's **exactly** my proposal. I do not want to backport >> >>> **all** >> >>>>>>> refactorings. That would be stupid. In my definition of 4. I want >> to >> >>>>>>> cherry-pick those pre-commit when it's small, and automated and >> when >> >>>> we can >> >>>>>>> easily anticipate it will make it quite likely someone (soon) will >> >>> do >> >>>>>>> another cherry-pick that will be conflicting. >> >>>>>>> >> >>>>>>> Yes. It's not "0/1" and quite a bit personal judgment, and yes it >> >>>> **might >> >>>>>>> not** result in conflict (so we might end up with YAGNI), but I >> >>> think >> >>>> we >> >>>>>>> should define (and trust committers judgment) when they do it, >> >>> because >> >>>> they >> >>>>>>> want to prevent "others" to have problems. >> >>>>>>> >> >>>>>>> For example in case of >> https://github.com/apache/airflow/pull/51992 >> >>>> -> the >> >>>>>>> only reason why I think it makes sense to cherry-pick it is >> because >> >>> it >> >>>> is >> >>>>>>> in the area of task-sdk that has been recently bug-fixed quite a >> >>>> number of >> >>>>>>> times and there are open related issues to this area that make it >> >>>> (IMHO) >> >>>>>>> quite likely to result in conflict. I think there are other areas >> >>> like >> >>>> that >> >>>>>>> (UI for example) where we clearly cherry-pick a number of changes, >> >>>> because >> >>>>>>> there are some pretty active "bug-fixing" in this area as 3.0 had >> >>>> still a >> >>>>>>> number of low priority but either known or anticipated bugs that >> are >> >>>> likely >> >>>>>>> to be fixed in 3.0.3, 3.0.4 >> >>>>>>> >> >>>>>>> So just to clarify - I do not want to cherry-pick "all" >> refactoring, >> >>>> but >> >>>>>>> leave to the judgment of the commiter merging the request (and >> >>> author) >> >>>> to >> >>>>>>> decide to cherry-pick such change anticipating it will make life >> >>>> easier for >> >>>>>>> others. >> >>>>>>> >> >>>>>>> J >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Jun 23, 2025 at 2:57 PM Amogh Desai < >> >>> amoghdesai....@gmail.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Agree with point 1 - 3 definitely. >> >>>>>>>> >> >>>>>>>> For 4, I do not have strong opinions on either front, but >> defining >> >>>> what >> >>>>>>> to >> >>>>>>>> do and what not should >> >>>>>>>> probably be that if that change makes it easy to backport >> something >> >>>> else, >> >>>>>>>> maybe have it? >> >>>>>>>> >> >>>>>>>> For ex: >> >>>>>>>> *PR 1 changes file1.py and file2.py* >> >>>>>>>> *PR 2 changes some lines in file2.py* >> >>>>>>>> >> >>>>>>>> *We backport PR 1 as it's a bug fix and do not for PR 2 as its >> some >> >>>>>>>> refactoring.* >> >>>>>>>> >> >>>>>>>> *Now while trying to backport PR 3 (bugfix), it conflicts and >> needs >> >>>> PR 2 >> >>>>>>> to >> >>>>>>>> be picked* >> >>>>>>>> *to land PR 3.* >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Thanks & Regards, >> >>>>>>>> Amogh Desai >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Mon, Jun 23, 2025 at 5:02 PM Ash Berlin-Taylor < >> a...@apache.org> >> >>>>>>> wrote: >> >>>>>>>>> I think We are approaching it from the same point of view, just >> >>> that >> >>>> we >> >>>>>>>>> have different conclusions. >> >>>>>>>>> >> >>>>>>>>> Points 1-3 I agree with. >> >>>>>>>>> >> >>>>>>>>> We do already have this written up >> >>>>>>>>> >> >>> >> https://github.com/apache/airflow/blob/130e9600443e06c08acc1b28c69a62c858d6e6a2/dev/README_RELEASE_AIRFLOW.md?plain=1#L116-L129 >> >>>>>>>>> On 4 I think of it in terms that “every change carries some >> risk” >> >>> — >> >>>> in >> >>>>>>>> the >> >>>>>>>>> linked example of 51992 the risk is almost zero, but generally: >> >>> 3.0.x >> >>>>>>> are >> >>>>>>>>> meant to be bug fix releases on top of 3.0.0, and if it’s not >> >>> fixing >> >>>> a >> >>>>>>>> bug >> >>>>>>>>> we don’t back port it. The one exception I have to this is if >> the >> >>>>>>> change >> >>>>>>>> is >> >>>>>>>>> needed to make it easy to backport a change that is a bug >> without >> >>>>>>>> conflicts. >> >>>>>>>>> I think our default approach has to be we don’t back port a >> change >> >>>>>>> unless >> >>>>>>>>> it is fixing a bug, otherwise the risk of “oh I’ll just fix >> this” >> >>>> ends >> >>>>>>> up >> >>>>>>>>> introducing more bugs than we fix. Stability of a Minor release >> >>>> series >> >>>>>>> is >> >>>>>>>>> my primary desire, and not changing things more than we have to >> is >> >>>> the >> >>>>>>>> best >> >>>>>>>>> way I know of doing that. >> >>>>>>>>> >> >>>>>>>>> Things are slightly different now that we have automated >> >>> cherry-picks >> >>>>>>> but >> >>>>>>>>> I still don’t think it is worth porting refactoring >> automatically. >> >>>> It’s >> >>>>>>>>> extra change and risk for almost zero benefit to users is my >> view. >> >>>>>>>>> >> >>>>>>>>> -ash >> >>>>>>>>> >> >>>>>>>>>> On 23 Jun 2025, at 11:43, Jarek Potiuk <ja...@potiuk.com> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> BTW. I'd be happy to capture result of this discussion if we >> can >> >>> get >> >>>>>>>> to a >> >>>>>>>>>> consensus or vote eventually in the "cherry-picking" >> guidelines. >> >>>>>>>>>> >> >>>>>>>>>> On Mon, Jun 23, 2025 at 12:42 PM Jarek Potiuk < >> ja...@potiuk.com> >> >>>>>>>> wrote: >> >>>>>>>>>>> I wanted to start a discussion on "things that we cherry-pick" >> >>> (to >> >>>>>>>> vX_Z >> >>>>>>>>>>> branch). >> >>>>>>>>>>> >> >>>>>>>>>>> I think there are different opinions on what kind of changes >> >>> should >> >>>>>>> be >> >>>>>>>>>>> cherry-picked and it might be a good idea to agree on a common >> >>>>>>>> approach. >> >>>>>>>>>>> I think (following the comment of Ash here) >> >>>>>>>>>>> >> >>> https://github.com/apache/airflow/pull/51992#issuecomment-2995632849 >> >>>>>>>>> that >> >>>>>>>>>>> we can use a very simplistic and (I'd say) dogmatic approach >> >>> "only >> >>>>>>>>>>> cherry-pick bug fixes. Full stop". But I believe (and past >> >>>>>>> experience >> >>>>>>>>> from >> >>>>>>>>>>> a lot of cherry-picking that I've been doing - multiple times >> >>>>>>> helping >> >>>>>>>> to >> >>>>>>>>>>> bring past branches to be green and spending countless hours >> on >> >>> it, >> >>>>>>>>> that it >> >>>>>>>>>>> should be a bit more nuanced. >> >>>>>>>>>>> >> >>>>>>>>>>> I would love to see what others think, but from my experience >> >>> those >> >>>>>>>> are >> >>>>>>>>>>> the things that we **should** cherry-pick: >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> 1) bug-fixes (of course) >> >>>>>>>>>>> 2) doc changes (when they are improvements or filling gaps) >> >>>>>>>>>>> 3) dev tool changes (every time we did not, it resulted in >> >>> hours of >> >>>>>>> my >> >>>>>>>>>>> time when things were breaking and I tried to reconcile it) >> >>>>>>>>>>> 4) results of automated refactorings that have very low risks >> >>> (in >> >>>>>>> the >> >>>>>>>>>>> areas that are likely to have cherry-picks) >> >>>>>>>>>>> >> >>>>>>>>>>> t) - is non-controversial I think >> >>>>>>>>>>> >> >>>>>>>>>>> 2) - is also relatively non-controversial and very low risk >> and >> >>>>>>> gives >> >>>>>>>>> our >> >>>>>>>>>>> users a chance to get better docs earlier (even today for >> >>> example I >> >>>>>>>>> cherry >> >>>>>>>>>>> picked this one: https://github.com/apache/airflow/pull/52068 >> - >> >>>>>>>> because >> >>>>>>>>>>> one of my friends who tries to learn Airflow 3 pinged me that >> >>>>>>>>>>> "ConfiuguringRuff" link that we have in 3.0.2 leads to 404 NOT >> >>>> found >> >>>>>>>>>>> 3) - it had always bitten us if we stopped cherry-picking dev >> >>> tool >> >>>>>>>>>>> changes. The thing is that external dependencies change all >> the >> >>>> time >> >>>>>>>>> and we >> >>>>>>>>>>> are continuously catching up with those, also we improve, >> speed >> >>> up >> >>>>>>> and >> >>>>>>>>>>> simplify the tooling - and often things that worked when >> branch >> >>> was >> >>>>>>>> cut, >> >>>>>>>>>>> does not work today - countless, countless hours lost in one >> or >> >>> two >> >>>>>>>>>>> branches when we stopped doing it - I think even once or >> twice I >> >>>> had >> >>>>>>>> to >> >>>>>>>>>>> just copy over most (but not all) the code from main to the >> >>> branch >> >>>>>>> and >> >>>>>>>>>>> commit one single "catch-up dev tooling with main" big change >> >>>>>>>>>>> >> >>>>>>>>>>> 4) Is likely most controversial - example here: >> >>>>>>>>>>> https://github.com/apache/airflow/pull/51992/ - those are the >> >>> kind >> >>>>>>> of >> >>>>>>>>>>> (really small) changes that are done in "active" area (i.e. >> area >> >>>>>>> that >> >>>>>>>>> had >> >>>>>>>>>>> and will have a lot of cherry-picks anyway, but they are done >> >>> with >> >>>>>>>>>>> automated refactoring - like renaming variables and such. This >> >>>>>>>>> introduces >> >>>>>>>>>>> clarity and readability, so this is good we are doing them. >> But >> >>> if >> >>>>>>> we >> >>>>>>>> do >> >>>>>>>>>>> not cherry-pick them and then we cherry-pick any change that >> >>>> touches >> >>>>>>>> the >> >>>>>>>>>>> same code, this lead to a conflict. Conflicts are frustrating, >> >>>>>>>>> especially >> >>>>>>>>>>> those kinds - you never know what you should do - should you >> >>>> "merge" >> >>>>>>>>> this >> >>>>>>>>>>> naming change with your change? or should you leave the >> original >> >>>>>>>>> namiing, >> >>>>>>>>>>> or should you try to find the past commit that changed it and >> >>>>>>>>> cherry-pick >> >>>>>>>>>>> it as well? This paired with the fact that we are using >> >>>>>>> cherry-picker >> >>>>>>>>> that >> >>>>>>>>>>> allows to cherry-pick stuff very quickly, automatically and >> >>>>>>> painlessly >> >>>>>>>>> when >> >>>>>>>>>>> there are no conflicts, make me think that yes - we should >> >>> cherry >> >>>>>>>> -pick >> >>>>>>>>>>> those changes proactively as a service to those contributors >> who >> >>>>>>> will >> >>>>>>>>>>> follow up with their cherry-picking. It's just "good service" >> >>> and >> >>>>>>>>> helping >> >>>>>>>>>>> others who will come after you. >> >>>>>>>>>>> >> >>>>>>>>>>> That's how my definition of "what we should cherry-pick" is... >> >>>>>>>>>>> >> >>>>>>>>>>> I wonder what others think about it ? >> >>>>>>>>>>> >> >>>>>>>>>>> J. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>> --------------------------------------------------------------------- >> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> >>>>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>>>>>>>> >> >>>>>>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> >>>> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>>> >> >>>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
