Might be a good idea, yes. On Tue, Jun 24, 2025 at 9:09 PM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote:
> Hi Jarek, > > one thought came into my mind... as you said it is a lot of trouble to > find which "dev" PR might be missing on a mainteance branch... should we > cherry-pick dev tooling PRs by default as well to active maintenance > branches? I mean... automating this in the CI like if `backport to`label > is set? > > Jens > > On 24.06.25 11:38, Jarek Potiuk wrote: > > https://github.com/apache/airflow/pull/52148 -> Proposed PR. Let's > iterate > > there if there are comments. > > > > On Tue, Jun 24, 2025 at 11:25 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> Yeah. I think what I am really after is expressed by Wei: +0.5 vote - > >> i.e. we should be fine with **some** selected cherry-picks that are not > >> bug-fixes that we proactively cherry-pick and do not "complain" that > this > >> is wrong because technically it's not a bugfix. I think It would be > great > >> if in every such cherry-pick we rationalise why we are doing it: > >> > >> "I know it's not a bugfix, but because this is an actively developed > part > >> that is cherry-picked heavily, let me cherry-pick that one to make it > >> easier for others to cherry-pick bugfixes". > >> > >> If we are all on-board with that (seems so) then I can update the > >> cherry-pick description to add it as possible exception. > >> > >> J. > >> > >> On Tue, Jun 24, 2025 at 10:14 AM Kaxil Naik <kaxiln...@gmail.com> > wrote: > >> > >>> It looks like everyone is mostly on the same page based on all the > emails > >>> - > >>> so no comments :) . > >>> > >>> On Tue, 24 Jun 2025 at 07:45, Wei Lee <weilee...@gmail.com> wrote: > >>> > >>>> I’m +1 for 1-3 (assuming the doc changes relate to the backported > >>> version). > >>>> +0.5 for 4. I hope that changes not related to new features will be > >>>> backported when feasible; however, we can skip them if the required > >>> effort > >>>> is substantial. This is because failing to backport these items could > >>>> potentially lead to future conflicts with points 1 or 3. > >>>> > >>>> Best, > >>>> Wei > >>>> > >>>>> On Jun 24, 2025, at 5:32 AM, Jens Scheffler > >>> <j_scheff...@gmx.de.INVALID> > >>>> wrote: > >>>>> I am +1 for (1) to (3) [also assuming that (2) is mostly like a doc > >>>> bugfix!] > >>>>> For (4) I am hesitant and would rather be conservative. Every > >>>> cherry-pick has a risk to break something in old codebase. As branches > >>>> change over time and backport PRs are clearly less cautious reviewed > it > >>>> might lead to introduced inconsistencies which might not be covered in > >>>> tests. I would also not back-port just for the sake of easier later > >>>> maintenance for other cherry-picks. But for (4) the rule might not be > >>> too > >>>> strict and every rule is made for exceptions which in (4) might be > >>> likely. > >>>> But I would not advertise to backport refactorings if no clear > benefit. > >>>>> On 23.06.25 15:33, Pavankumar Gopidesu wrote: > >>>>>> Thanks Jarek, for starting this discussion, > >>>>>> > >>>>>> I agree with all the points. > >>>>>> > >>>>>> The real intention behind to backport > >>>>>> https://github.com/apache/airflow/pull/51992 is , this area has a > >>> lot > >>>> of > >>>>>> ongoing development going and I felt it's worth porting to > v3-0-test. > >>>>>> I myself faced situations where I tried to backport some doc changes > >>> , I > >>>>>> failed because of conflicts that previous changes were not ported > >>> back > >>>> on > >>>>>> the same file etc or conflicts with some lines. > >>>>>> So I am very strong on this if there are any areas with heavy > >>>> development > >>>>>> activity going. I feel it's worth backporting the changes and IMHO i > >>>>>> don't see any problem. > >>>>>> > >>>>>> Pavan > >>>>>> > >>>>>> > >>>>>> On Mon, Jun 23, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> > >>> wrote: > >>>>>>>> I think We are approaching it from the same point of view, just > >>> that > >>>> we > >>>>>>> have different conclusions. > >>>>>>> > >>>>>>> I agree. I think that simply there is one special case that we > >>> should > >>>>>>> "allow". Details below. > >>>>>>> > >>>>>>>> For 4, I do not have strong opinions on either front, but defining > >>>> what > >>>>>>> to > >>>>>>>> do and what not should > >>>>>>>> probably be that if that change makes it easy to backport > something > >>>> else, > >>>>>>>> maybe have it? > >>>>>>> Yep. That's **exactly** my proposal. I do not want to backport > >>> **all** > >>>>>>> refactorings. That would be stupid. In my definition of 4. I want > to > >>>>>>> cherry-pick those pre-commit when it's small, and automated and > when > >>>> we can > >>>>>>> easily anticipate it will make it quite likely someone (soon) will > >>> do > >>>>>>> another cherry-pick that will be conflicting. > >>>>>>> > >>>>>>> Yes. It's not "0/1" and quite a bit personal judgment, and yes it > >>>> **might > >>>>>>> not** result in conflict (so we might end up with YAGNI), but I > >>> think > >>>> we > >>>>>>> should define (and trust committers judgment) when they do it, > >>> because > >>>> they > >>>>>>> want to prevent "others" to have problems. > >>>>>>> > >>>>>>> For example in case of > https://github.com/apache/airflow/pull/51992 > >>>> -> the > >>>>>>> only reason why I think it makes sense to cherry-pick it is because > >>> it > >>>> is > >>>>>>> in the area of task-sdk that has been recently bug-fixed quite a > >>>> number of > >>>>>>> times and there are open related issues to this area that make it > >>>> (IMHO) > >>>>>>> quite likely to result in conflict. I think there are other areas > >>> like > >>>> that > >>>>>>> (UI for example) where we clearly cherry-pick a number of changes, > >>>> because > >>>>>>> there are some pretty active "bug-fixing" in this area as 3.0 had > >>>> still a > >>>>>>> number of low priority but either known or anticipated bugs that > are > >>>> likely > >>>>>>> to be fixed in 3.0.3, 3.0.4 > >>>>>>> > >>>>>>> So just to clarify - I do not want to cherry-pick "all" > refactoring, > >>>> but > >>>>>>> leave to the judgment of the commiter merging the request (and > >>> author) > >>>> to > >>>>>>> decide to cherry-pick such change anticipating it will make life > >>>> easier for > >>>>>>> others. > >>>>>>> > >>>>>>> J > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Jun 23, 2025 at 2:57 PM Amogh Desai < > >>> amoghdesai....@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Agree with point 1 - 3 definitely. > >>>>>>>> > >>>>>>>> For 4, I do not have strong opinions on either front, but defining > >>>> what > >>>>>>> to > >>>>>>>> do and what not should > >>>>>>>> probably be that if that change makes it easy to backport > something > >>>> else, > >>>>>>>> maybe have it? > >>>>>>>> > >>>>>>>> For ex: > >>>>>>>> *PR 1 changes file1.py and file2.py* > >>>>>>>> *PR 2 changes some lines in file2.py* > >>>>>>>> > >>>>>>>> *We backport PR 1 as it's a bug fix and do not for PR 2 as its > some > >>>>>>>> refactoring.* > >>>>>>>> > >>>>>>>> *Now while trying to backport PR 3 (bugfix), it conflicts and > needs > >>>> PR 2 > >>>>>>> to > >>>>>>>> be picked* > >>>>>>>> *to land PR 3.* > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks & Regards, > >>>>>>>> Amogh Desai > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, Jun 23, 2025 at 5:02 PM Ash Berlin-Taylor <a...@apache.org > > > >>>>>>> wrote: > >>>>>>>>> I think We are approaching it from the same point of view, just > >>> that > >>>> we > >>>>>>>>> have different conclusions. > >>>>>>>>> > >>>>>>>>> Points 1-3 I agree with. > >>>>>>>>> > >>>>>>>>> We do already have this written up > >>>>>>>>> > >>> > https://github.com/apache/airflow/blob/130e9600443e06c08acc1b28c69a62c858d6e6a2/dev/README_RELEASE_AIRFLOW.md?plain=1#L116-L129 > >>>>>>>>> On 4 I think of it in terms that “every change carries some risk” > >>> — > >>>> in > >>>>>>>> the > >>>>>>>>> linked example of 51992 the risk is almost zero, but generally: > >>> 3.0.x > >>>>>>> are > >>>>>>>>> meant to be bug fix releases on top of 3.0.0, and if it’s not > >>> fixing > >>>> a > >>>>>>>> bug > >>>>>>>>> we don’t back port it. The one exception I have to this is if the > >>>>>>> change > >>>>>>>> is > >>>>>>>>> needed to make it easy to backport a change that is a bug without > >>>>>>>> conflicts. > >>>>>>>>> I think our default approach has to be we don’t back port a > change > >>>>>>> unless > >>>>>>>>> it is fixing a bug, otherwise the risk of “oh I’ll just fix this” > >>>> ends > >>>>>>> up > >>>>>>>>> introducing more bugs than we fix. Stability of a Minor release > >>>> series > >>>>>>> is > >>>>>>>>> my primary desire, and not changing things more than we have to > is > >>>> the > >>>>>>>> best > >>>>>>>>> way I know of doing that. > >>>>>>>>> > >>>>>>>>> Things are slightly different now that we have automated > >>> cherry-picks > >>>>>>> but > >>>>>>>>> I still don’t think it is worth porting refactoring > automatically. > >>>> It’s > >>>>>>>>> extra change and risk for almost zero benefit to users is my > view. > >>>>>>>>> > >>>>>>>>> -ash > >>>>>>>>> > >>>>>>>>>> On 23 Jun 2025, at 11:43, Jarek Potiuk <ja...@potiuk.com> > wrote: > >>>>>>>>>> > >>>>>>>>>> BTW. I'd be happy to capture result of this discussion if we can > >>> get > >>>>>>>> to a > >>>>>>>>>> consensus or vote eventually in the "cherry-picking" guidelines. > >>>>>>>>>> > >>>>>>>>>> On Mon, Jun 23, 2025 at 12:42 PM Jarek Potiuk <ja...@potiuk.com > > > >>>>>>>> wrote: > >>>>>>>>>>> I wanted to start a discussion on "things that we cherry-pick" > >>> (to > >>>>>>>> vX_Z > >>>>>>>>>>> branch). > >>>>>>>>>>> > >>>>>>>>>>> I think there are different opinions on what kind of changes > >>> should > >>>>>>> be > >>>>>>>>>>> cherry-picked and it might be a good idea to agree on a common > >>>>>>>> approach. > >>>>>>>>>>> I think (following the comment of Ash here) > >>>>>>>>>>> > >>> https://github.com/apache/airflow/pull/51992#issuecomment-2995632849 > >>>>>>>>> that > >>>>>>>>>>> we can use a very simplistic and (I'd say) dogmatic approach > >>> "only > >>>>>>>>>>> cherry-pick bug fixes. Full stop". But I believe (and past > >>>>>>> experience > >>>>>>>>> from > >>>>>>>>>>> a lot of cherry-picking that I've been doing - multiple times > >>>>>>> helping > >>>>>>>> to > >>>>>>>>>>> bring past branches to be green and spending countless hours on > >>> it, > >>>>>>>>> that it > >>>>>>>>>>> should be a bit more nuanced. > >>>>>>>>>>> > >>>>>>>>>>> I would love to see what others think, but from my experience > >>> those > >>>>>>>> are > >>>>>>>>>>> the things that we **should** cherry-pick: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> 1) bug-fixes (of course) > >>>>>>>>>>> 2) doc changes (when they are improvements or filling gaps) > >>>>>>>>>>> 3) dev tool changes (every time we did not, it resulted in > >>> hours of > >>>>>>> my > >>>>>>>>>>> time when things were breaking and I tried to reconcile it) > >>>>>>>>>>> 4) results of automated refactorings that have very low risks > >>> (in > >>>>>>> the > >>>>>>>>>>> areas that are likely to have cherry-picks) > >>>>>>>>>>> > >>>>>>>>>>> t) - is non-controversial I think > >>>>>>>>>>> > >>>>>>>>>>> 2) - is also relatively non-controversial and very low risk and > >>>>>>> gives > >>>>>>>>> our > >>>>>>>>>>> users a chance to get better docs earlier (even today for > >>> example I > >>>>>>>>> cherry > >>>>>>>>>>> picked this one: https://github.com/apache/airflow/pull/52068 > - > >>>>>>>> because > >>>>>>>>>>> one of my friends who tries to learn Airflow 3 pinged me that > >>>>>>>>>>> "ConfiuguringRuff" link that we have in 3.0.2 leads to 404 NOT > >>>> found > >>>>>>>>>>> 3) - it had always bitten us if we stopped cherry-picking dev > >>> tool > >>>>>>>>>>> changes. The thing is that external dependencies change all the > >>>> time > >>>>>>>>> and we > >>>>>>>>>>> are continuously catching up with those, also we improve, speed > >>> up > >>>>>>> and > >>>>>>>>>>> simplify the tooling - and often things that worked when branch > >>> was > >>>>>>>> cut, > >>>>>>>>>>> does not work today - countless, countless hours lost in one or > >>> two > >>>>>>>>>>> branches when we stopped doing it - I think even once or twice > I > >>>> had > >>>>>>>> to > >>>>>>>>>>> just copy over most (but not all) the code from main to the > >>> branch > >>>>>>> and > >>>>>>>>>>> commit one single "catch-up dev tooling with main" big change > >>>>>>>>>>> > >>>>>>>>>>> 4) Is likely most controversial - example here: > >>>>>>>>>>> https://github.com/apache/airflow/pull/51992/ - those are the > >>> kind > >>>>>>> of > >>>>>>>>>>> (really small) changes that are done in "active" area (i.e. > area > >>>>>>> that > >>>>>>>>> had > >>>>>>>>>>> and will have a lot of cherry-picks anyway, but they are done > >>> with > >>>>>>>>>>> automated refactoring - like renaming variables and such. This > >>>>>>>>> introduces > >>>>>>>>>>> clarity and readability, so this is good we are doing them. But > >>> if > >>>>>>> we > >>>>>>>> do > >>>>>>>>>>> not cherry-pick them and then we cherry-pick any change that > >>>> touches > >>>>>>>> the > >>>>>>>>>>> same code, this lead to a conflict. Conflicts are frustrating, > >>>>>>>>> especially > >>>>>>>>>>> those kinds - you never know what you should do - should you > >>>> "merge" > >>>>>>>>> this > >>>>>>>>>>> naming change with your change? or should you leave the > original > >>>>>>>>> namiing, > >>>>>>>>>>> or should you try to find the past commit that changed it and > >>>>>>>>> cherry-pick > >>>>>>>>>>> it as well? This paired with the fact that we are using > >>>>>>> cherry-picker > >>>>>>>>> that > >>>>>>>>>>> allows to cherry-pick stuff very quickly, automatically and > >>>>>>> painlessly > >>>>>>>>> when > >>>>>>>>>>> there are no conflicts, make me think that yes - we should > >>> cherry > >>>>>>>> -pick > >>>>>>>>>>> those changes proactively as a service to those contributors > who > >>>>>>> will > >>>>>>>>>>> follow up with their cherry-picking. It's just "good service" > >>> and > >>>>>>>>> helping > >>>>>>>>>>> others who will come after you. > >>>>>>>>>>> > >>>>>>>>>>> That's how my definition of "what we should cherry-pick" is... > >>>>>>>>>>> > >>>>>>>>>>> I wonder what others think about it ? > >>>>>>>>>>> > >>>>>>>>>>> J. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>> --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>>>>>>> > >>>>>>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>> > >>>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
