Nice, good discussion! Thanks & Regards, Amogh Desai
On Wed, Jun 25, 2025 at 12:57 AM Jarek Potiuk <ja...@potiuk.com> wrote: > PR here to add backport label automatically for dev tools changes (and to > maintain it with minor version upgrades) > https://github.com/apache/airflow/pull/52189 > > On Tue, Jun 24, 2025 at 9:14 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Might be a good idea, yes. > > > > On Tue, Jun 24, 2025 at 9:09 PM Jens Scheffler > <j_scheff...@gmx.de.invalid> > > wrote: > > > >> Hi Jarek, > >> > >> one thought came into my mind... as you said it is a lot of trouble to > >> find which "dev" PR might be missing on a mainteance branch... should we > >> cherry-pick dev tooling PRs by default as well to active maintenance > >> branches? I mean... automating this in the CI like if `backport to`label > >> is set? > >> > >> Jens > >> > >> On 24.06.25 11:38, Jarek Potiuk wrote: > >> > https://github.com/apache/airflow/pull/52148 -> Proposed PR. Let's > >> iterate > >> > there if there are comments. > >> > > >> > On Tue, Jun 24, 2025 at 11:25 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > >> > > >> >> Yeah. I think what I am really after is expressed by Wei: +0.5 vote > - > >> >> i.e. we should be fine with **some** selected cherry-picks that are > not > >> >> bug-fixes that we proactively cherry-pick and do not "complain" that > >> this > >> >> is wrong because technically it's not a bugfix. I think It would be > >> great > >> >> if in every such cherry-pick we rationalise why we are doing it: > >> >> > >> >> "I know it's not a bugfix, but because this is an actively developed > >> part > >> >> that is cherry-picked heavily, let me cherry-pick that one to make it > >> >> easier for others to cherry-pick bugfixes". > >> >> > >> >> If we are all on-board with that (seems so) then I can update the > >> >> cherry-pick description to add it as possible exception. > >> >> > >> >> J. > >> >> > >> >> On Tue, Jun 24, 2025 at 10:14 AM Kaxil Naik <kaxiln...@gmail.com> > >> wrote: > >> >> > >> >>> It looks like everyone is mostly on the same page based on all the > >> emails > >> >>> - > >> >>> so no comments :) . > >> >>> > >> >>> On Tue, 24 Jun 2025 at 07:45, Wei Lee <weilee...@gmail.com> wrote: > >> >>> > >> >>>> I’m +1 for 1-3 (assuming the doc changes relate to the backported > >> >>> version). > >> >>>> +0.5 for 4. I hope that changes not related to new features will be > >> >>>> backported when feasible; however, we can skip them if the required > >> >>> effort > >> >>>> is substantial. This is because failing to backport these items > could > >> >>>> potentially lead to future conflicts with points 1 or 3. > >> >>>> > >> >>>> Best, > >> >>>> Wei > >> >>>> > >> >>>>> On Jun 24, 2025, at 5:32 AM, Jens Scheffler > >> >>> <j_scheff...@gmx.de.INVALID> > >> >>>> wrote: > >> >>>>> I am +1 for (1) to (3) [also assuming that (2) is mostly like a > doc > >> >>>> bugfix!] > >> >>>>> For (4) I am hesitant and would rather be conservative. Every > >> >>>> cherry-pick has a risk to break something in old codebase. As > >> branches > >> >>>> change over time and backport PRs are clearly less cautious > reviewed > >> it > >> >>>> might lead to introduced inconsistencies which might not be covered > >> in > >> >>>> tests. I would also not back-port just for the sake of easier later > >> >>>> maintenance for other cherry-picks. But for (4) the rule might not > be > >> >>> too > >> >>>> strict and every rule is made for exceptions which in (4) might be > >> >>> likely. > >> >>>> But I would not advertise to backport refactorings if no clear > >> benefit. > >> >>>>> On 23.06.25 15:33, Pavankumar Gopidesu wrote: > >> >>>>>> Thanks Jarek, for starting this discussion, > >> >>>>>> > >> >>>>>> I agree with all the points. > >> >>>>>> > >> >>>>>> The real intention behind to backport > >> >>>>>> https://github.com/apache/airflow/pull/51992 is , this area has > a > >> >>> lot > >> >>>> of > >> >>>>>> ongoing development going and I felt it's worth porting to > >> v3-0-test. > >> >>>>>> I myself faced situations where I tried to backport some doc > >> changes > >> >>> , I > >> >>>>>> failed because of conflicts that previous changes were not ported > >> >>> back > >> >>>> on > >> >>>>>> the same file etc or conflicts with some lines. > >> >>>>>> So I am very strong on this if there are any areas with heavy > >> >>>> development > >> >>>>>> activity going. I feel it's worth backporting the changes and > IMHO > >> i > >> >>>>>> don't see any problem. > >> >>>>>> > >> >>>>>> Pavan > >> >>>>>> > >> >>>>>> > >> >>>>>> On Mon, Jun 23, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> > >> >>> wrote: > >> >>>>>>>> I think We are approaching it from the same point of view, just > >> >>> that > >> >>>> we > >> >>>>>>> have different conclusions. > >> >>>>>>> > >> >>>>>>> I agree. I think that simply there is one special case that we > >> >>> should > >> >>>>>>> "allow". Details below. > >> >>>>>>> > >> >>>>>>>> For 4, I do not have strong opinions on either front, but > >> defining > >> >>>> what > >> >>>>>>> to > >> >>>>>>>> do and what not should > >> >>>>>>>> probably be that if that change makes it easy to backport > >> something > >> >>>> else, > >> >>>>>>>> maybe have it? > >> >>>>>>> Yep. That's **exactly** my proposal. I do not want to backport > >> >>> **all** > >> >>>>>>> refactorings. That would be stupid. In my definition of 4. I > want > >> to > >> >>>>>>> cherry-pick those pre-commit when it's small, and automated and > >> when > >> >>>> we can > >> >>>>>>> easily anticipate it will make it quite likely someone (soon) > will > >> >>> do > >> >>>>>>> another cherry-pick that will be conflicting. > >> >>>>>>> > >> >>>>>>> Yes. It's not "0/1" and quite a bit personal judgment, and yes > it > >> >>>> **might > >> >>>>>>> not** result in conflict (so we might end up with YAGNI), but I > >> >>> think > >> >>>> we > >> >>>>>>> should define (and trust committers judgment) when they do it, > >> >>> because > >> >>>> they > >> >>>>>>> want to prevent "others" to have problems. > >> >>>>>>> > >> >>>>>>> For example in case of > >> https://github.com/apache/airflow/pull/51992 > >> >>>> -> the > >> >>>>>>> only reason why I think it makes sense to cherry-pick it is > >> because > >> >>> it > >> >>>> is > >> >>>>>>> in the area of task-sdk that has been recently bug-fixed quite a > >> >>>> number of > >> >>>>>>> times and there are open related issues to this area that make > it > >> >>>> (IMHO) > >> >>>>>>> quite likely to result in conflict. I think there are other > areas > >> >>> like > >> >>>> that > >> >>>>>>> (UI for example) where we clearly cherry-pick a number of > changes, > >> >>>> because > >> >>>>>>> there are some pretty active "bug-fixing" in this area as 3.0 > had > >> >>>> still a > >> >>>>>>> number of low priority but either known or anticipated bugs that > >> are > >> >>>> likely > >> >>>>>>> to be fixed in 3.0.3, 3.0.4 > >> >>>>>>> > >> >>>>>>> So just to clarify - I do not want to cherry-pick "all" > >> refactoring, > >> >>>> but > >> >>>>>>> leave to the judgment of the commiter merging the request (and > >> >>> author) > >> >>>> to > >> >>>>>>> decide to cherry-pick such change anticipating it will make life > >> >>>> easier for > >> >>>>>>> others. > >> >>>>>>> > >> >>>>>>> J > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Mon, Jun 23, 2025 at 2:57 PM Amogh Desai < > >> >>> amoghdesai....@gmail.com> > >> >>>>>>> wrote: > >> >>>>>>> > >> >>>>>>>> Agree with point 1 - 3 definitely. > >> >>>>>>>> > >> >>>>>>>> For 4, I do not have strong opinions on either front, but > >> defining > >> >>>> what > >> >>>>>>> to > >> >>>>>>>> do and what not should > >> >>>>>>>> probably be that if that change makes it easy to backport > >> something > >> >>>> else, > >> >>>>>>>> maybe have it? > >> >>>>>>>> > >> >>>>>>>> For ex: > >> >>>>>>>> *PR 1 changes file1.py and file2.py* > >> >>>>>>>> *PR 2 changes some lines in file2.py* > >> >>>>>>>> > >> >>>>>>>> *We backport PR 1 as it's a bug fix and do not for PR 2 as its > >> some > >> >>>>>>>> refactoring.* > >> >>>>>>>> > >> >>>>>>>> *Now while trying to backport PR 3 (bugfix), it conflicts and > >> needs > >> >>>> PR 2 > >> >>>>>>> to > >> >>>>>>>> be picked* > >> >>>>>>>> *to land PR 3.* > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> Thanks & Regards, > >> >>>>>>>> Amogh Desai > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Mon, Jun 23, 2025 at 5:02 PM Ash Berlin-Taylor < > >> a...@apache.org> > >> >>>>>>> wrote: > >> >>>>>>>>> I think We are approaching it from the same point of view, > just > >> >>> that > >> >>>> we > >> >>>>>>>>> have different conclusions. > >> >>>>>>>>> > >> >>>>>>>>> Points 1-3 I agree with. > >> >>>>>>>>> > >> >>>>>>>>> We do already have this written up > >> >>>>>>>>> > >> >>> > >> > https://github.com/apache/airflow/blob/130e9600443e06c08acc1b28c69a62c858d6e6a2/dev/README_RELEASE_AIRFLOW.md?plain=1#L116-L129 > >> >>>>>>>>> On 4 I think of it in terms that “every change carries some > >> risk” > >> >>> — > >> >>>> in > >> >>>>>>>> the > >> >>>>>>>>> linked example of 51992 the risk is almost zero, but > generally: > >> >>> 3.0.x > >> >>>>>>> are > >> >>>>>>>>> meant to be bug fix releases on top of 3.0.0, and if it’s not > >> >>> fixing > >> >>>> a > >> >>>>>>>> bug > >> >>>>>>>>> we don’t back port it. The one exception I have to this is if > >> the > >> >>>>>>> change > >> >>>>>>>> is > >> >>>>>>>>> needed to make it easy to backport a change that is a bug > >> without > >> >>>>>>>> conflicts. > >> >>>>>>>>> I think our default approach has to be we don’t back port a > >> change > >> >>>>>>> unless > >> >>>>>>>>> it is fixing a bug, otherwise the risk of “oh I’ll just fix > >> this” > >> >>>> ends > >> >>>>>>> up > >> >>>>>>>>> introducing more bugs than we fix. Stability of a Minor > release > >> >>>> series > >> >>>>>>> is > >> >>>>>>>>> my primary desire, and not changing things more than we have > to > >> is > >> >>>> the > >> >>>>>>>> best > >> >>>>>>>>> way I know of doing that. > >> >>>>>>>>> > >> >>>>>>>>> Things are slightly different now that we have automated > >> >>> cherry-picks > >> >>>>>>> but > >> >>>>>>>>> I still don’t think it is worth porting refactoring > >> automatically. > >> >>>> It’s > >> >>>>>>>>> extra change and risk for almost zero benefit to users is my > >> view. > >> >>>>>>>>> > >> >>>>>>>>> -ash > >> >>>>>>>>> > >> >>>>>>>>>> On 23 Jun 2025, at 11:43, Jarek Potiuk <ja...@potiuk.com> > >> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> BTW. I'd be happy to capture result of this discussion if we > >> can > >> >>> get > >> >>>>>>>> to a > >> >>>>>>>>>> consensus or vote eventually in the "cherry-picking" > >> guidelines. > >> >>>>>>>>>> > >> >>>>>>>>>> On Mon, Jun 23, 2025 at 12:42 PM Jarek Potiuk < > >> ja...@potiuk.com> > >> >>>>>>>> wrote: > >> >>>>>>>>>>> I wanted to start a discussion on "things that we > cherry-pick" > >> >>> (to > >> >>>>>>>> vX_Z > >> >>>>>>>>>>> branch). > >> >>>>>>>>>>> > >> >>>>>>>>>>> I think there are different opinions on what kind of changes > >> >>> should > >> >>>>>>> be > >> >>>>>>>>>>> cherry-picked and it might be a good idea to agree on a > common > >> >>>>>>>> approach. > >> >>>>>>>>>>> I think (following the comment of Ash here) > >> >>>>>>>>>>> > >> >>> > https://github.com/apache/airflow/pull/51992#issuecomment-2995632849 > >> >>>>>>>>> that > >> >>>>>>>>>>> we can use a very simplistic and (I'd say) dogmatic approach > >> >>> "only > >> >>>>>>>>>>> cherry-pick bug fixes. Full stop". But I believe (and past > >> >>>>>>> experience > >> >>>>>>>>> from > >> >>>>>>>>>>> a lot of cherry-picking that I've been doing - multiple > times > >> >>>>>>> helping > >> >>>>>>>> to > >> >>>>>>>>>>> bring past branches to be green and spending countless hours > >> on > >> >>> it, > >> >>>>>>>>> that it > >> >>>>>>>>>>> should be a bit more nuanced. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I would love to see what others think, but from my > experience > >> >>> those > >> >>>>>>>> are > >> >>>>>>>>>>> the things that we **should** cherry-pick: > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> 1) bug-fixes (of course) > >> >>>>>>>>>>> 2) doc changes (when they are improvements or filling gaps) > >> >>>>>>>>>>> 3) dev tool changes (every time we did not, it resulted in > >> >>> hours of > >> >>>>>>> my > >> >>>>>>>>>>> time when things were breaking and I tried to reconcile it) > >> >>>>>>>>>>> 4) results of automated refactorings that have very low > risks > >> >>> (in > >> >>>>>>> the > >> >>>>>>>>>>> areas that are likely to have cherry-picks) > >> >>>>>>>>>>> > >> >>>>>>>>>>> t) - is non-controversial I think > >> >>>>>>>>>>> > >> >>>>>>>>>>> 2) - is also relatively non-controversial and very low risk > >> and > >> >>>>>>> gives > >> >>>>>>>>> our > >> >>>>>>>>>>> users a chance to get better docs earlier (even today for > >> >>> example I > >> >>>>>>>>> cherry > >> >>>>>>>>>>> picked this one: > https://github.com/apache/airflow/pull/52068 > >> - > >> >>>>>>>> because > >> >>>>>>>>>>> one of my friends who tries to learn Airflow 3 pinged me > that > >> >>>>>>>>>>> "ConfiuguringRuff" link that we have in 3.0.2 leads to 404 > NOT > >> >>>> found > >> >>>>>>>>>>> 3) - it had always bitten us if we stopped cherry-picking > dev > >> >>> tool > >> >>>>>>>>>>> changes. The thing is that external dependencies change all > >> the > >> >>>> time > >> >>>>>>>>> and we > >> >>>>>>>>>>> are continuously catching up with those, also we improve, > >> speed > >> >>> up > >> >>>>>>> and > >> >>>>>>>>>>> simplify the tooling - and often things that worked when > >> branch > >> >>> was > >> >>>>>>>> cut, > >> >>>>>>>>>>> does not work today - countless, countless hours lost in one > >> or > >> >>> two > >> >>>>>>>>>>> branches when we stopped doing it - I think even once or > >> twice I > >> >>>> had > >> >>>>>>>> to > >> >>>>>>>>>>> just copy over most (but not all) the code from main to the > >> >>> branch > >> >>>>>>> and > >> >>>>>>>>>>> commit one single "catch-up dev tooling with main" big > change > >> >>>>>>>>>>> > >> >>>>>>>>>>> 4) Is likely most controversial - example here: > >> >>>>>>>>>>> https://github.com/apache/airflow/pull/51992/ - those are > the > >> >>> kind > >> >>>>>>> of > >> >>>>>>>>>>> (really small) changes that are done in "active" area (i.e. > >> area > >> >>>>>>> that > >> >>>>>>>>> had > >> >>>>>>>>>>> and will have a lot of cherry-picks anyway, but they are > done > >> >>> with > >> >>>>>>>>>>> automated refactoring - like renaming variables and such. > This > >> >>>>>>>>> introduces > >> >>>>>>>>>>> clarity and readability, so this is good we are doing them. > >> But > >> >>> if > >> >>>>>>> we > >> >>>>>>>> do > >> >>>>>>>>>>> not cherry-pick them and then we cherry-pick any change that > >> >>>> touches > >> >>>>>>>> the > >> >>>>>>>>>>> same code, this lead to a conflict. Conflicts are > frustrating, > >> >>>>>>>>> especially > >> >>>>>>>>>>> those kinds - you never know what you should do - should you > >> >>>> "merge" > >> >>>>>>>>> this > >> >>>>>>>>>>> naming change with your change? or should you leave the > >> original > >> >>>>>>>>> namiing, > >> >>>>>>>>>>> or should you try to find the past commit that changed it > and > >> >>>>>>>>> cherry-pick > >> >>>>>>>>>>> it as well? This paired with the fact that we are using > >> >>>>>>> cherry-picker > >> >>>>>>>>> that > >> >>>>>>>>>>> allows to cherry-pick stuff very quickly, automatically and > >> >>>>>>> painlessly > >> >>>>>>>>> when > >> >>>>>>>>>>> there are no conflicts, make me think that yes - we should > >> >>> cherry > >> >>>>>>>> -pick > >> >>>>>>>>>>> those changes proactively as a service to those contributors > >> who > >> >>>>>>> will > >> >>>>>>>>>>> follow up with their cherry-picking. It's just "good > service" > >> >>> and > >> >>>>>>>>> helping > >> >>>>>>>>>>> others who will come after you. > >> >>>>>>>>>>> > >> >>>>>>>>>>> That's how my definition of "what we should cherry-pick" > is... > >> >>>>>>>>>>> > >> >>>>>>>>>>> I wonder what others think about it ? > >> >>>>>>>>>>> > >> >>>>>>>>>>> J. > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>> > --------------------------------------------------------------------- > >> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> >>>>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>> > >> --------------------------------------------------------------------- > >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >> >>>>> > >> >>>> > >> >>>> > --------------------------------------------------------------------- > >> >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >> >>>> > >> >>>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> >
