Thanks Holden, that seems to be Arrow though, not Airflow? > On 26 Oct 2026, at 19:42, Holden Karau <[email protected]> wrote: > > So a number of ASF projects have been called on the reduce CI load > (including Airflow and Spark). There's been a number of e-mails about this, > one of the airflow e-mails is > https://lists.apache.org/thread.html/0cs9lo9lqxcdbzpqt1yhzvzx5pn1h7dt?find_parent=true > > On Mon, Oct 26, 2026 at 12:20 PM Ash Berlin-Taylor <[email protected]> wrote: > >> First off: If a proposal is big enough to need an AIP, definitionally, it >> is not appropriate for lazy consensus. AIP’s are for big changes. Big >> changes get actively voted upon. And since it _does_ have an AIP, you >> already acknowledged at some level it is a big change. >> >> AIP-120 starts out by saying "Apache Airflow receives more pull requests >> than its maintainers can review”. That framing is very different to “ASF >> org asked us to reduce our CI usage”. >> >> Where in the AIP document or previous discussion thread was the “we need >> to reduce our CI usage” called out as a goal? Ci time is mentioned, but >> never “we need to reduce this because the ASF org has asked us to”. I too >> would like my CI to jobs to run quicker. Who wouldn’t? >> >> Where have we been called upon to reduce CI load? It was on this list? I >> can’t see it. And if it isn’t on list it essentially didn’t happen. Another >> ASF maxim, of which you are also painfully aware. >> >> I feel you are taking liberties here and making big changes to the project >> almost unilaterally. And I think you should apologise for doing it in a >> tone deaf way without true consensus. >> >> >> >>> On 26 Oct 2026, at 18:29, Jarek Potiuk <[email protected]> wrote: >>> >>> BTW. Airflow has been called upon many times last week by other projects >> as >>> *biggest resource hog" many times, people were angry, people asked Infra >>> to limit Airflow's resource usage. They literally called Infra to shut us >>> down during the release process. >>> >>> The only way I contained that was by telling people we are already >> working >>> on it and we have a clear plan to follow. This required a lot of "being >>> firm". One comment I received was that after 3 weeks of effort, I finally >>> refined the design enough that it raised no objections, allowing me to >>> follow through and help other projects—including helping them reduce >> their >>> usage - which benefits everyone. >>> >>> I added this page >> https://infra.apache.org/github-actions-optimization.html >>> and added a feature in Magpie that allows people to optimize their >>> workflows practically automatically.. >>> >>> All that following all the rules we agreed to. >>> >>> I will leave it at that. >>> >>> J. >>> >>> >>> On Mon, Oct 26, 2026 at 7:22 PM Jarek Potiuk <[email protected]> wrote: >>> >>>> Sorry. We reduced our resource usage by 2/3 >>>> >>>> On Mon, Oct 26, 2026 at 7:20 PM Jarek Potiuk <[email protected]> wrote: >>>> >>>>> Ash, >>>>> >>>>> * The original discussion where I posted my last response from me to >> you >>>>> on the change was Sep 22 and waited for your answer. That was 1.5 >> months >>>>> ago. >>>>> https://lists.apache.org/thread.html/w74655o8y1jzrzcnygntyyovocf7zmzj >> - >>>>> I have not seen you respond to it and I believe I answered your >> concerns >>>>> >>>>> *.I implemented PR limits as the first step (with your consent) of >>>>> improvements and announced my plan for the next steps >>>>> https://lists.apache.org/thread.html/y0vzry5s55k2gy17rc93632zp0d5733o >>>>> >>>>> * I adapted design, respondsed to your concerns and explained what I am >>>>> going to do on this thread - where I started DISCUSSION on the updated >>>>> design - >>>>> https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c >> - >>>>> that was 21 days ago. - it did not receive any objection from you >>>>> >>>>> * the discussion run until - 10 days ago >>>>> https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c >> - >>>>> where no-one raised significant objections, this discussion had links >> to >>>>> design, AIP, Draft PRs and my clear intentions to run LAZY CONSENSUS >>>>> >>>>> * I raised Lazy consensus >>>>> https://lists.apache.org/thread.html/nfno4r1jqx5dj75v867prjjgg6o77odx >> - >>>>> 7 days ago >>>>> >>>>> * I announced result on Friday: - 4 days after it has been waiting >>>>> >>>>> Yes. AIP-120 is approved. I cannot account for someone not paying >>>>> attention for 1.5 months, sorry. You are just one of the PMC members >> here - >>>>> nobody else does not object. You are not more important than others >> and we >>>>> are not going to wait with decisions, that also help the whole ASF deal >>>>> with the difficult situation of CI resources being exhausted by >>>>> AI—especially since we gave ample time to respond to two discussions >> and >>>>> the lazy consensus. >>>>> >>>>> Get over that, Ash. If you would like to change somethign in the way >> how >>>>> AIP-120 is configured feel free to start discussiosn (you already did >> in >>>>> #contributors and made some suggestions - which is good). >>>>> >>>>> It is implemented as is now. We can disable it any time if we do not >> want >>>>> to lose our contributors. We have many configuration options, and we >>>>> already reduced our resource usage by one-third, unblocking not only >> our >>>>> queue but also other projects' queues. >>>>> >>>>> Let's discuss what we want to change and how to configure it, simply. >>>>> Let's work toward the common goal of faster and better CI for Airflow. >>>>> >>>>> J. >>>>> >>>>> >>>>> On Mon, Oct 26, 2026 at 6:44 PM Ash Berlin-Taylor <[email protected]> >> wrote: >>>>> >>>>>> No. >>>>>> >>>>>> Jarek: this behaviour _was not okay_. You knew when creating the lazy >>>>>> consensus that there was not consensus. >>>>>> >>>>>> This was never a valid use case of a Lazy Consensus. This discussion >>>>>> already had strong disagreements. I was the most vocal, but I wasn’t >> the >>>>>> only person to mention disagreement. And yet you really thought lazy >>>>>> consensus was appropriate tool here? How would you feel if I passed a >>>>>> “Remove Breeze” via lazy consensus when I knew you were on holiday? >>>>>> >>>>>> As per ASF’s own guidance, which I know you already know. >>>>>> >>>>>>> Sometimes lazy consensus is not appropriate. In such cases it is >>>>>> necessary to make a proposal to the email list and discuss options >>>>>> >>>>>> And since the discussion _already_ had pushback, lazy consensus was >>>>>> categorically not appropriate. >>>>>> >>>>>> This is not an appropriate response Jarek. Not even remotely. If you >>>>>> think it is you need to re-evaluate your priorities around community >>>>>> building. >>>>>> >>>>>> I repeat. There is no consensus on AIP-120. >>>>>> >>>>>>> On 26 Oct 2026, at 14:59, Jarek Potiuk <[email protected]> wrote: >>>>>>> >>>>>>>> There is no consensus reached here. >>>>>>> >>>>>>> Consensus has been reached. We've been through that before; there >> was a >>>>>>> long discussion followed by arguments from the other side. Your only >>>>>>> opposition was that it is aimed only to save CI time of shared >> runners >>>>>>> (which also it does). >>>>>>> And as the last time we discussed - you can always call to revert >> that >>>>>>> decision. AIP-120 was approved 3 days ago. >>>>>>> >>>>>>>> Calling a lazy consensus on this AIP given how strongly I voiced my >>>>>>> opposition to it is appropriate. >>>>>>> >>>>>>> You can always call a change to remove it. In the meantime, we >> proceed >>>>>> with >>>>>>> testing it. Also, an alternative is that we would likely have to cut >>>>>> down >>>>>>> our usage of shared ASF CI because we are taking nearly 30% of it. >>>>>> Some of >>>>>>> the changes and optimizations Shahar and I implemented over the >> weekend >>>>>>> reduced our usage to one-third (also Arrow and Datafusion also made >>>>>> similar >>>>>>> optimizations with room for more) - which already had an effect on >>>>>>> significant decrease of queue time (which was also goal of AIP-120). >>>>>>> >>>>>>> But yes - feel free to start a new discussion on reverting it. This >>>>>> is the >>>>>>> way how decisions are being made here. >>>>>>> >>>>>>> J >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 26, 2026 at 2:23 PM Ash Berlin-Taylor <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>>> I’m angry and making typos. Is _not appropriate_. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On 26 Oct 2026, at 13:22, Ash Berlin-Taylor <[email protected]> >> wrote: >>>>>>>>> >>>>>>>>> Calling a lazy consensus on this AIP given how strongly I voiced my >>>>>>>> opposition to it is appropriate. >>>>>>>>> >>>>>>>>> This should have never a candidate for lazy consensus, and was >> opened >>>>>>>> and closed when many of us were busy working on the 3.4 release. >>>>>> There is >>>>>>>> no consensus reached here. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 23 Oct 2026, at 21:52, Jarek Potiuk <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hello everyone, >>>>>>>>>> >>>>>>>>>> The lazy consensus on AIP-120 [1] ended on 2026-10-22 12:00 UTC >> with >>>>>>>>>> no objections, so AIP-120 is accepted. >>>>>>>>>> >>>>>>>>>> Next steps, as described in the proposal: merge the design PR and >>>>>> the >>>>>>>>>> stacked implementation PRs (everything ships switched off), run >> the >>>>>>>>>> opt-in pilot, then switch it on for all external contributors in a >>>>>>>>>> separate one-line change. >>>>>>>>>> >>>>>>>>>> Reviews on the stack are very welcome: #73378, #74819, #74820, >>>>>> #74822, >>>>>>>>>> #74823, #74825. >>>>>>>>>> >>>>>>>>>> [1] >>>>>> https://lists.apache.org/thread/nfno4r1jqx5dj75v867prjjgg6o77odx >>>>>>>>>> >>>>>>>>>> J. >>>>>>>>>> >>>>>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
