On Tue, Nov 3, 2026 at 11:37 AM Mark Thomas via dev <[email protected]> wrote: > > On 02/11/2026 13:57, Rémy Maucherat via dev wrote: > > On Mon, Nov 2, 2026 at 2:27 PM Dimitris Soumis <[email protected]> wrote: > >> > >> On Mon, Nov 2, 2026 at 2:24 PM Rémy Maucherat via dev > >> <[email protected]> wrote: > >>> > >>> On Mon, Nov 2, 2026 at 12:41 PM Mark Thomas via dev > >>> <[email protected]> wrote: > >>>> > >>>> All, > >>>> > >>>> The start of the month is here so we should be thinking about tagging > >>>> the November releases. > >>>> > >>>> The various AI reviews have resulted in significantly higher numbers of > >>>> commits and PRs. There are also a handful of open bugs. > >>>> > >>>> Normally, I'd aim to review all of the commits, resolve the new PRs and > >>>> the open bugs before release. > >>>> > >>>> Currently, I am 2-3 weeks behind and I am unsure what to do. > >>>> > >>>> Any BZ issue or PR that relates to an enhancement can wait until the > >>>> next release but that still needs those PRs and bugs to be triaged. The > >>>> commit reviews are harder. I was hoping that I'd be able to work through > >>>> them fairly quickly but that isn't the case. I am finding additional > >>>> fixes that are required or regressions quite frequently. Many of these > >>>> could be classes as no worse than the original code but there are > >>>> occasionally more serious regressions. > >>> > >>> Yes, the extra fixes are very useful. It would be better if they were > >>> not needed of course (working on that ...). > > My reviews have speed up considerably this morning. There is a clear > correlation between code complexity and speed of review / further fixes > being required. That suggests to me that we need to be using more > powerful models for the more complex code. Maybe look at using the free > AI credits for those models that have been offered to ASF committers?
Obviously if the issue is simple, it will work out better ! (also applies to me if I try to fix an issue !) To be honest, I think the models I have on hand are more than good enough and are able to do complex tasks. However, there are some issues and I need to use them better in some cases. The main limitation I have to work around when I'm using a "small" "fast" model is that the context is limited to 256K, and for some models the quality degrades a lot after compaction. Also my review step was very suboptimal. And of course my manual review step after the changes is far from perfect (= it's the worst). Although the HTTP/3 experiment was written with the small models, the review steps were done by "big" models (with 1M context; technically only 512K was ever reached). The main reason is that the components are large and would lead to non stop compaction and useless results on small models. I expect the small open models will show another capability leap simply by increasing the context. I know I can use some stuff like rope with the current models, but it seems accuracy goes down. > >> To me, it seems clear that the whole development process has changed. > >> Therefore, I suggest changing the review process to introduce a risk-based > >> approach. For committed reviews, maybe focus review on potential > >> high-impact areas and limit nonessential changes ~1 week before tagging. > > I agree the development process is changing. We can't not review code > and I've seen enough obscure bugs in what I thought was non-critical > code lead to a vulnerability that I'd be very cautious of reducing the > level of review to any code. > > My current thinking is using more powerful models will enable the > initial fix to be cleaner. That addresses the time problem by reducing > the time needed for the subsequent review. Ok. > >> If we wanted to take it one step further, it might be worth considering > >> altering the release cycle from monthly to something more frequent so > >> reported regressions can be fixed much faster. To reduce the overhead of > >> this, we could also explore utilizing the Apache Trusted Releases project. > > Increasing the release rate to address an increased volume of > regressions seems like the wrong approach to me. I'd rather focus on > reducing the regressions (see above). > > I'm also not sure our user base could keep up. I'm fairly sure only a > very small percentage of our users are keeping up with the latest > releases as it is. I agree that the way things are going they do need to > do a better job of keeping up. > > >> AI is accelerating development, and I think our processes should evolve to > >> accommodate that. > > It is still relatively early days with AI. At the moment, I think we are > applying AI inconsistently across the development process so it is > speeding up some areas and creating bottlenecks in others. I think we > need to apply a little more AI upfront - particularly to the changes to > more complex areas - to reduce the time spent on reviews which is where > the bottleneck is currently. > > > LLMs seem to be making things different overall (for the worse, of > > course, given the volume of very very hard to find CVEs that would > > have never been found without it). > > Code reviews produce (lots of) defects (PRs are now all produced by > > LLMs and are basically the same thing). These defects are really non > > critical: we know Tomcat is stable and worked fine 6 months ago. So > > some edge cases nobody ever cared or ran into are getting fixed, > > because once we know about them, we have to fix them. However, > > individually, the value for each seems infinitesimal. Only if > > everything gets fixed, we could assume a marginal reliability > > improvement (not to be underestimated either since the last bit of > > reliability requires a maximum amount of effort to achieve). > > Agreed. I do wonder if some the claimed edge cases are even possible at > all but in most cases it takes less time to fix the edge case that it > does to reason it doesn't apply. > > > We also have new features that are possible. Nobody would have ever > > cared about doing a Manager 2.0 webapp, well, LLMs do a decent job at > > that. This seems useful to users (webapp is "better" than the existing > > one), but the benefit is also slim. > > We can also do features that nobody really wants (QUIC). Similar low > > benefit. > > I would argue that features like TLS support for clustering do provide > real benefit and there is no way I could have implemented that in the > time I did without AI support. Oh yes, I forgot about PSK. My bad :) > I did briefly start to look at using AI to do refactoring of some of the > more complex code to simplify it but ran out of time before I made any > real progress. I hope that this is an area that we will be able to come > back to. This is clearly an area where you need a consistent stable model, not necessarily the best, but with a very large context. Trying to investigate on (or review) a large chunk of code requires it. Rémy > > Overall: not great, but I feel like I have to do it. > > Ack. I feel the same way. > > > Rémy > > > >>>> > >>>> Do we tag anyway and address regressions as they are reported? > >> > >> > >> +1 > > I'm not a huge fan of this approach but I recognise we might need to do > this. > > >>>> Is there a better way to triage a large number of commits for closer > >>>> review? > >>>> > >>>> Does delaying the tag gain us anything? > >>> > >>> The [VOTE] was posted on the 6th last month, so there's time. > > With the speed I am currently working through the reviews, I am a lot > more hopeful. I am ignoring the jdbc-pool changes for now on the grounds > that DBCP is the default and users that find a regression in jdbc-pool > can always use DBCP (or a jdbc-pool JAR from a previous Tomcat version). > > >>> There are 5 BZs to look at right now, so it's not that bad. To be > >>> honest, even AI centric projects have given up long ago. > >>> > >>> Note: the BZ "Long Format" button allows saving the html and > >>> triaging/reproducing issues fast (usually at least). I used it to > >>> triage the jdbc-pool BZs. > > Good to know. > > Mark > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
