On Tue, Nov 3, 2026 at 12:46 PM Mark Thomas via dev <[email protected]> wrote:
> On 02/11/2026 13:57, Rémy Maucherat via dev wrote: > > On Mon, Nov 2, 2026 at 2:27 PM Dimitris Soumis <[email protected]> > wrote: > >> > >> On Mon, Nov 2, 2026 at 2:24 PM Rémy Maucherat via dev < > [email protected]> wrote: > >>> > >>> On Mon, Nov 2, 2026 at 12:41 PM Mark Thomas via dev > >>> <[email protected]> wrote: > >>>> > >>>> All, > >>>> > >>>> The start of the month is here so we should be thinking about tagging > >>>> the November releases. > >>>> > >>>> The various AI reviews have resulted in significantly higher numbers > of > >>>> commits and PRs. There are also a handful of open bugs. > >>>> > >>>> Normally, I'd aim to review all of the commits, resolve the new PRs > and > >>>> the open bugs before release. > >>>> > >>>> Currently, I am 2-3 weeks behind and I am unsure what to do. > >>>> > >>>> Any BZ issue or PR that relates to an enhancement can wait until the > >>>> next release but that still needs those PRs and bugs to be triaged. > The > >>>> commit reviews are harder. I was hoping that I'd be able to work > through > >>>> them fairly quickly but that isn't the case. I am finding additional > >>>> fixes that are required or regressions quite frequently. Many of these > >>>> could be classes as no worse than the original code but there are > >>>> occasionally more serious regressions. > >>> > >>> Yes, the extra fixes are very useful. It would be better if they were > >>> not needed of course (working on that ...). > > My reviews have speed up considerably this morning. There is a clear > correlation between code complexity and speed of review / further fixes > being required. That suggests to me that we need to be using more > powerful models for the more complex code. Maybe look at using the free > AI credits for those models that have been offered to ASF committers? > We could distribute the review work across committers by establishing a consistent, well-defined AI-assisted review process. For example, a reviewer should always use a powerful review-capable model and adhere to a specific review skill developed for the agent to follow when submitting a review for a range of commits. We could also leverage https://llm.apache.org/models individually or in a centralized automated pipeline. There are also proprietary programs, such as https://claude.com/contact-sales/claude-for-oss. This isn't a long-term solution, but it can be explored individually. > >> To me, it seems clear that the whole development process has changed. > Therefore, I suggest changing the review process to introduce a risk-based > approach. For committed reviews, maybe focus review on potential > high-impact areas and limit nonessential changes ~1 week before tagging. > > I agree the development process is changing. We can't not review code > and I've seen enough obscure bugs in what I thought was non-critical > code lead to a vulnerability that I'd be very cautious of reducing the > level of review to any code. > > My current thinking is using more powerful models will enable the > initial fix to be cleaner. That addresses the time problem by reducing > the time needed for the subsequent review. > > >> If we wanted to take it one step further, it might be worth considering > altering the release cycle from monthly to something more frequent so > reported regressions can be fixed much faster. To reduce the overhead of > this, we could also explore utilizing the Apache Trusted Releases project. > > Increasing the release rate to address an increased volume of > regressions seems like the wrong approach to me. I'd rather focus on > reducing the regressions (see above). > > I'm also not sure our user base could keep up. I'm fairly sure only a > very small percentage of our users are keeping up with the latest > releases as it is. I agree that the way things are going they do need to > do a better job of keeping up. > > >> AI is accelerating development, and I think our processes should evolve > to accommodate that. > > It is still relatively early days with AI. At the moment, I think we are > applying AI inconsistently across the development process so it is > speeding up some areas and creating bottlenecks in others. I think we > need to apply a little more AI upfront - particularly to the changes to > more complex areas - to reduce the time spent on reviews which is where > the bottleneck is currently. > > > LLMs seem to be making things different overall (for the worse, of > > course, given the volume of very very hard to find CVEs that would > > have never been found without it). > > Code reviews produce (lots of) defects (PRs are now all produced by > > LLMs and are basically the same thing). These defects are really non > > critical: we know Tomcat is stable and worked fine 6 months ago. So > > some edge cases nobody ever cared or ran into are getting fixed, > > because once we know about them, we have to fix them. However, > > individually, the value for each seems infinitesimal. Only if > > everything gets fixed, we could assume a marginal reliability > > improvement (not to be underestimated either since the last bit of > > reliability requires a maximum amount of effort to achieve). > > Agreed. I do wonder if some the claimed edge cases are even possible at > all but in most cases it takes less time to fix the edge case that it > does to reason it doesn't apply. > > > We also have new features that are possible. Nobody would have ever > > cared about doing a Manager 2.0 webapp, well, LLMs do a decent job at > > that. This seems useful to users (webapp is "better" than the existing > > one), but the benefit is also slim. > > We can also do features that nobody really wants (QUIC). Similar low > benefit. > > I would argue that features like TLS support for clustering do provide > real benefit and there is no way I could have implemented that in the > time I did without AI support. > > I did briefly start to look at using AI to do refactoring of some of the > more complex code to simplify it but ran out of time before I made any > real progress. I hope that this is an area that we will be able to come > back to. > > > Overall: not great, but I feel like I have to do it. > > Ack. I feel the same way. > > > Rémy > > > >>>> > >>>> Do we tag anyway and address regressions as they are reported? > >> > >> > >> +1 > > I'm not a huge fan of this approach but I recognise we might need to do > this. > > >>>> Is there a better way to triage a large number of commits for closer > review? > >>>> > >>>> Does delaying the tag gain us anything? > >>> > >>> The [VOTE] was posted on the 6th last month, so there's time. > > With the speed I am currently working through the reviews, I am a lot > more hopeful. I am ignoring the jdbc-pool changes for now on the grounds > that DBCP is the default and users that find a regression in jdbc-pool > can always use DBCP (or a jdbc-pool JAR from a previous Tomcat version). > > >>> There are 5 BZs to look at right now, so it's not that bad. To be > >>> honest, even AI centric projects have given up long ago. > >>> > >>> Note: the BZ "Long Format" button allows saving the html and > >>> triaging/reproducing issues fast (usually at least). I used it to > >>> triage the jdbc-pool BZs. > > Good to know. > > Mark > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > Dimitris
