Mark,
On 5/12/26 12:07 PM, Mark Thomas wrote:
All,
Historically, our approach to security fixes has been:
- take advantage of any opportunity to disguise what is actually being
fixed
- commit the fix with a non-obviously security related commit message
- tag
- vote
- release
The time between the commit and tag depended on:
- the severity of the issue (more severe, closer to the tag)
- how obvious the fix was (more obvious, closer to the tag)
The vote period is usually the standard >= 72 hours but we have reduced
that to a few hours when we have needed to release in a hurry.
AI changes all of this. It is trivial (the ASF security team did this
with the 9.0.118 tag) to get AI to review the diff between tags and
identify the security fixes. It took about 20 minutes and several of the
CVEs I've just published were in the first few results. Work colleagues
have been doing something similar for a while too.
Given this change in circumstances, I think it is worth reconsidering
how we approach security vulnerabilities and releases.
A few random ideas to get the discussion started:
- Where mitigation is available (e.g. via configuration) publish the CVE
at the same time as the fix is committed - i.e. before the release.
This seems like something to consider.
- Build and vote on releases in private then push to the pubic repos and
announce the CVEs as part of the release process.
I don't like private voting at all. Logistically, it's ugly, then we
need to tally votes in private and post them later. They might be forged
or misrepresented (though unlikely). We have to have a private repo,
more digital paperwork, more opportunities to break something during the
rush to push the release out the door, etc.
Finally, if the release drops at the same time we push the private ->
public, then AI has 20 minutes to find the vulnerabilities and the
admins of the world also have 20 minutes to install their updates.
None of that sounds good to me.
- Run some (which?) AI security scans on the Tomcat code base to try get
ahead (unlikely) but at least keep up with anything an attacker could
find.
This sounds like a good thing to do. I suspect that some committers
employers are already setting up tooling along these lines. We could ask
them to throw Tomcat's code into the mix for scanning.
- Reduce the voting period.
I'm not sure this is okay per ASF policy. I seem to remember that some
project (Subversion?) never held release votes precisely because they
didn't want to deal with the official rules on "voting". So they just
pushed "versions" without any voting.
- Something else?
Is "no changes" an option?
Or is your position that if AI can figure out the changes in 20 minutes,
there is no point in putting up any kind of smoke-screen; we may as well
just commit security fixes directly and without any attempt to hide them?
I'm curious about the report of ASF Security doing this with the 9.0.118
tag and coming up with the same CVEs that we had already fixed. That
sounds almost too convenient, unless you mean it looked at the diffs
from 9.0.117 and said "it looks like they fixed XXX YYY and ZZZ in spite
of the weird commit comments".
I remember someone from the httpd team presenting at an ASF conference
one year saying "we are lying to our users when we silently commit
security fixes and then publish later, and it doesn't make us feel
good". Does anyone know if httpd ever changed their methodology?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
