I'd like to suggest some improvements for identifying and announcing
broken branches and to avoid pushing commits to broken branches.
For CI we should have two gates.
The first is pre-commit testing, which we have already discussed, e.g.
either ci-cassandra or circleci can be used (and repeated tests are
expected to be run on circleci).
The second gate is whether the branch the commit is being merged to is
in a healthy state. Our canonical CI system is ci-cassandra, and for
post-commit health it is our only CI. Last week ci-cassandra was
broken on all branches from 4.0 up. The cause was two reasons,
neither our fault: debian packaging (CASSANDRA-18910) and xerces2 xml
file processing OOM (cassandra-builds:8d11eea).
Knowing if a branch is broken (before pushing) is just to check
ci-cassandra.apache.org
Folk have suggested this is not enough, and that a message to the dev@
would also help, but part of the problem is that there's no one place
that people check before pushing (there's no requirement on anyone to
be keeping up to date with dev@ at all times).
To summarise, I feel we currently don't have good practices for
- identifying and announcing a broken CI,
- knowing who is investigating it,
- labelling it with the cause,
- knowing who is working on a fix
The suggested actions I'm proposing for us all to adopt are:
1. Before committing please check dev@ and ci-cassandra.a.o
2. If you see ci-cassandra is red on a branch, and no dev@ thread has
been started, please start the dev@ thread and create the ticket,
3. Put the ticket id into the description of the first red build
("Add description")
4. By default, hold off on pushing to broken branches.
WRT (2), we should just be able to send the automated build failures
to dev@ instead of builds@, but failed builds are often not sending
such notifications, so this isn't something we can rely on yet.
Reference slack threads:
- https://the-asf.slack.com/archives/CK23JSY2K/p1696693542832489
- https://the-asf.slack.com/archives/CK23JSY2K/p1696599019480519
- https://the-asf.slack.com/archives/CK23JSY2K/p1696351208371029
- https://the-asf.slack.com/archives/CK23JSY2K/p1695878499669699
