What about having a nag-bot that notifies #cassandra-dev on ASF slack hourly if
the builds are broken?
On Tue, Oct 10, 2023, at 8:02 AM, Mick Semb Wever wrote:
> I'd like to suggest some improvements for identifying and announcing
> broken branches and to avoid pushing commits to broken branches.
>
> For CI we should have two gates.
>
> The first is pre-commit testing, which we have already discussed, e.g.
> either ci-cassandra or circleci can be used (and repeated tests are
> expected to be run on circleci).
>
> The second gate is whether the branch the commit is being merged to is
> in a healthy state. Our canonical CI system is ci-cassandra, and for
> post-commit health it is our only CI. Last week ci-cassandra was
> broken on all branches from 4.0 up. The cause was two reasons,
> neither our fault: debian packaging (CASSANDRA-18910) and xerces2 xml
> file processing OOM (cassandra-builds:8d11eea).
>
> Knowing if a branch is broken (before pushing) is just to check
> ci-cassandra.apache.org
>
> Folk have suggested this is not enough, and that a message to the dev@
> would also help, but part of the problem is that there's no one place
> that people check before pushing (there's no requirement on anyone to
> be keeping up to date with dev@ at all times).
>
> To summarise, I feel we currently don't have good practices for
> - identifying and announcing a broken CI,
> - knowing who is investigating it,
> - labelling it with the cause,
> - knowing who is working on a fix
>
>
> The suggested actions I'm proposing for us all to adopt are:
>
> 1. Before committing please check dev@ and ci-cassandra.a.o
> 2. If you see ci-cassandra is red on a branch, and no dev@ thread has
> been started, please start the dev@ thread and create the ticket,
> 3. Put the ticket id into the description of the first red build
> ("Add description")
> 4. By default, hold off on pushing to broken branches.
>
>
> WRT (2), we should just be able to send the automated build failures
> to dev@ instead of builds@, but failed builds are often not sending
> such notifications, so this isn't something we can rely on yet.
>
>
> Reference slack threads:
> - https://the-asf.slack.com/archives/CK23JSY2K/p1696693542832489
> - https://the-asf.slack.com/archives/CK23JSY2K/p1696599019480519
> - https://the-asf.slack.com/archives/CK23JSY2K/p1696351208371029
> - https://the-asf.slack.com/archives/CK23JSY2K/p1695878499669699
>
