Re: [ Revised Proposal ] Continuous Builds in GNOME

[Tristan Van Berkom]

On Sat, 2016-06-18 at 08:15 +0100, Emmanuele Bassi wrote:
[...]
> Tristan: I understand you don't give a crap about GNOME as a holistic
> project; you've made it *painfully* clear over the years. I'd like to
> point out that without GNOME working as a single unit nothing we do,
> or did up until now, really matters in the grand scheme of things.
> The
> *only* successful thing out of GNOME is the GNOME desktop environment
> and its whole stack of technologies; everything else has been a
> failure, mostly caused by this inane obsession of considering the
> GNOME platform as a bag of loosely coupled projects that somebody has
> to work hard into identifying and keep working together, granting the
> position of gatekeeper to a bunch of middle men. This position is
> ignorant of the history of the project, especially of its failures;
> it's insulting of the work of hundreds of people who want to keep
> this
> thing working and keep it approachable to newcomers; and completely
> invalidated by the reality of what happens every day in the
> commercial
> space. I also don't understand your opposition to keeping the only
> integration branch we have, the 'master' branch, building constantly,
> considering this is a basic engineering practice, but at this point I
> can barely muster the strength to even care. In any case, as I said
> to
> Milan, I'll gladly consider any project you maintain at the same
> level
> as an external dependency, and thus just tag it. Feel free to go
> through the Continuous manifest and Jhbuild moduleset and point out
> which modules you still maintain.


Emmanuele,

Thank you for sharing this. You make the perfect example.

You will note, that my "astronauting" is in fact a sincere attempt to
reconcile both of these views with some plausible technical approach,
so while your goals are clearly not the same as mine; I am being
sensitive to your goals, in fact I want some team within GNOME to be
successful at integrating all of these projects into some coherent user
experience - I only want that team to accept that the components they
intend to integrate have bigger plans than *just* being a part of this
integrated whole.

You on the other hand seem threatened by my views and consequently act
intolerant towards the existing diversity of goals of projects within
GNOME - I don't think this intolerance is an attitude we should have to
deal with, we should be instead looking for creative ways to reconcile
both views.

Yes, the GNOME Desktop Environment as a unified thing has never been a
personal goal for me to work towards, rather the most important to me
is to build a solid and stable platform with good development tools,
and I want it to be the platform of choice in the FOSS world - if the
GNOME Desktop Environment turns out to be the only consumer of the work
I've done over the years, this will be the biggest failure of all in my
eyes.

You may have accepted defeat but as long as I spend any energy in GNOME
at all, I simply can not.

Regards,
    -Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Sébastien Wilmet]

On Sat, Jun 18, 2016 at 08:15:19AM +0100, Emmanuele Bassi wrote:
> So, I want to stop this thread on my side because not only is making
> me not want to contribute anything ever again, but it also is
> absolutely pointless busy work that relies on two people architecture
> astronauting without the minimal intention of building anything or
> even consensus.

Citing myself from several mails ago: "With master/master-stable, it's a
more relaxed atmosphere."
https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg2.html

There is a proposal that is simple to understand: when tagging a module
in GContinuous, this would mirror what jhbuild builds by default. With a
branch. That's it.

Then there are more complicated proposals where everything is fully
automated, delivering a stable desktop continuously to end users. I
didn't talk about it, but it would enable to do canary releasing and A/B
testing.

And there is a gray scale between those two proposals.

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Emmanuele Bassi]

So, I want to stop this thread on my side because not only is making
me not want to contribute anything ever again, but it also is
absolutely pointless busy work that relies on two people architecture
astronauting without the minimal intention of building anything or
even consensus.

I consider the matter that started this thread — ensuring that jhbuild
does non-srcdir builds by default — solved, and any talk about
continuous integration in GNOME absolutely moot until somebody comes
up with an actual CI server and tool that is *not* Continuous, and
shows that it works well enough that we can start talking about commit
and integration policies. I also consider the matter of fixing the
Continuous builds solved, in the sense that I'll continue tagging
badly behaving modules and file bugs. That is, obviously, if I can
muster caring about this stuff — something that I keep finding hard to
conjure out of thin air and my limited amount of time, given the
feedback I've received.

Instead of talking about these two matters, let me talk to you about
the only two tools we have at our disposal, and why I (possibly
naively) thought they were important enough to the GNOME project to
raise their continued working state as an important topic.

Let's start with jhbuild. I personally don't care if you're a
long-standing contributor to the GNOME project and you feel like you
don't need to use jhbuild because you either don't give a crap about
your system's state, or because you're competent enough to deal with
it; jhbuild *is* the tool that we provide to newcomers to the project.
It's the major blocker for people that come to use and wish to build
and contribute to GNOME. Since GNOME is a free software project and
contributors are a slowly renewable resource, introducing barriers
like having stuff *not* build is a perfect way to stagnate the
contributors pool, and turn enthusiasts and people willing to
contribute into people that will negatively impact the perception of
GNOME — and with good reason. Our Git repositories, and jhbuild, are
part of our developer outreach. Ensuring that potential new
contributors can follow a simple step tutorial without the build
crapping its pants two thirds into the process is the way we grow our
contributor's base. As much as I wish to retain one existing
maintainer, I'd rather have 10 new contributors every month.

Let's move on to Continuous. As much as it's a build bot, and a
continuous delivery testing mechanism, Continuous is much more, and
much simpler, than that: it's our Minimum Viable Product. It's the
closest thing we have to present to OSVs and OEMs and say: "You can
take this list of modules, with the given commit ids, and build
options; build them; and plop the resulting file system tree into a
product — and we can guarantee you that it'll boot, log in, and give
you the exact same results we seen". You literally have *no* idea how
powerful this is. It allows companies to create a whole operating
system without having to deal with the whims and policies of Linux
distributions. It gives us the ability to QA what we create. It gives
Linux distributions the ability to determine a baseline for their own
packaging, and check if something is a bug they introduced or if it's
upstream. Continuous is GNOME, and GNOME is Continuous.

Both of these things are so important because they demonstrate that
GNOME is bigger than the sum of its parts.

To return to the original topic: Milan, I understand you don't like
any policy imposed on your projects. That's perfectly understandable,
so I will consider Evolution and all its dependent modules as external
dependencies, and treat them as such. I'd even be happy to only build
the stable branches in Continuous, so we can keep the burden at a
minimum, and we'll ensure that GNOME does not use unstable API until
it's been released, but for the time being I'll just keep tagging
stuff — both in Continuous and in Jhbuild.

Tristan: I understand you don't give a crap about GNOME as a holistic
project; you've made it *painfully* clear over the years. I'd like to
point out that without GNOME working as a single unit nothing we do,
or did up until now, really matters in the grand scheme of things. The
*only* successful thing out of GNOME is the GNOME desktop environment
and its whole stack of technologies; everything else has been a
failure, mostly caused by this inane obsession of considering the
GNOME platform as a bag of loosely coupled projects that somebody has
to work hard into identifying and keep working together, granting the
position of gatekeeper to a bunch of middle men. This position is
ignorant of the history of the project, especially of its failures;
it's insulting of the work of hundreds of people who want to keep this
thing working and keep it approachable to newcomers; and completely
invalidated by the reality of what happens every day in the commercial
space. I also don't understand your opposition to keeping the only
integration branch 

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Tristan Van Berkom]

First,
  I've been on vacation this week and have been able to put time into
this, with the hope that next week I will be able to focus on $dayjob
without regretting too much not being able to take part in this
important debate. I hope we are going somewhere. Also, in closing in
this mail, I will just highlight my motivations for bringing up this
proposal in the first place.

Sri,
  Please just gloss over the technical stuff and read the end, I think
you are in a position to effect change and this is written with you in
mind (from the "Why do I think..." heading).

On Fri, 2016-06-17 at 15:17 +0200, Milan Crha wrote:
> On Fri, 2016-06-17 at 17:11 +0900, Tristan Van Berkom wrote:
> > 
> > I dont believe you have any such functionality in jhbuild.
> > 
> > At best, you can specify the sha1 of the commit you want to build
> > of
> > a given module, this would not let you single out one commit,
> > omitting it from history in integration and at least attempt to
> > apply
> > subsequent commits without that one included, re-including that
> > commit in the correct place in history only when CI infrastructure
> > would find integration unbroken.
>   Hi,
> I never thought of a single commit removal from a set of commits. To
> be
> honest, that sounds even more crazy (not 'crazy' like 'insane'). :)
> We are talking about master, the development version of the software.
> If you'd like to tell me that you are at commit X and feature which I
> added at commit X-3 does not work for you, then trying to figure out
> why exactly you do not see that feature is a real nightmare and waste
> of time for both the reporter and the developer, especially when you
> want to remove only single commit from a history. Not talking that
> following commit can "build on top of the previous commit", which is
> quite usual.
> 
> > 
> > The thing is, either way; your work flow will not be interrupted at
> > all with this approach, ...
> See above.

Milan,
    I think that if your commit broke integration with the wider GNOME
ecosystem, that will matter to you and you will be quite aware of which
commit in master broke it and why until it is fixed. I think we have to
ensure that people take integration breaking commits seriously and
ultimately, an integration breaking commit would block any stable
release.

That said, in your above scenario; a disparity between master and
integration is a serious issue, but consider the alternative:

If a new comer wants to get involved in GNOME, they probably have a
specific itch to scratch, say with Evolution or EDS; they will either
be able to build everything up to EDS & Evolution easily; have an easy
experience, and be able to submit a patch to bugzilla, or, they will
fail to build and get side tracked because the bleeding edge of master
for all GNOME modules doesn't build.

In other words (and this has absolutely nothing to do with using
jhbuild or not) when you build everything from master and it breaks
before you are even able to work on the issue you wanted to work on,
this is discouraging; this means that we are offloading our technical
debts to newcomers, forcing them to deal with our bugs before being
able to even submit a patch they wanted to contribute.

Would you prefer that a potential EDS contributor be able to submit a
patch to bugzilla, even if it does not match up *exactly* against
master ? Or would you prefer that your potential contributor get
discouraged while trying to build EDS against the latest gcr,
libsecret, libgdata or such, or just get bogged down trying to fix an
EDS bug to adjust to some API churn in a dependent GNOME library ?

I think receiving a patch for what the contributor wanted to work on is
preferable, even if you happen to fall on a patch that doesn't apply
exactly against master, in the 1% of the time that EDS integration is
broken and lagging behind.

[...]
> I said earlier in the thread that the current situation works for me
> the best. There is clearly stated what the continuous builds from, at
> which exact commit it "stopped updating" certain module, because of
> some breakage, and everything references the real master branch. That
> the continuous uses jhbuild might be just a coincidence, even it
> sounds
> like the moduleset is targeted for the Continuous builds, when it can
> influence an environment for any developer using the jhbuild (I do
> not
> use it, this is how I understood how it works in the jhbuild world).

I'll just note again this is not about jhbuild particularly, jhbuild in
master tries to build everything from master, sometimes it breaks and I
think that's natural in the present state of affairs.

gnome-continuous does *not* build jhbuild modulesets, it uses it's own
json format for, also building everything from master:

https://git.gnome.org/browse/gnome-continuous/tree/manifest.json

This is about getting the bleeding edge, or something as close as
possible to the bleeding edge to be always buildable; for two reasons:

  a.) 

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Milan Crha]

On Fri, 2016-06-17 at 17:11 +0900, Tristan Van Berkom wrote:
> I dont believe you have any such functionality in jhbuild.
> 
> At best, you can specify the sha1 of the commit you want to build of
> a given module, this would not let you single out one commit,
> omitting it from history in integration and at least attempt to apply
> subsequent commits without that one included, re-including that
> commit in the correct place in history only when CI infrastructure
> would find integration unbroken.

Hi,
I never thought of a single commit removal from a set of commits. To be
honest, that sounds even more crazy (not 'crazy' like 'insane'). :)
We are talking about master, the development version of the software.
If you'd like to tell me that you are at commit X and feature which I
added at commit X-3 does not work for you, then trying to figure out
why exactly you do not see that feature is a real nightmare and waste
of time for both the reporter and the developer, especially when you
want to remove only single commit from a history. Not talking that
following commit can "build on top of the previous commit", which is
quite usual.

> The thing is, either way; your work flow will not be interrupted at
> all with this approach, ...

See above.

> I realize my answer is quickly written and I would like to keep my
> mind open and receptive, but I would also like to hear a more
> detailed counter proposal on your end too, weighing the benefits of
> managing this in a jhbuild moduleset vs a unified project wide git
> branch

I didn't think of my workflow changes in any deep way, also because I
do not use jhbuild (that's not a tool for me). I just want to keep
things simple as much as possible. Not only for me, but for the others
too. Adding a new branch adds some non-trivial complexity.

Consider the early adopters, whom will fill bugs against the developer
version of some module. The developer will not only ask "what steps to
do to reproduce the issue", but also "which branch was used and with
what commits". And it happens often that the developers do not react on
bugs "quickly", thus the commit numbers in the integration branch can
be easily lost (branch rebuilt), thus making the bug report more or
less useless.

I said earlier in the thread that the current situation works for me
the best. There is clearly stated what the continuous builds from, at
which exact commit it "stopped updating" certain module, because of
some breakage, and everything references the real master branch. That
the continuous uses jhbuild might be just a coincidence, even it sounds
like the moduleset is targeted for the Continuous builds, when it can
influence an environment for any developer using the jhbuild (I do not
use it, this is how I understood how it works in the jhbuild world).

I do not care about the new branch, if you (plural 'you') will decide
to use it, then I've nothing significant against. I only think it'll
add too much confusion for the developers, that it'll strike back in
some way in the future. I only do not like the idea of random reverts,
the master branch is the pure bleeding edge, thus if the automatically
managed branch is a way to go for you (plural 'you'), to avoid those
reverts, then feel free to do it.

Maybe, only, it would be ideal if the new branch won't influence daily
`git pull` on the projects, due to bandwidth limitation, connection
speed and similar constraints. That would, maybe, better fit to create
a new project for the automatic builds, with the "interesting" git
sub-modules there at certain commits. There might not be much
difference between the new project with submodules and 10K branches in
each module, right? Your new tool would still work on the git commits.
That's still too complex, too limiting for a daily work on the master
(not for me, but for jhbuild users to whom some modules will
automatically stop updating at certain commit which can break other
modules, like with the API changes, but the developer of the affected
module might not ever notice, because he/she will stay at the commit
which precedes the API change).

There had been said in this thread already that having a continuous
integration (automatic builds) is not easy, especially for such a large
project like GNOME. Whenever a "Continuous" was said in this thread a
shortly after followed "jhbuild". It's understood. The jhbuild is the
current tool for the automatic builds. Supporting random automatic
build systems, which is what you said could be done with the
'integration' branches, is not needed from my point of view, due to the
GNOME project complexity. Thus target jhbuild only. Using modulesets
(it's called like that, right?) for jhbuild and writing an
export/import utility to transform the jhbuild's moduleset into another
automated build integration could be also doable, hypothetically
speaking. If any such thing would be needed in the future. That is,
let's target jhbuild first. The jhbuild already has a functionality 

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Milan Crha]

On Thu, 2016-06-16 at 14:16 +0900, Tristan Van Berkom wrote:
>   o The integration branch of every module is continuously
> reconstructed based on that module's master branch.

Hi,
as the 'integration' branch targets only jhbuild, then it doesn't make
sense to add it at each git module, no? Thinking of it, instead of
playing with a jhbuild only, you propose to add one more layer (and
place of possible issues) between the git master <-> jhbuild
connection. The proposal, as I understand it, is:
  git master <-> git integration <-> jhbuild
where the git integration can be sometimes skipped in the jhbuild. But
as you already have a functionality to skip some commits inside the
jhbuild, and the integration branch is meant to do exactly that, then
why to have one software which would eat only space on a disk in each
git module with a new branch history, instead of writing the automated
build tool which would cooperate directly with the jhbuild setup? You
also want to add some restrictions on the git integration branch, which
is even more unnecessary work.

Not having the branch will make things easier for the developers and
new comers too, as there will be no confusion for them. Having
somewhere a super-detailed description of the integration branch
intention is nice, but it'll be easy to forget of it during the time.
Bye,
Milan
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Tristan Van Berkom]

On Wed, 2016-06-15 at 17:49 +, Sriram Ramkrishna wrote:
[...]
> I think with a couple of iteration we can work out an agreement.  :)

Hi Sri,

I can see you read through this diagonally, so I'll try a TL;DR version
below...

> So to wit, let me summarize your points:
> 
> * Master is unstable and can be in unbuildable state until we apply a
> tag making it stable.

Basically what it is today, mostly always buildable modulo transitional
periods in times of API churn, and modulo some mistakes which may take
time to fix for maintainers in different timezones with dayjobs.

> * We have a 'integration' branch or something like that which is
> always in buildable state but lags behind master by some undetermined
> number of commits.

The integration branch as proposed does not lag behind except by
commits which break integration, in all other respects it is always
exactly master.

> I'm a little unclear what a maintainer's workflow is, and how can we
> ascertain that they push to this integration branch when there is no
> social policy in place to do so.  I mean some maintainers can
> completely ignore that if they wanted to.

Nobody ever pushes to 'integration' (at least not developers and
maintainers), and this is why I want some strict git hooks in place to
ensure that any pushes to the integration branch are immediately
rejected ('git push origin integration' shows you an error).

TL;DR version is basically:

  o Yes, this is going to take some actual work to implement, technical
work, but not an astronomical amount, maybe a few weeks.

  o The integration branch of every module is continuously
reconstructed based on that module's master branch.

This must be an automated process, which runs perhaps every 10 min
or so, updating integration branches of all modules whenever new
commits are available on master.

At first it wont be completely automated, it will require some
human intervention to assist in maintaining the blacklist of
broken commits (we have a problem to solve which is deciding
which commits to blacklist in the cases of API churn).

Ideally though, the buildbots should be able to reconstruct
integration by performing many variants and deciding on the
blacklist with least changes omitted.

  o Notifications are delivered, perhaps bugs filed, against commits
in master which dont integrate properly into 'integration'

No need for a grace period, when your commit to master breaks
integration, it is omitted from integration and you get a bug
report.

  o Default for jhbuild is 'integration' of everything, which is
    almost always buildable (in fact it could be made to be literally
    always buildable with a bit more effort, assuming some tries
    are performed by the build bots before constructing the integration
    branches).

    In terms of workflow this would mean people would always work
    against the integration branch, except that developers/maintainers
    of a given specific module would checkout a few modules as master.

    Commits only ever enter the history via regular master or stable
    branches.


Basic gist of the idea is like this.

Cheers,
-Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: [ Revised Proposal ] Continuous Builds in GNOME

[Sriram Ramkrishna]

On Tue, Jun 7, 2016 at 9:45 PM Tristan Van Berkom 
wrote:

> On Tue, 2016-06-07 at 22:24 +0200, Sébastien Wilmet wrote:
> [...]
> > > This of course, requires real thought and engineering, but I wanted
> > > to
> > > at least offer some technical input, some starting point that could
> > > be
> > > explored - this wont be solved without real engineering, trial and
> > > error.
> > It's similar to the "master/master-stable-next/master-stable"
> > branches
> > concept that I thought about in this previous sub-thread. See those
> > two
> > mails:
> > https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg2
> > .html
> > https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg8
> > .html
> >
> > It doesn't look like rocket science to me, and it would get us closer
> > to
> > continuous delivery.
>
> Honestly I hadn't put a huge amount of thought to this, but I did
> figure in that once there is API churn, there will be multiple
> variations/combinations that are desirable to build - one might want to
> build a certain app which has not yet adapted to an API change, in
> which case they need an amalgamation of branches which gets them their
> app building without the API change in a lower level library/service.
> On the other hand someone who wants bleeding edge of the given
> library/service will want it with the API change included... but I
> admit these considerations are possibly not worthwhile for our basic
> goals: improve CI for development cycles and help have a more friendly
> starting point for onboarding newcomers.
>
> I also woke up this morning to an interesting conversation on #gnome-
> hackers, which I missed due to timezone skew, but made me think of
> something cheaper also more inline with what you suggest with master-
> stable, master-stable-next etc.
>
> So here it goes, after reading the interesting conversation I think
> that we are close to something that is a step forward which will not
> cost very much at all, I think that the main point we disagree on is
> reverts in *master*, and the idea that *master* is something that can
> be forced to be always stable.
>

I think with a couple of iteration we can work out an agreement.  :)

So to wit, let me summarize your points:

* Master is unstable and can be in unbuildable state until we apply a tag
making it stable.
* We have a 'integration' branch or something like that which is always in
buildable state but lags behind master by some undetermined number of
commits.

I'm a little unclear what a maintainer's workflow is, and how can we
ascertain that they push to this integration branch when there is no social
policy in place to do so.  I mean some maintainers can completely ignore
that if they wanted to.

I think in general we still need to figure out how to socially have
compliance as I can envision some maintainers not wanting extra work for
sake of argument.

 [snip]


> Any thoughts ?
>
>
Thanks for taking the time to putting your thoughts and contributing
positively to this discussion and making suggestion, well appreciated by
the rest of us.  I don't think we have gotten any feedback on this and I'm
bumping this again to see if people can look at Tristan's proposal as
something workable and acceptable as a base of a conversation around doing
this.

sri
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

[ Revised Proposal ] Continuous Builds in GNOME

[Tristan Van Berkom]

On Tue, 2016-06-07 at 22:24 +0200, Sébastien Wilmet wrote:
[...]
> > This of course, requires real thought and engineering, but I wanted
> > to
> > at least offer some technical input, some starting point that could
> > be
> > explored - this wont be solved without real engineering, trial and
> > error.
> It's similar to the "master/master-stable-next/master-stable"
> branches
> concept that I thought about in this previous sub-thread. See those
> two
> mails:
> https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg2
> .html
> https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg8
> .html
> 
> It doesn't look like rocket science to me, and it would get us closer
> to
> continuous delivery.

Honestly I hadn't put a huge amount of thought to this, but I did
figure in that once there is API churn, there will be multiple
variations/combinations that are desirable to build - one might want to
build a certain app which has not yet adapted to an API change, in
which case they need an amalgamation of branches which gets them their
app building without the API change in a lower level library/service.
On the other hand someone who wants bleeding edge of the given
library/service will want it with the API change included... but I
admit these considerations are possibly not worthwhile for our basic
goals: improve CI for development cycles and help have a more friendly
starting point for onboarding newcomers.

I also woke up this morning to an interesting conversation on #gnome-
hackers, which I missed due to timezone skew, but made me think of
something cheaper also more inline with what you suggest with master-
stable, master-stable-next etc.

So here it goes, after reading the interesting conversation I think
that we are close to something that is a step forward which will not
cost very much at all, I think that the main point we disagree on is
reverts in *master*, and the idea that *master* is something that can
be forced to be always stable.

For some of us, Milan and I at least, I think we disagree most
particularly on dirtying master, this creates work especially if we
have to pick up the pieces after some API change or such, and it
dirties the mainline development history which is a bad thing in
general, it makes bisections dirty and history tracking dirty, we've
come to have some discipline over the years to even avoid any
whitespace in commits because we value the history that much, we want
to know that every commit in mainline is relevant.

Simply put, if master is broken, it needs to be fixed the correct way,
and a maintainer has never advertised master as being stable until we
tag it as stable, we want to keep this.

So what I thought up after reading that IRC conversation, is that there
is no reason at all why all of this has to be done in master, and no
justification to want this behavior on master either - we just want to
be able to provide a closest-to-master as possible build that always
builds, and we want that to be the "easy way" that we advertise to
newcomers - more mature developers when they grow up might want to
build master anyway and see the actual breakages in real time.

Sorry I dont have time to spend hours reducing this huge text and being
all concise, but I want to contribute to this, so here goes:

Cheap proposal for a build-bot/release-team/build-sherif owned
integration branch
~~
So what I would propose here is a setup with some automation, similar
to the proposed master-stable but I would call it 'integration' and
with a few changes.

Starting with our current workflow, all our projects/modules use the
master branch as mainline development, feature branches are merged into
master and eventually come release time, master is tagged and branched
as stable and we release tarballs for each release tag.

 [ feature ]   |
 \ |
  \
  [ master ]
   |
   |
   [ release tag ]
\
   |  \
 [ feature ]   |\
           \   |  \
            \  |\
             \ |  [ release branch ]
  \|   |
   |   |
   [ bugfix ] ---> |
   |   |
   |  [ bugfix release ]

I dont think the above should change at all; patch flow for production
continues to flow in this traditional way that we have come to work
with very well. Master remains mainline.

What I would suggest, in addition to our setup, is that there is
a new 'integration' branch for every module in the moduleset, this
integration branch is automatically managed.

Workflow for contributors can look more like this:
[ integration ]   [ master ] [ bugzilla patches ]
   |   |/
   |   |  /  patches possibly
   |   |/made against
   |   |  /