Re: Build sheriffs for GNOME

[SAHIL SAREEN]

Inspired from this thread, I build a graph based application to check the 
health of GNOME games which uses gnome-continuous and builds a Neo4j graph on 
which one can see the build status of modules, see their dependencies and query 
the graph(an example on my blog post which filters dependencies that are only 
in use by a module whose build has failed).

Check out my blog post: 
https://blogs.gnome.org/ssareen/2016/01/31/gnome-games-health-analytics/
and project on GitHub: https://github.com/sahilsareen/GNOMEGamesHealthAnalytics

Comments and suggestions much appreciated.


Cheers!
---

Sahil Sareen
Foundation Member and Game Dev, GNOME
Software Dev Engineer, Amazon


From: desktop-devel-list  on behalf of 
Michael Catanzaro 
Sent: Tuesday, January 26, 2016 4:30 AM
To: desktop-devel-list@gnome.org
Subject: Re: Build sheriffs for GNOME

On Thu, 2016-01-21 at 12:19 -0600, Michael Catanzaro wrote:
> The next step is to convert either jhbuild or gnome-continuous to use
> the others' modulesets.

In retrospect, this was an ignorant suggestion. The Continuous manifest
does not specify dependencies, so it's unsuitable for use by JHBuild. I
don't think Continuous would benefit one bit by adopting JHBuild
modulesets.

What we want is two separate CI systems: GNOME Continuous as it is now,
and a separate system for testing JHBuild.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Thu, 2016-01-21 at 12:19 -0600, Michael Catanzaro wrote:
> The next step is to convert either jhbuild or gnome-continuous to use
> the others' modulesets.

In retrospect, this was an ignorant suggestion. The Continuous manifest
does not specify dependencies, so it's unsuitable for use by JHBuild. I
don't think Continuous would benefit one bit by adopting JHBuild
modulesets.

What we want is two separate CI systems: GNOME Continuous as it is now,
and a separate system for testing JHBuild.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Alexandre Mazari]

Not being a gnome developer, what follows is just a report of what I
experienced in a continuous integration/delivery environments at
$BIGCORPs where HEADs always build together, test stay green and
developers are (hopefully) able to evolve the design and
implementation without much hindrance.

Some ingredients are fundamentals toward those goals:
pre-hitting-master build/tests, clever versioning/dependencies
expression and atomic cross-repo pushes.

Pre-hitting-master build/tests:
As Lasse Shuirmann proposed, pushes are made to one-shot "quarantine"
remotes|branches instead of master directly. A CI bot instance is
launched, rebuilding the whole modules set and executes all the tests
suits.
If everything's fine, the commit goes to master. Yay! Else the
committer is notified and try again.
+ master HEADs always build together.
+ the feedback loop for the developer is hopefully direct and fast
- lot of resources consumption
- patches to different repos cannot be applied at the same time and
some sequencing is mandatory.

Clever versioning:
Semver, that you already do, ensure that API breaks are clearly
expressed in version bumps. The point regarding API breakage is moot
for build dependencies: a client project expresses dep on (range of)
specific versions known to match the API they expect. The dependency
version is bumped once the appropriate porting work is done.

Atomic cross repositories pushes:
But actually, not all dependencies are build-time enforceable. DBus
APIs, DConf schemas, GIR symbols used by dynamic languages bindings
etc... are all mostly resolved at runtime. Breakages of those wont be
caught by autotools and the compiler. Ideally, those runtime
requirements should be stressed by extensive tests coverage (or even
better, a proper type-system :P) but does such a thing even exist?

Furthermore, sometime API clients must be updated at the same time as
the API provider. If the API breakage is required to fix a critical
bug or security issue.

In such case, API producers and consumers *must* be updated at the
same time. As they often lay in different repositories, some tools
(gitlab, gerrit AFAIK) introduced the notion of atomic cross
repositories push/merge requests. Those are a set of patches to
different repositories that may only be applied or reverted together
as a unit.

All boils down to the following dichotomy: in a modular environment,
and for the sake of system health as a whole, is inter-modules
contracts enforcement the responsibility of the producers or the
consumers?

(As a side note, why would you ever commit a public API breakage if it
wasn't called in by, and validated with, at least one of your
consumers ? It sounds like breaking for the sake of breaking, IMHO)

Not sure how this all fits in current Gnome environment but keeping
HEADs buildable and green as an organization rule made my (and my
coworker's) daily duties a lot easier: devenv bootstrapping,
pinpointing regressions, not having to wait on others fixes and so
on... are very confortable.

On Fri, Jan 22, 2016 at 8:41 PM, Michael Catanzaro  wrote:
> On Fri, 2016-01-22 at 13:32 -0600, Michael Catanzaro wrote:
>> Ah, yeah, we should not be reverting entire large sequences of
>> commits... I agree that would be excessive. In cases like this, we
>> can
>> tag your module in Continuous and branch it in jhbuild, until other
>> modules have caught up with your API changes. Build sheriffs should
>> try
>> the least-disruptive option possible to get things fixed.
>
> We could also keep a list of maintainers who request that we not touch
> their modules directly, so we know to branch and tag your module when
> something breaks instead of committing to it directly. I suspect not
> many maintainers will opt-in to this, but you've made it clear there is
> demand!
>
> Michael
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
"If you open your mind too much, you brain will fall out"
Tim Minchin
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Sun, 2016-01-24 at 19:02 +0900, Tristan Van Berkom wrote:
> What jhbuild is not, is a delivery system to users who want to try
> the
> latest bleeding edge version of every project in GNOME together. Even
> though it is possible that 80% of the time or even more, every
> project's master branch happens to build together harmoniously, it's
> simply a fallacy to presume that it can always be so.

I will just say, I very strongly disagree

I would like us to keep all of GNOME buildable. I don't think it's so
hard.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Tristan Van Berkom]

On Sun, 2016-01-24 at 05:18 +0100, Vamp898 wrote:
> Hi, 
> 

Hi Shirakawa,

I see you replied off list, I am returning this to the list because it
is a good question to answer and it's rather wasteful to compose a full
reply to this and not have the benefit of it being recorded in the
archives for posterity. Also if we shake the tree on the list, maybe
something interesting will fall out :)

> I hope this mail doesn't sound too stupid (I'm not that into the
> GNOME build system). 
> 
> Jhbuild checks out the git repository and builds them. 
> 
> I personally would say that even master should be buildable,
> otherwise it's not of much use, or is it? 

No it does not sound stupid at all, without a deeper understanding of
how the projects inside GNOME function when they are under active
development combined with some understanding of the history of how
things were done before we had jhbuild, it is perfectly understandable
that you might think that is what jhbuild is for.

In fact, it is even possible that some people within GNOME have
presented jhbuild to you as if it were a tool you could use to always
build and "obtain" the latest master of everything, and that it should
always work, that if it did not build it was in fact a problem that
some module could be blamed for. Things are a bit more complex than
just this, though.

I would in fact argue that jhbuild is exactly the opposite of this, one
of the great benefits of jhbuild is that we find incompatibilities and
breakages early on in the cycle rather than in the last few weeks
before release time.

> I rather have an old eds than none (and if master is broken due to
> changes in eds, it's like having nothing). 

There is one thing to take in context here: GNOME has never had a
policy which mandates ABI or even API stability for ALL projects which
participate in GNOME. API stability is only a hard requirement for core
platform modules who have opted in and advertised this, understandably
because these platform components are relevant also to the real world
outside of GNOME. Historically we have a competitive platform, and lets
hope that with work and discipline our platform will again be a viable
and top choice for mobile, IVI and other embedded sectors (but I am
digressing...).

The point is, for any project in GNOME which does not advertise API
stability, these projects are entirely free to change the API contract
during the course of a release cycle. Of course when this happens,
there are periods where other projects in GNOME depending on these will
be broken, and this is not incorrect or undesired. This does not happen
that frequently, but it's reasonable to expect that some applications
wont build with "everything master" for periods of weeks at a time.

We could mandate that every project which provides an API be always API
stable and extend this rule beyond core platform components, but this
would generate a lot more work for some not funded volunteer driven
projects, and it's an entirely different discussion.

> I'd say touching git (in terms of reverting commits) is the worst
> idea. 
> 
> Jhbuild should take care of that. Jhbuild should find ( with the help
> of tags in git?) what is the latest thing that still builds in
> combination and check out that. 

It's not possible for jhbuild or continuous to distinguish at this
time:
  a.) When an application is broken
  b.) When a library broke that application
  c.) When a library changed their API contract and the application
  is rightfully broken in a transitional period

Further, it would not be desirable for jhbuild to automatically do what
you suggest, because then we would not see the breakage, exposing that
breakage as early as possible is one of the primary values jhbuild has
given us.

In order to shed some light, I want to share a little bit of history of
how things were done before we had jhbuild, because I will argue that
jhbuild is not a delivery system of latest GNOME, but that does not
decrease the value of jhbuild, it has really, really been a great step
forward in our development cycles.

So, before jhbuild, most developers of GNOME projects had their own way
of building their project against bleeding edge versions of other
components. Everyone had a different OS with different third party
dependency versions installed, and everyone did everything by hand.

What that means, is usually I would have a few lines of environment
script I would use to build:

  PREFIX=/opt/devel
  PATH=${PREFIX}/bin:$PATH
  LD_LIBRARY_PATH=${PREFIX}/lib:${LD_LIBRARY_PATH}
  PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}
  ACLOCAL_FLAGS="-I ${PREFIX}/share/aclocal"

  export PATH LD_LIBRARY_PATH PKG_CONFIG_PATH ACLOCAL_FLAGS

Personally I would mostly be building Glade, so to do that, usually I
would have to get the latest version of GTK+, so usually I would have
to update atk, cairo, pango, gdk-pixbuf and gtk+, and I would have to
do so in that order specifically.

I would source my enviro

Re: Build sheriffs for GNOME

[Bastien Nocera]

> On 22 Jan 2016, at 17:32, Michael Catanzaro  wrote:
> 
>> On Sat, 2016-01-23 at 03:14 +0900, Tristan Van Berkom wrote:
>> I do not support the vague notion that a "sheriff" can come along and
>> revert an entire branch I've just landed which changes the API
>> contract
>> early enough in the release cycle for depending modules to catch on
>> and
>> adapt to.
> 
> Ah, yeah, we should not be reverting entire large sequences of
> commits... I agree that would be excessive. In cases like this, we can
> tag your module in Continuous and branch it in jhbuild, until other
> modules have caught up with your API changes. Build sheriffs should try
> the least-disruptive option possible to get things fixed.
> 
> In practice, I don't think this is very often an issue; usually it's
> just little things that break. The most recent example comparable to
> your scenario would be grilo-3.0. In such cases, it usually makes more
> sense to tag/branch the module that broke API for a bit, or to try
> fixing the build in the affected modules.

Huh. All the modules I knew about using grill were ported within a couple of 
hours, the change was announced and an older version of the API was available 
in a branch (again, listed in the announcement).

You're basically complaining that the Music maintainers didn't update jhbuild. 
Which I eventually did after porting the app.

There are certainly plenty of better examples than this one.

> 
> Michael
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Fri, 2016-01-22 at 13:32 -0600, Michael Catanzaro wrote:
> Ah, yeah, we should not be reverting entire large sequences of
> commits... I agree that would be excessive. In cases like this, we
> can
> tag your module in Continuous and branch it in jhbuild, until other
> modules have caught up with your API changes. Build sheriffs should
> try
> the least-disruptive option possible to get things fixed.

We could also keep a list of maintainers who request that we not touch
their modules directly, so we know to branch and tag your module when
something breaks instead of committing to it directly. I suspect not
many maintainers will opt-in to this, but you've made it clear there is
demand!

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Sat, 2016-01-23 at 03:14 +0900, Tristan Van Berkom wrote:
> I do not support the vague notion that a "sheriff" can come along and
> revert an entire branch I've just landed which changes the API
> contract
> early enough in the release cycle for depending modules to catch on
> and
> adapt to.

Ah, yeah, we should not be reverting entire large sequences of
commits... I agree that would be excessive. In cases like this, we can
tag your module in Continuous and branch it in jhbuild, until other
modules have caught up with your API changes. Build sheriffs should try
the least-disruptive option possible to get things fixed.

In practice, I don't think this is very often an issue; usually it's
just little things that break. The most recent example comparable to
your scenario would be grilo-3.0. In such cases, it usually makes more
sense to tag/branch the module that broke API for a bit, or to try
fixing the build in the affected modules.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Tristan Van Berkom]

On Fri, 2016-01-22 at 11:40 -0600, Michael Catanzaro wrote:
> On Fri, 2016-01-22 at 09:25 -0800, Jasper St. Pierre wrote:
> > It's more frequent than you might think.
> > 
> > In the past week, alone, we've had to tag glib, gnome-calculator,
> > e-d-s, gstreamer, and more, all because of broken builds. Take a
> > look
> > at how busy gnome-continuous is as a project:
> > https://git.gnome.org/browse/gnome-continuous/log/
> 
> Absolutely. I should be more clear: it doesn't happen so often for
> any
> PARTICULAR module, so the git history of any particular module is not
> likely to get polluted by many revert commits. (And even if it does
> --
> so what if your history doesn't look quite as nice as you'd like.)
> But
> it happens a LOT for GNOME as a whole, and it's a big problem, which
> is
> why I'm so supportive of Emmanuele's plan.

I am still a bit baffled by this, as I expect a higher standard of
quality from GNOME contributors than I would for your regular corporate
r&d lab.

So let me reiterate what I mentioned in my first email, I am not
against this proposal for the stable branches, and, for master: I think
Emmanuele's proposal needs clarification.

To clarify what I mean by that, is it should be clear what a sheriff
has, and has not the right to revert. If I committed a spelling error
which obviously broke the build of my own module, *doh*, of course,
it's a single commit, a stupid one, and I really couldnt care less if
it were to be reverted by someone else who caught the error. Jasper
seems to indicate that this kind of thing actually happens more than I
would expect, this is indeed sloppy and should be addressed.

I do not support the vague notion that a "sheriff" can come along and
revert an entire branch I've just landed which changes the API contract
early enough in the release cycle for depending modules to catch on and
adapt to.

As I have elaborated enough in my reply to Shaun, this kind of thing
happens, frequently enough, and there are transitional periods where
building all of GNOME and expecting everything to build is just not
rational, responsible maintainers will be careful to introduce churn as
early as possible in the cycle.

Best Regards,
    -Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Tristan Van Berkom]

On Fri, 2016-01-22 at 12:12 -0500, Shaun McCance wrote:
> On Fri, 2016-01-22 at 16:03 +0900, Tristan Van Berkom wrote:
> > On Thu, 2016-01-21 at 14:54 +, Emmanuele Bassi wrote:
> > [...]
> > > This is not enough, and it does not raise the bar in keeping
> > > GNOME
> > > in
> > > a buildable state. It actually lowers it a fair , to the
> > > effective
> > > point that *nobody* cares about Continuous builds.
> > > 
> > > I want this to change. I want to be able to revert failing
> > > commits
> > > on
> > > the offending modules, if they are hosted on GNOME
> > > infrastructure,
> > > if
> > > they fail for more than N hours, and *then* open a bug about it.
> > 
> > Hi Emmanuele,
> > 
> > In GNOME, we have got along well for many years with many projects
> > sharing the same infrastructure mostly because we find that in
> > practice, people are generally respectful enough of eachothers work
> > to
> > recognize a module maintainer's word as scripture. As such we do
> > not
> > commit changes to others modules without expressed permission
> > unless
> > it
> > is for a minor change that we are absolutely sure about.
> 
> It has always been my understanding that master (née trunk née HEAD)
> is
> supposed to be buildable in GNOME, and that the release team has the
> power to make commits to ensure it is.

And it has always been my understanding that it is simply not the case,
notably because of said cross module API changes.

It annoys me that in recent years, notably after the introduction of
jhbuild, which has saved us a lot of trouble of setting up local
environment scripts and building everything by hand, that this
expectation has somehow arisen, this expectation is just causing
friction, it simply cannot be expected within reason.

> I am sympathetic to the difficulty of synchronizing cross-module API
> changes. But is there any reason those changes can't all be done in
> development branches in each module, with all of them merged to
> master
> at the same time when they're ready?

Yes.

Here is one example:

Michael Catanzaro writes in this very thread:
> Milan, you have full commit access to every repo on git.gnome.org.
> Any breakage that could register on Continuous is fully within your 
> ability to fix.

I find this statement very disconcerting, and it relates and is a good
example.

Milan maintains EDS, and EDS is a relatively *huge* and complex source
base, I personally know Milan to be considerate about API breakages and
stability in general and I trust that if he has to change the API
contract, it is in the interest of long term maintainability of this
very complex project which has gone though a *lot* of churn over the
last decade.

Milan does not maintain libfolks, and he does not maintain gnome-
contacts, if he is going to spend weeks refactoring EDS and Evolution,
I am not going to tell him that his word is not the law, his word on
the EDS contract *is* law, and I'm confident that he will announce it,
and that maintainers of related modules will have a heads up with
enough time to adjust for the next stable GNOME release.

I am not going to tell Milan that hey, I know you spent weeks or months
of your life refactoring and making EDS better, but I will not accept
that you land your work in your own module unless you spend another
week modifying libfolks and/or gnome-contacts.

Changes like this need to be introduced as soon as they are ready in
master, which is the unstable development branch until the next stable
release is made. It cannot be delayed until the maintainer of libfolks
returns from vacation and wraps his head around the new changes, then
we lose time and we cause friction with other patches landing in
master, accumulating conflicts and generating more work - this work is
not necessary, because nobody every guaranteed that "all of GNOME
master" would build harmoniously together at a given time.

Here is another example, Benjemin Otte one week or two ago landed some
huge refactoring in GTK+ related to drawing - as a result, Glade is
entirely unusable. It's entirely possible that I may have had an
installable unit test in Glade to raise a flag on this.

Are you, really, going to go and revert the landing of his branch
because the unit test flagged GTK+ as broken ?

Sure, for a single project which builds regularly against a stable
platform, the expectation is always that master builds. But this
analogy of a single project cannot not carry on to GNOME, which is an
assembly of distinctly separately run projects which are developed
under the same infrastructure, integration issues will obviously arise
and the best that we can hope for is introducing those issues as early
as possible in the cycle and try our best to stabilize by the end of
the cycle, when churn comes to a halt and individual modules stabilize.

Best Regards,
    -Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/li

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Fri, 2016-01-22 at 09:25 -0800, Jasper St. Pierre wrote:
> It's more frequent than you might think.
> 
> In the past week, alone, we've had to tag glib, gnome-calculator,
> e-d-s, gstreamer, and more, all because of broken builds. Take a look
> at how busy gnome-continuous is as a project:
> https://git.gnome.org/browse/gnome-continuous/log/

Absolutely. I should be more clear: it doesn't happen so often for any
PARTICULAR module, so the git history of any particular module is not
likely to get polluted by many revert commits. (And even if it does --
so what if your history doesn't look quite as nice as you'd like.) But
it happens a LOT for GNOME as a whole, and it's a big problem, which is
why I'm so supportive of Emmanuele's plan.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Sat, 2016-01-23 at 01:55 +0900, Tristan Van Berkom wrote:
> This kind of breakage is just a fact of life, you cannot sweep it
> under
> the rug and pretend it does not exist.
>
> Sure, it's a barrier of entry for newcomers and an annoyance for
> experienced developers who realize the facts of life. But that's just
> the way it is, there is no reason to complain about it.
> 
> Software development is difficult, integration is difficult when you
> have many moving parts that are in a development stage before being
> release ready.

Of course we can't pretend it does not exist, but we certainly can work
on side branches to keep things building. It's pretty easy and it does
solve the problem

> And to be doubly clear: the master branch is *not* what is advertised
> by upstream maintainers as stable, it is the bleeding edge, it can
> have
> churn, it may not always match up to other modules perfectly, other
> modules may have not yet adapted to the churn. Building master may be
> asking for trouble, again, this is just a fact of life if you are
> trying to integrate with all bleeding edge component in the middle of
> their development cycle, instead of what maintainers have published
> as
> stable.
> 
> If you want something that you are sure will build, take the stable
> release tag and build that.

But master is what other GNOME contributors need to be able to build in
order to contribute to your module or any modules that depend on your
module. I would very much appreciate it if you would work in a
sidebranch if you need to make a change you know will break the build
for some module.

Thanks,

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Jasper St. Pierre]

It's more frequent than you might think.

In the past week, alone, we've had to tag glib, gnome-calculator,
e-d-s, gstreamer, and more, all because of broken builds. Take a look
at how busy gnome-continuous is as a project:
https://git.gnome.org/browse/gnome-continuous/log/

We currently have a team of people (usually Michael, Emmanuele, Colin
and Javier) who babysit gnome-continuous to file bugs for broken
components, tag the relevant component in GNOME Continuous, but it's
not a great workflow, in that it doesn't tend to keep master
buildable.

The first mail didn't really say how often build breakages in GNOME
actually happen -- they happen *a lot*.

On Fri, Jan 22, 2016 at 8:55 AM, Tristan Van Berkom
 wrote:
> On Fri, 2016-01-22 at 10:37 -0600, Michael Catanzaro wrote:
> [...]
>> I really don't think it's a big deal to have a few reverts in the git
>> history. And anyway, reverts should be relatively rare, because build
>> breakage should be relatively rare.
>>
>> The master branch may be unstable, but that doesn't mean build
>> breakage
>> is ever acceptable. If your code is not buildable at all times in
>> Continuous, keep it on a *side branch* -- otherwise, tag your module
>> in
>> Continuous, branch it in jhbuild, and do whatever you want in master.
>> But as long as you have an official GNOME module, if it's not
>> branched
>> and tagged, master needs to build. This isn't the wild west.
>
> Of course, when you commit something and your *own* code does not
> compile, this is a very, very rare thing, rare enough to not be worth
> discussion I think, it that's happening, it's a problem.
>
> But it's common enough with many moving parts that the breakage is
> indeed intentional, in which case screwing with the history is only
> intrusive.
>
>>
>> > My interpretation of your proposal is that you would want to
>> > revert
>> > this api break or behavioral change if it were found to break
>> > the
>> > build (or the tests) of another module, during this unstable
>> > development period.
>> >
>> > This would obviously be wrong and would just be an obstacle to
>> > progress in the cases where breakage is intentional and the
>> > only
>> > correct solution is to have applications adapt to the new API
>> > or
>> > behavior before the next stable release.
>>
>> It's a big problem that some modules break for days or weeks at a
>> time
>> during API transitions; this makes it very difficult for new
>> contributors to build and run GNOME, and it's inconvenient for
>> experienced contributors as well.
>
> This kind of breakage is just a fact of life, you cannot sweep it under
> the rug and pretend it does not exist.
>
> Sure, it's a barrier of entry for newcomers and an annoyance for
> experienced developers who realize the facts of life. But that's just
> the way it is, there is no reason to complain about it.
>
> Software development is difficult, integration is difficult when you
> have many moving parts that are in a development stage before being
> release ready.
>
> [...]
>> To be clear: if you've branched and tagged your module, then you do
>> whatever you want in master without hurting anyone else. But
>> otherwise,
>> master needs to be buildable.
>
> And to be doubly clear: the master branch is *not* what is advertised
> by upstream maintainers as stable, it is the bleeding edge, it can have
> churn, it may not always match up to other modules perfectly, other
> modules may have not yet adapted to the churn. Building master may be
> asking for trouble, again, this is just a fact of life if you are
> trying to integrate with all bleeding edge component in the middle of
> their development cycle, instead of what maintainers have published as
> stable.
>
> If you want something that you are sure will build, take the stable
> release tag and build that.
>
> Best,
> -Tristan
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
  Jasper
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Shaun McCance]

On Fri, 2016-01-22 at 16:03 +0900, Tristan Van Berkom wrote:
> On Thu, 2016-01-21 at 14:54 +, Emmanuele Bassi wrote:
> [...]
> > This is not enough, and it does not raise the bar in keeping GNOME
> > in
> > a buildable state. It actually lowers it a fair , to the effective
> > point that *nobody* cares about Continuous builds.
> > 
> > I want this to change. I want to be able to revert failing commits
> > on
> > the offending modules, if they are hosted on GNOME infrastructure,
> > if
> > they fail for more than N hours, and *then* open a bug about it.
> 
> Hi Emmanuele,
> 
> In GNOME, we have got along well for many years with many projects
> sharing the same infrastructure mostly because we find that in
> practice, people are generally respectful enough of eachothers work
> to
> recognize a module maintainer's word as scripture. As such we do not
> commit changes to others modules without expressed permission unless
> it
> is for a minor change that we are absolutely sure about.

It has always been my understanding that master (née trunk née HEAD) is
supposed to be buildable in GNOME, and that the release team has the
power to make commits to ensure it is.

I am sympathetic to the difficulty of synchronizing cross-module API
changes. But is there any reason those changes can't all be done in
development branches in each module, with all of them merged to master
at the same time when they're ready?

>  There is/was a
> page describing this policy to those who receive commit access
> explaining the responsibility that comes with being a committer, it
> has
> either gone missing or I cannot currently find the text.
> 
> In that context, I admit that I first interpreted your proposal as
> hostile, at least in it's current form, because it violates this
> policy
> which has allowed our projects to exist in harmony for so many years
> under the GNOME umbrella, by awarding some kind of revert power to an
> arbitrary group of individuals. I can see much opportunity here for
> unnecessary dispute, if special care is not taken, we may risk
> incentivizing maintainers to host their projects elsewhere.
> 
> On the other hand, it is desirable to have things building regularly,
> of course the master branch is inherently unstable - that's what it's
> for, but I would however support your proposal without hesitation
> were
> it to be applied to stable branches only.
> 
> Were we to apply your proposed policy to master, I think we need
> further thought and clarification before carrying that out:
> 
>   o We are a volunteer driven project with contributors distributed
> across timezones, who have dayjobs, it is ridiculous to expect
> that
> maintainers can be reachable within a 3 hour turnaround period.
> 
> This leads me to believe that you fully expect to cause friction
> by
> reverting commits before said maintainers have the time to even
> respond, and doing so on the unstable master branch.
> 
> The master branch should ideally not be littered with reverts and
> reverts of reverts, it is the main history of project development
> and adding such noise to master should be avoided as much as
> possible for the purpose of preserving the integrity and value of
> the history.
> 
> I think this is unfriendly at best and I would prescribe a one
> week
> period for master. We should really wait for the maintainers word
> before intruding and reverting commits which introduce breakages
> which may have even been intentional.
> 
> 
>   o Not all libraries advertize API/ABI stability, especially when it
> comes to libraries whos only consumers are GNOME applications, we
> allow ourselves much more leniency here than with platform
> libraries which are also relevant outside of the GNOME ecosystem.
> 
> During the course of unstable development, it is entirely common
> for a library to introduce a new API and then to later change it,
> even more than once before it is ready for a stable release.
> For instance, EDS's backend facing APIs are not stable, and there
> have been regular issues in the vala API it exposes and libfolks
> consumes.
> 
> There are periods where it must be accepted that things are just
> broken in a transitional phase.
> 
> My interpretation of your proposal is that you would want to
> revert
> this api break or behavioral change if it were found to break the
> build (or the tests) of another module, during this unstable
> development period.
> 
> This would obviously be wrong and would just be an obstacle to
> progress in the cases where breakage is intentional and the only
> correct solution is to have applications adapt to the new API or
> behavior before the next stable release.
> 
> In summary, I am not opposed to applying your proposal as is to the
> stable builds, there is no justification *ever* for breakage in
> stable
> branches.
> 
> For master, I only think thi

Re: Build sheriffs for GNOME

[Alberto Ruiz]

2016-01-22 16:55 GMT+00:00 Tristan Van Berkom :

> Of course, when you commit something and your *own* code does not
> compile, this is a very, very rare thing, rare enough to not be worth
> discussion I think, it that's happening, it's a problem.
>
> But it's common enough with many moving parts that the breakage is
> indeed intentional, in which case screwing with the history is only
> intrusive.
>

If you're breaking things intentionally, work on a wip/myfeature branch
until it doesn't break anymore? What's the big deal here?

If we want to keep the history clean  working on a wip branch that you can
rebase interactively squashing unmeaninful commits is what we should be
doing anyway.

-- 
Cheers,
Alberto Ruiz
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Fri, 2016-01-22 at 10:55 +0100, Milan Crha wrote:
>   Hi,
> I agree with Tristan. I was just about to write something along his
> lines. Seeing semi-automated reverts in the projects would be quite
> depressing, especially when the semi-automat has no idea about the
> project and what the change was meant for.

Why does it matter what the change was meant for? If people cannot
build GNOME, that is a much more serious problem than whatever your
commit is fixing.

> It makes sense to make sure the build works *before the release*, but
> when we are in the middle of two releases, then the build break might
> be just a red flag, not an excuse to break someone's work by the
> revert.

Let's take e-d-s as a specific example. If we were building tarball
releases of e-d-s in jhbuild and Continuous, this would make perfect
sense. We're not doing that. We build e-d-s from git master because it
is a core component of GNOME, and we want jhbuild to build the latest
version of everything, and we want contributors to be able to
contribute to e-d-s after all!, hence the need for a git repo. master
needs to be buildable at all times. This isn't for you, it's for
*everyone else* trying to build GNOME.

Now, you could choose to work around this if you want, by tagging e-d-s 
in Continuous and branching it in jhbuild, then updating that branch
with every release. Or, you could work in a "next" branch that gets
merged into master each release. Personally, I would continue to work
in master, and accept that once in a while a change of mine will get
reverted until I have figured out what is wrong with it

> I sometimes find it hard to plan some larger work due to too often
> releases (it's for a really large work, which can take several weeks
> to
> accomplish, then some more to fine tune in the real world usage). If
> you add this precedence, to always let someone ask: "oh, well, can I
> commit it? It's large, would anyone revert it? My change touches API
> and influences like 5+ projects, where I have only 3 under my direct
> control, thus the main commit is almost immediately followed by
> corresponding fixes in those projects I've under control, but some
> still can be broken", then I'm afraid the "hostile environment" would
> be a very nice word about GNOME and its infrastructure (people would
> use much much worse in the case you'd really make your proposal
> alive).

Milan, you have full commit access to every repo on git.gnome.org. Any
breakage that could register on Continuous is fully within your ability
to fix.

> This is also about different environments, you partly mentioned it. I
> believe most of the maintainers build before commit. I do, but I'm a
> human and mistakes happen. Some mistakes are independent of the build
> environment (I'm sorry for the before-Christmas breakage of the eds),
> some are not. I recall broken builds even in jhbuild when API changes
> happened (clean build was required to make it work, there was nothing
> wrong about the sources).

Yeah, we don't expect incremental builds to always work. There's just
no way to make that possible. We do expect clean builds to work.

> Even for example Ubuntu environment is different from that I use,
> causing undefined symbols on places which work just fine for me. What
> would you revert then in my project? I know, Continuous, it built an
> hour earlier, it should build now too.

The value of Continuous is that it is a neutral environment. I'd like
to set up additional jhbuild builders to do CI in Ubuntu, Fedora, Arch,
etc. so that we can catch build breakage in more environments.

> Anyway, doing any reverts might not be implicit, not without a
> previous
> discussion with the respective maintainers (unless they are not
> reachable), and only a day before the release - or on the Friday when
> the notification mail is sent, though for Weekend Contributors it
> would
> make more sense for Monday morning of the release day? I do not know,
> people can fix things just before the release too. It's because the
> GNOME should build from the release, not necessarily from the git
> snapshot.
> 
> The Continuous idea is great to discover build breaks early, but it's
> not an excuse to damage anyone's work.

Milan, I honestly feel that you might be better off using a "next"
branch if you really don't want any commits to be reverted. (That might
be the compromise solution here.) Or change jhbuild and gnome-
continuous to build your stable branch instead.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Fri, 2016-01-22 at 13:38 +, Emmanuele Bassi wrote:
> Continuous generates VM images that you can easily install inside
> Boxes, and keep them updated (they are based on OSTree and allow
> atomic updates).
> 
> The latest successfully built image is always at:
> http://build.gnome.org/continuous/buildmaster/images/z/current/

Would be awesome to advertise this on the build.gnome.org landing page,
so people can find it.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Shaun McCance]

On Fri, 2016-01-22 at 13:31 +, Emmanuele Bassi wrote:
> Hi;
> 

> > Continuous is a great first step (the response time is *stunningly*
> > good) and we're better off now that we have it, but it's not good
> > enough as it's not testing the build tool we use for development,
> > it's
> > not testing what we release.
> 
> If you're releasing something different than what you have in master
> then you're doing something impressively wrong.

Surely s/master/master or a stable release branch/ ?

--
Shaun

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Tristan Van Berkom]

On Fri, 2016-01-22 at 10:37 -0600, Michael Catanzaro wrote:
[...]
> I really don't think it's a big deal to have a few reverts in the git
> history. And anyway, reverts should be relatively rare, because build
> breakage should be relatively rare.
> 
> The master branch may be unstable, but that doesn't mean build
> breakage
> is ever acceptable. If your code is not buildable at all times in
> Continuous, keep it on a *side branch* -- otherwise, tag your module
> in
> Continuous, branch it in jhbuild, and do whatever you want in master.
> But as long as you have an official GNOME module, if it's not
> branched
> and tagged, master needs to build. This isn't the wild west.

Of course, when you commit something and your *own* code does not
compile, this is a very, very rare thing, rare enough to not be worth
discussion I think, it that's happening, it's a problem.

But it's common enough with many moving parts that the breakage is
indeed intentional, in which case screwing with the history is only
intrusive.

> 
> > My interpretation of your proposal is that you would want to
> > revert
> >     this api break or behavioral change if it were found to break
> > the
> >     build (or the tests) of another module, during this unstable
> >     development period.
> > 
> > This would obviously be wrong and would just be an obstacle to
> > progress in the cases where breakage is intentional and the
> > only
> >     correct solution is to have applications adapt to the new API
> > or
> >     behavior before the next stable release.
> 
> It's a big problem that some modules break for days or weeks at a
> time
> during API transitions; this makes it very difficult for new
> contributors to build and run GNOME, and it's inconvenient for
> experienced contributors as well.

This kind of breakage is just a fact of life, you cannot sweep it under
the rug and pretend it does not exist.

Sure, it's a barrier of entry for newcomers and an annoyance for
experienced developers who realize the facts of life. But that's just
the way it is, there is no reason to complain about it.

Software development is difficult, integration is difficult when you
have many moving parts that are in a development stage before being
release ready.

[...]
> To be clear: if you've branched and tagged your module, then you do
> whatever you want in master without hurting anyone else. But
> otherwise,
> master needs to be buildable.

And to be doubly clear: the master branch is *not* what is advertised
by upstream maintainers as stable, it is the bleeding edge, it can have
churn, it may not always match up to other modules perfectly, other
modules may have not yet adapted to the churn. Building master may be
asking for trouble, again, this is just a fact of life if you are
trying to integrate with all bleeding edge component in the middle of
their development cycle, instead of what maintainers have published as
stable.

If you want something that you are sure will build, take the stable
release tag and build that.

Best,
    -Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Fri, 2016-01-22 at 16:03 +0900, Tristan Van Berkom wrote:
>   o We are a volunteer driven project with contributors distributed
>     across timezones, who have dayjobs, it is ridiculous to expect
> that
>     maintainers can be reachable within a 3 hour turnaround period.
> 
> This leads me to believe that you fully expect to cause friction
> by
>     reverting commits before said maintainers have the time to even
>     respond, and doing so on the unstable master branch.
> 
> The master branch should ideally not be littered with reverts and
>     reverts of reverts, it is the main history of project development
> and adding such noise to master should be avoided as much as
>     possible for the purpose of preserving the integrity and value of
>     the history.

I really don't think it's a big deal to have a few reverts in the git
history. And anyway, reverts should be relatively rare, because build
breakage should be relatively rare.

The master branch may be unstable, but that doesn't mean build breakage
is ever acceptable. If your code is not buildable at all times in
Continuous, keep it on a *side branch* -- otherwise, tag your module in
Continuous, branch it in jhbuild, and do whatever you want in master.
But as long as you have an official GNOME module, if it's not branched
and tagged, master needs to build. This isn't the wild west.


> My interpretation of your proposal is that you would want to
> revert
>     this api break or behavioral change if it were found to break the
>     build (or the tests) of another module, during this unstable
>     development period.
> 
> This would obviously be wrong and would just be an obstacle to
> progress in the cases where breakage is intentional and the only
>     correct solution is to have applications adapt to the new API or
>     behavior before the next stable release.

It's a big problem that some modules break for days or weeks at a time
during API transitions; this makes it very difficult for new
contributors to build and run GNOME, and it's inconvenient for
experienced contributors as well. We need to land these transitions
better, by ensuring either that they occur in side branches, where all
modules' side branches are merged into master at the same time, or else
by tagging the relevant modules in Continuous and branching them in
jhbuild until the transition is completed.

To be clear: if you've branched and tagged your module, then you do
whatever you want in master without hurting anyone else. But otherwise,
master needs to be buildable.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

To be clear on what I'd like to see:

The most important requirement for our CI system is to run 'jhbuild
build' for each module with a clean install prefix, to find missing
dependencies in our moduleset. Not having to maintain jhbuild
modulesets separate from the Continuous manifest would be good too.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Emmanuele Bassi]

Hi;

On 22 January 2016 at 13:03, Simon McVittie
 wrote:
> On 21/01/16 14:54, Emmanuele Bassi wrote:
>> In short, I want to ensure that GNOME maintainers become
>> a bit more proactive in giving a crap about their modules breaking on
>> something that is not their own computers.
>
> Is there a way in which contributors/maintainers can reproduce something
> equivalent to what Continuous would build (or a large enough subset of
> the dependency stack to reach their packages), in a local virtual
> machine where they can debug issues interactively?

Continuous generates VM images that you can easily install inside
Boxes, and keep them updated (they are based on OSTree and allow
atomic updates).

The latest successfully built image is always at:
http://build.gnome.org/continuous/buildmaster/images/z/current/

On build failures you get the full log and environment of what was
built and how: 
http://build.gnome.org/continuous/buildmaster/builds/2016/01/22/37/build/

If you want to, you can also run gnome-continuous on your local
machine: https://git.gnome.org/browse/gnome-continuous/tree/README-build.md

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Emmanuele Bassi]

Hi Alberto;

you're asking for a try server, which is a fairly complex piece of machinery.

Mozilla has one, based on buildbot; it may be possible to have a
machine dedicated to try builds, but that's not what Continuous does,
and it would not be what Continuous aims to be.

Ciao,
 Emmanuele.


On 22 January 2016 at 13:13, Alberto Ruiz  wrote:
> Another thing that would be really useful in this line is to be able to
> submit specific branches for testing. Say I'm working on wip/foobar to avoid
> breaking master, and before pulling that stuff from master I want to make
> sure I won't break the builds.
>
> Is anything like this possible at all?
>
> 2016-01-22 13:03 GMT+00:00 Simon McVittie :
>>
>> On 21/01/16 14:54, Emmanuele Bassi wrote:
>> > In short, I want to ensure that GNOME maintainers become
>> > a bit more proactive in giving a crap about their modules breaking on
>> > something that is not their own computers.
>>
>> Is there a way in which contributors/maintainers can reproduce something
>> equivalent to what Continuous would build (or a large enough subset of
>> the dependency stack to reach their packages), in a local virtual
>> machine where they can debug issues interactively?
>>
>> In the autobuild systems that I've worked with, I've found that the key
>> feature that makes them most useful is to avoid "magic" centralized
>> infrastructure that only a few sysadmins/maintainers can access.
>>
>> For instance, one of the things I really like about Debian's autopkgtest
>> initiative (a package-oriented and potentially system-level equivalent
>> of GNOME installed-tests) is that it's quite well-documented how to get
>> a local mirror of the environment in which tests are run on the official
>> infrastructure[1] and how to get something basically equivalent as a
>> virtual machine image[2].
>>
>> [1] https://ci.debian.net/doc/file.MAINTAINERS.html
>> [2] http://manpages.debian.org/cgi-bin/man.cgi?query=adt-virt-qemu
>>
>> --
>> Simon McVittie
>> Collabora Ltd. 
>>
>> ___
>> desktop-devel-list mailing list
>> desktop-devel-list@gnome.org
>> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>
>
>
>
> --
> Cheers,
> Alberto Ruiz
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Emmanuele Bassi]

Hi;

On 21 January 2016 at 18:19, Michael Catanzaro  wrote:

> The next step is to convert either jhbuild or gnome-continuous to use
> the others' modulesets. Currently the jhbuild modulesets are the
> official "what we release,"

"What we release" and "what we actually run" are two fairly difference concepts.

I'd contend that the GNOME Continuous manifest is actually much more
close to what distributions ship, and thus what we run on a daily
basis, than what we put into the configure arguments when we
distcheck, or the arguments we put into the moduleset. Mostly, because
nobody, except the release team, smoketest a system composed by
jhbuild, whereas Continuous does smoketesting on a fully composed file
system image.

Continuous also builds the whole stack down to systemd, whereas
JHBuild does not.

JHBuild is meant to be assisted and attended by a human being;
Continuous is mostly meant for automation.

There's also the issue that the JHBuild modulesets cater to a fairly
different crowd, and have flexibility to build on OSX or Windows, with
conditionals and suggested packages.

We could make JHBuild read the Continuous manifest, and start from a
specific module, but the two systems are *not* interchangeable. Each
one has its own requirements and goals, and confusing the two is going
to leave use cases out.

We should certainly ensure that what goes into JHBuild also goes into
Continuous, and possibly with minimal fuss. Ideally, we should move
the JHBuild modulesets and the Continuous manifest out of their
respective repositories, and into a shared location.

> and they are more important because they
> affect the ability of new contributors to build GNOME, so that's what I
> focus on maintaining. Continuous is useful to catch many build
> failures, but it misses many, many more failures than could only be
> detected were it to use the official modulesets.

If Continuous fails to catch build failures then you should update the
manifest until you trigger them. That's the whole point of having
automated builds.

> Continuous is a great first step (the response time is *stunningly*
> good) and we're better off now that we have it, but it's not good
> enough as it's not testing the build tool we use for development, it's
> not testing what we release.

If you're releasing something different than what you have in master
then you're doing something impressively wrong.

You (and by "you" I mean "maintainers") should check what flags and
dependencies are listed in the JHBuild moduleset and which ones are in
the Continuous manifest, and try to make them close or identical.

Bugs will happen, and I consider keeping the Continuous manifest up to
date also part of the job description of a build sheriff.

>>  Many are not even on
>> #gnome-hackers, which means they miss the notification that the
>> something broke the build.
>
> Anecdote: I am on #gnome-hackers so you can ping me, but I don't turn
> on notifications for every message in the room (too many!),

The IRC bot does not notify you personally. You can add a notification
on a specific substring, like 'continuous:build * failed'; or you
could just look at the red message that says that the build failed.
:-)

> so it's
> rare that I notice a failure in one of my modules unless I happen to be
> looking at #gnome-hackers at the time, or someone else pings me. It
> would be really awesome if Continuous could learn to guess who to blame
> for breakage and include the IRC name in the channel;

As I said, it can be incredibly hard to determine that from an
automation standpoint.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Lasse Schuirmann]

One ideal end result IMO would be that some service similar to GNOME
continuous that validates any commit or patch ahead of time. If a
commit wasn't validated and somebody tried to push it to master, the
server should reject it. (GitLab does this implicitly with protected
branches and merge requests, GitHub lately got something similar too.)
I personally would love to have a bot that listens to bugzilla and
tests all patches automatically; also any pushed branch should be
tested automatically.

Testing code and buildability should be as simple as submitting or
pushing a patch.

2016-01-22 14:13 GMT+01:00 Alberto Ruiz :
> Another thing that would be really useful in this line is to be able to
> submit specific branches for testing. Say I'm working on wip/foobar to avoid
> breaking master, and before pulling that stuff from master I want to make
> sure I won't break the builds.
>
> Is anything like this possible at all?
>
> 2016-01-22 13:03 GMT+00:00 Simon McVittie :
>>
>> On 21/01/16 14:54, Emmanuele Bassi wrote:
>> > In short, I want to ensure that GNOME maintainers become
>> > a bit more proactive in giving a crap about their modules breaking on
>> > something that is not their own computers.
>>
>> Is there a way in which contributors/maintainers can reproduce something
>> equivalent to what Continuous would build (or a large enough subset of
>> the dependency stack to reach their packages), in a local virtual
>> machine where they can debug issues interactively?
>>
>> In the autobuild systems that I've worked with, I've found that the key
>> feature that makes them most useful is to avoid "magic" centralized
>> infrastructure that only a few sysadmins/maintainers can access.
>>
>> For instance, one of the things I really like about Debian's autopkgtest
>> initiative (a package-oriented and potentially system-level equivalent
>> of GNOME installed-tests) is that it's quite well-documented how to get
>> a local mirror of the environment in which tests are run on the official
>> infrastructure[1] and how to get something basically equivalent as a
>> virtual machine image[2].
>>
>> [1] https://ci.debian.net/doc/file.MAINTAINERS.html
>> [2] http://manpages.debian.org/cgi-bin/man.cgi?query=adt-virt-qemu
>>
>> --
>> Simon McVittie
>> Collabora Ltd. 
>>
>> ___
>> desktop-devel-list mailing list
>> desktop-devel-list@gnome.org
>> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>
>
>
>
> --
> Cheers,
> Alberto Ruiz
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Alberto Ruiz]

Another thing that would be really useful in this line is to be able to
submit specific branches for testing. Say I'm working on wip/foobar to
avoid breaking master, and before pulling that stuff from master I want to
make sure I won't break the builds.

Is anything like this possible at all?

2016-01-22 13:03 GMT+00:00 Simon McVittie :

> On 21/01/16 14:54, Emmanuele Bassi wrote:
> > In short, I want to ensure that GNOME maintainers become
> > a bit more proactive in giving a crap about their modules breaking on
> > something that is not their own computers.
>
> Is there a way in which contributors/maintainers can reproduce something
> equivalent to what Continuous would build (or a large enough subset of
> the dependency stack to reach their packages), in a local virtual
> machine where they can debug issues interactively?
>
> In the autobuild systems that I've worked with, I've found that the key
> feature that makes them most useful is to avoid "magic" centralized
> infrastructure that only a few sysadmins/maintainers can access.
>
> For instance, one of the things I really like about Debian's autopkgtest
> initiative (a package-oriented and potentially system-level equivalent
> of GNOME installed-tests) is that it's quite well-documented how to get
> a local mirror of the environment in which tests are run on the official
> infrastructure[1] and how to get something basically equivalent as a
> virtual machine image[2].
>
> [1] https://ci.debian.net/doc/file.MAINTAINERS.html
> [2] http://manpages.debian.org/cgi-bin/man.cgi?query=adt-virt-qemu
>
> --
> Simon McVittie
> Collabora Ltd. 
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>



-- 
Cheers,
Alberto Ruiz
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Simon McVittie]

On 21/01/16 14:54, Emmanuele Bassi wrote:
> In short, I want to ensure that GNOME maintainers become
> a bit more proactive in giving a crap about their modules breaking on
> something that is not their own computers.

Is there a way in which contributors/maintainers can reproduce something
equivalent to what Continuous would build (or a large enough subset of
the dependency stack to reach their packages), in a local virtual
machine where they can debug issues interactively?

In the autobuild systems that I've worked with, I've found that the key
feature that makes them most useful is to avoid "magic" centralized
infrastructure that only a few sysadmins/maintainers can access.

For instance, one of the things I really like about Debian's autopkgtest
initiative (a package-oriented and potentially system-level equivalent
of GNOME installed-tests) is that it's quite well-documented how to get
a local mirror of the environment in which tests are run on the official
infrastructure[1] and how to get something basically equivalent as a
virtual machine image[2].

[1] https://ci.debian.net/doc/file.MAINTAINERS.html
[2] http://manpages.debian.org/cgi-bin/man.cgi?query=adt-virt-qemu

-- 
Simon McVittie
Collabora Ltd. 

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Milan Crha]

On Fri, 2016-01-22 at 16:03 +0900, Tristan Van Berkom wrote:
> In summary, I am not opposed to applying your proposal as is to the
> stable builds, there is no justification *ever* for breakage in stable
> branches.
> 
> For master, I only think this needs to be detailed properly, perhaps it
> would be enough to ensure we had policy ensuring that intentional
> breakage is announced (on this list ?) and that "sheriffs" are
> responsible for following this list and not reverting commits which
> break things intentionally in a transitional period.

Hi,
I agree with Tristan. I was just about to write something along his
lines. Seeing semi-automated reverts in the projects would be quite
depressing, especially when the semi-automat has no idea about the
project and what the change was meant for.

It makes sense to make sure the build works *before the release*, but
when we are in the middle of two releases, then the build break might
be just a red flag, not an excuse to break someone's work by the
revert.

I sometimes find it hard to plan some larger work due to too often
releases (it's for a really large work, which can take several weeks to
accomplish, then some more to fine tune in the real world usage). If
you add this precedence, to always let someone ask: "oh, well, can I
commit it? It's large, would anyone revert it? My change touches API
and influences like 5+ projects, where I have only 3 under my direct
control, thus the main commit is almost immediately followed by
corresponding fixes in those projects I've under control, but some
still can be broken", then I'm afraid the "hostile environment" would
be a very nice word about GNOME and its infrastructure (people would
use much much worse in the case you'd really make your proposal alive).

This is also about different environments, you partly mentioned it. I
believe most of the maintainers build before commit. I do, but I'm a
human and mistakes happen. Some mistakes are independent of the build
environment (I'm sorry for the before-Christmas breakage of the eds),
some are not. I recall broken builds even in jhbuild when API changes
happened (clean build was required to make it work, there was nothing
wrong about the sources).

Even for example Ubuntu environment is different from that I use,
causing undefined symbols on places which work just fine for me. What
would you revert then in my project? I know, Continuous, it built an
hour earlier, it should build now too.

Anyway, doing any reverts might not be implicit, not without a previous
discussion with the respective maintainers (unless they are not
reachable), and only a day before the release - or on the Friday when
the notification mail is sent, though for Weekend Contributors it would
make more sense for Monday morning of the release day? I do not know,
people can fix things just before the release too. It's because the
GNOME should build from the release, not necessarily from the git
snapshot.

The Continuous idea is great to discover build breaks early, but it's
not an excuse to damage anyone's work.
Bye,
Milan
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Tristan Van Berkom]

On Thu, 2016-01-21 at 14:54 +, Emmanuele Bassi wrote:
[...]
> This is not enough, and it does not raise the bar in keeping GNOME in
> a buildable state. It actually lowers it a fair , to the effective
> point that *nobody* cares about Continuous builds.
> 
> I want this to change. I want to be able to revert failing commits on
> the offending modules, if they are hosted on GNOME infrastructure, if
> they fail for more than N hours, and *then* open a bug about it.

Hi Emmanuele,

In GNOME, we have got along well for many years with many projects
sharing the same infrastructure mostly because we find that in
practice, people are generally respectful enough of eachothers work to
recognize a module maintainer's word as scripture. As such we do not
commit changes to others modules without expressed permission unless it
is for a minor change that we are absolutely sure about. There is/was a
page describing this policy to those who receive commit access
explaining the responsibility that comes with being a committer, it has
either gone missing or I cannot currently find the text.

In that context, I admit that I first interpreted your proposal as
hostile, at least in it's current form, because it violates this policy
which has allowed our projects to exist in harmony for so many years
under the GNOME umbrella, by awarding some kind of revert power to an
arbitrary group of individuals. I can see much opportunity here for
unnecessary dispute, if special care is not taken, we may risk
incentivizing maintainers to host their projects elsewhere.

On the other hand, it is desirable to have things building regularly,
of course the master branch is inherently unstable - that's what it's
for, but I would however support your proposal without hesitation were
it to be applied to stable branches only.

Were we to apply your proposed policy to master, I think we need
further thought and clarification before carrying that out:

  o We are a volunteer driven project with contributors distributed
    across timezones, who have dayjobs, it is ridiculous to expect that
    maintainers can be reachable within a 3 hour turnaround period.

This leads me to believe that you fully expect to cause friction by
    reverting commits before said maintainers have the time to even
    respond, and doing so on the unstable master branch.

The master branch should ideally not be littered with reverts and
    reverts of reverts, it is the main history of project development
and adding such noise to master should be avoided as much as
    possible for the purpose of preserving the integrity and value of
    the history.

I think this is unfriendly at best and I would prescribe a one week
period for master. We should really wait for the maintainers word
before intruding and reverting commits which introduce breakages
    which may have even been intentional.


  o Not all libraries advertize API/ABI stability, especially when it
    comes to libraries whos only consumers are GNOME applications, we
    allow ourselves much more leniency here than with platform
    libraries which are also relevant outside of the GNOME ecosystem.

During the course of unstable development, it is entirely common
for a library to introduce a new API and then to later change it,
even more than once before it is ready for a stable release.
For instance, EDS's backend facing APIs are not stable, and there
have been regular issues in the vala API it exposes and libfolks
consumes.

    There are periods where it must be accepted that things are just
    broken in a transitional phase.

My interpretation of your proposal is that you would want to revert
    this api break or behavioral change if it were found to break the
    build (or the tests) of another module, during this unstable
    development period.

This would obviously be wrong and would just be an obstacle to
progress in the cases where breakage is intentional and the only
    correct solution is to have applications adapt to the new API or
    behavior before the next stable release.

In summary, I am not opposed to applying your proposal as is to the
stable builds, there is no justification *ever* for breakage in stable
branches.

For master, I only think this needs to be detailed properly, perhaps it
would be enough to ensure we had policy ensuring that intentional
breakage is announced (on this list ?) and that "sheriffs" are
responsible for following this list and not reverting commits which
break things intentionally in a transitional period.

Regards,
    -Tristan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

On Thu, 2016-01-21 at 12:19 -0600, Michael Catanzaro wrote:
> Thank you Emmanuele, this will really help with keeping GNOME
> building.
> I strongly support this change.

Also, to clarify, this surely means that maintainers are responsible
for updating the Continuous manifest in addition to JHBuild modulesets
when adding new dependencies.

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Michael Catanzaro]

Thank you Emmanuele, this will really help with keeping GNOME building.
I strongly support this change.

On Thu, 2016-01-21 at 14:54 +, Emmanuele Bassi wrote:
> This effort led to various benefits, including JHBuild not constantly
> being in a broken state, and most projects hosted on git.gnome.org
> finally building when builddir != srcdir, which should improve the
> ability of maintainers to do `make distcheck` on release days.

The next step is to convert either jhbuild or gnome-continuous to use
the others' modulesets. Currently the jhbuild modulesets are the
official "what we release," and they are more important because they
affect the ability of new contributors to build GNOME, so that's what I
focus on maintaining. Continuous is useful to catch many build
failures, but it misses many, many more failures than could only be
detected were it to use the official modulesets.

Continuous is a great first step (the response time is *stunningly*
good) and we're better off now that we have it, but it's not good
enough as it's not testing the build tool we use for development, it's
not testing what we release. We need to test that each module builds
with 'jhbuild build' with a completely empty install prefix, else
there's no way to find and fix the bajillions of dependency errors in
our modulesets. This is why new contributors say it is so hard to use
jhbuild, so hard to get started.

> What we need now, though, are "build sheriffs", i.e. people that
> watch
> the build process and ensure that it continues smoothly. Ideally,
> every maintainer would be on IRC, and would join the #testable
> channel; sadly, that's not the case.

Anecdote: Because Empathy uses a crummy notebook for tabs, opening more
rooms makes it much harder to switch between rooms and see which
important once have changes, hence I minimize the number of rooms I
join, hence I'm not going to join #testable as I am already in too many
rooms. I bet I'm not the only one. If anyone wants to fix Empathy to
have a saner view of rooms -- to use a sidebar like Polari, for example
-- I would be willing to join way more rooms.

>  Many are not even on
> #gnome-hackers, which means they miss the notification that the
> something broke the build.

Anecdote: I am on #gnome-hackers so you can ping me, but I don't turn
on notifications for every message in the room (too many!), so it's
rare that I notice a failure in one of my modules unless I happen to be
looking at #gnome-hackers at the time, or someone else pings me. It
would be really awesome if Continuous could learn to guess who to blame
for breakage and include the IRC name in the channel; I know it's hard
to decide who *actually* broke the build considering it could be due to
a change in a dependency, but even a dumb guess like the people in the
doap file for the broken project would be better than nothing, so at
least the maintainer knows and can investigate. (This would also be
incentive for past maintainers to remove their names from the doap
file. :)

It's not necessary -- Bugzilla works just as well, if slower -- but an
enhancement to consider if someone has time to spare to implement
it

>   - if nothing happens for a reasonable amount of time (I'd give one
> to three hours)

I would support even less time: five minutes (if the maintainer cannot
be immediately contacted on IRC). Build breakage is a tremendous
obstacle to new contributors. In contrast, once I've found the problem
with a commit that got reverted, un-reverting it requires almost zero
effort. So if I'm not immediately available to look at my broken
commit, just revert it... no worries.


Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Build sheriffs for GNOME

[Emmanuele Bassi]

Hi;

On 21 January 2016 at 15:22, SAHIL SAREEN  wrote:

> I feel we should spend time building a simple "build sheriff" (maybe an 
> extension of https://build.gnome.org) that auto-files a "cri" bug for the 
> module whose build failed. This should make sure that it isn't creating 
> duplicate bugs(reopening the old one if exists) for the same module and 
> append a comment giving the link to the respective build failure.
>
> The package maintainers(or the #testable s) should then either revert the bad 
> commits or fix them and close the bug.
>
> Thoughts?

There are two issues with that, and they cannot be solved via automation:

  1. since Continuous does not do reverse dependencies when building,
the component that may be breaking the build may not be the actual
component that needs a tag or a revert. If, say, component X
introduces an API change, then component X will likely build fine, but
every other component that depends on component X and uses that API
will fail to build. We cannot send an email/notify on IRC/open a bug
for other components.

  2. the goal is not to notify maintainers; the notification is
actually an anti-goal. The goal, here, is to ensure that the maximum
time for a build breakage is around an hour. The goal is to ensure
that the *whole* of GNOME *always* builds at any given time of day.

We notify maintainers today, by filing bugs; by joining IRC channels;
by sending emails. That can take ages to get a response because
maintainers are humans, and have other things to do. I filed a bug and
tagged a component of the manifest because of a broken build (caused
by one of its dependencies) mid-December, and the maintainer was on
vacation. This means that the components in question were not built
for three weeks because of the tag. A revert would have avoided the
tag, and kept the world building.

Bugzilla can deal with duplicate bugs; the amount of effort spent
there is negligible.

Ciao,
 Emmanuele.

> From: desktop-devel-list  on behalf of 
> Emmanuele Bassi 
> Sent: Thursday, January 21, 2016 8:24 PM
> To: Desktop Development List
> Subject: Build sheriffs for GNOME
>
> Hi all;
>
> Many of you know about GNOME Continuous, and build.gnome.org — and for
> those who don't, here's two handy links:
>
>   - https://build.gnome.org
>   - https://wiki.gnome.org/Projects/GnomeContinuous
>
> In short: we're currently building the core GNOME and some
> applications every time something gets committed to git.gnome.org;
> we're also doing various tests — like smoketesting the session and
> running applications — and building VM images out of the build
> process.
>
> This effort led to various benefits, including JHBuild not constantly
> being in a broken state, and most projects hosted on git.gnome.org
> finally building when builddir != srcdir, which should improve the
> ability of maintainers to do `make distcheck` on release days.
>
> What we need now, though, are "build sheriffs", i.e. people that watch
> the build process and ensure that it continues smoothly. Ideally,
> every maintainer would be on IRC, and would join the #testable
> channel; sadly, that's not the case. Many are not even on
> #gnome-hackers, which means they miss the notification that the
> something broke the build. We cannot always send emails to the last
> committer of a broken module because GNOME is a complex project, and a
> change in a dependency may indeed break your project, even if you
> didn't know about it.
>
> For the past few months a few people, myself included, have been
> hunting down build breakages; if we exclude the issues with the build
> machine itself throwing a fit — something that usually gets fixed by
> Colin kicking it — the vast majority of build breakages come from
> issues inside GNOME projects.
>
> What usually happens when a build goes into perma-red (i.e. it keeps
> failing over the same component) is that somebody on the #testable IRC
> (usually me or Colin Walters) tags the module inside the manifest,
> opens a bug, and hopes that a fix get applied and  on the channel so
> that the tag gets reverted.
>
> This is not enough, and it does not raise the bar in keeping GNOME in
> a buildable state. It actually lowers it a fair , to the effective
> point that *nobody* cares about Continuous builds.
>
> I want this to change. I want to be able to revert failing commits on
> the offending modules, if they are hosted on GNOME infrastructure, if
> they fail for more than N hours, and *then* open a bug about it.
> Ideally, I want to tag only modules that are *not* hosted on GNOME
> infrastructure, as they are beyond our control and commit
> capabilities. In short, I want to ensure that GNOME maintainers become
> a bit more proactive in giving a crap about their modules breaking on
> something that is not their own computers.
>
> So, here's what will happen in the near future in case of a build break:
>
>   - a build sheriff will identify the component that is actually
> breaking; this usually take

Re: Build sheriffs for GNOME

[SAHIL SAREEN]

Re: Build sheriffs for GNOME

SAHIL SAREEN
Reply|
To:
Emmanuele Bassi ; Thu 1/21/2016 8:47 PM
Hey ebassi

I feel we should spend time building a simple "build sheriff" (maybe an 
extension of https://build.gnome.org) that auto-files a "cri" bug for the 
module whose build failed. This should make sure that it isn't creating 
duplicate bugs(reopening the old one if exists) for the same module and append 
a comment giving the link to the respective build failure.

The package maintainers(or the #testable s) should then either revert the bad 
commits or fix them and close the bug.

Thoughts?

-Sahil


From: desktop-devel-list  on behalf of 
Emmanuele Bassi 
Sent: Thursday, January 21, 2016 8:24 PM
To: Desktop Development List
Subject: Build sheriffs for GNOME

Hi all;

Many of you know about GNOME Continuous, and build.gnome.org — and for
those who don't, here's two handy links:

  - https://build.gnome.org
  - https://wiki.gnome.org/Projects/GnomeContinuous

In short: we're currently building the core GNOME and some
applications every time something gets committed to git.gnome.org;
we're also doing various tests — like smoketesting the session and
running applications — and building VM images out of the build
process.

This effort led to various benefits, including JHBuild not constantly
being in a broken state, and most projects hosted on git.gnome.org
finally building when builddir != srcdir, which should improve the
ability of maintainers to do `make distcheck` on release days.

What we need now, though, are "build sheriffs", i.e. people that watch
the build process and ensure that it continues smoothly. Ideally,
every maintainer would be on IRC, and would join the #testable
channel; sadly, that's not the case. Many are not even on
#gnome-hackers, which means they miss the notification that the
something broke the build. We cannot always send emails to the last
committer of a broken module because GNOME is a complex project, and a
change in a dependency may indeed break your project, even if you
didn't know about it.

For the past few months a few people, myself included, have been
hunting down build breakages; if we exclude the issues with the build
machine itself throwing a fit — something that usually gets fixed by
Colin kicking it — the vast majority of build breakages come from
issues inside GNOME projects.

What usually happens when a build goes into perma-red (i.e. it keeps
failing over the same component) is that somebody on the #testable IRC
(usually me or Colin Walters) tags the module inside the manifest,
opens a bug, and hopes that a fix get applied and  on the channel so
that the tag gets reverted.

This is not enough, and it does not raise the bar in keeping GNOME in
a buildable state. It actually lowers it a fair , to the effective
point that *nobody* cares about Continuous builds.

I want this to change. I want to be able to revert failing commits on
the offending modules, if they are hosted on GNOME infrastructure, if
they fail for more than N hours, and *then* open a bug about it.
Ideally, I want to tag only modules that are *not* hosted on GNOME
infrastructure, as they are beyond our control and commit
capabilities. In short, I want to ensure that GNOME maintainers become
a bit more proactive in giving a crap about their modules breaking on
something that is not their own computers.

So, here's what will happen in the near future in case of a build break:

  - a build sheriff will identify the component that is actually
breaking; this usually takes about a minute of reading the build log
linked by the IRC bot
  - the build sheriff will try to reach the maintainer of the
component and see if they are on IRC or any other form of IM; I
strongly suggest you either use the same nickname as your GNOME user
id in the DOAP file of your projects, or we'll need to figure out a
way to link your IRC nickname with the project description
  - if nothing happens for a reasonable amount of time (I'd give one
to three hours)
- if the offending component is not hosted on git.gnome.org, the
build sheriff will tag the component in the gnome-continuous manifest
file
- if the offending component is hosted on git.gnome.org, the build
sheriff will revert the commits that caused the breakage
  - in either case, a bug will be filed in the bug tracker for the
component, if one is not already open (that's why you should always
list bugs into the commit message)

What does a build sheriff need to know? Usually, breakages involve the
build system. If you know autotools and/or CMake enough to fix issues
with them, you should consider joining. The wiki page on GNOME
Continuous describes the system — so you should take some time to
familiarise yourself with it:

https://git.gnome.org/browse/gnome-continuous

Especially the manifest file.

The process of keeping GNOME building will b