Re: Announcement/RFC: jhbuild continuous integration testing
Hello again,
first, thanks for everyone who jumped in and helped to fix failures! I
wanted to send a quick update.
Martin Pitt [2013-02-12 7:43 +0100]:
> - colord: recently started depending on libsystemd-login, which we
>don't have yet; that's a fault on the Ubuntu side
I installed the libsystemd-login libs and pushed a fix for a missing
library link to trunk. Working now.
> - e-d-s: calls an undeclared g_cond_timed_wait(), not sure what this
>is about
I filed a bug, Matthew Barnes quickly fixed it in trunk. After that it
was working for two runs. (Not any more, see below)
> - folks: this started failing very recently, and thus is a perfect
>example why this is useful (unqualified ambiguous usage of
>HashTable)
Philip quickly fixed that one, working now.
> - gst-plugins-bad: unknown type GStaticRecMutex; this might be due to
>recent changes in streamer? That smells like a case of "broken by
>change in dependency, needs updating to new API"
Still outstanding.
> - mutter: worked until Jan 7, now failing on unknown XIBarrierEvent;
>that might be a fault in Ubuntu's X.org packages or upstream, I
>haven't investigated this yet
Jasper pointed out the problem in Ubuntu'x libxi packages, I'll take
care of this.
> Test failures:
> - gst-plugins-good, empathy: one test failure, the other tests work
gst-plugins-good got fixed by Tim, empathy still outstanding.
> - realmd: This looks like the test suite is making some assumptions
>about the environment which aren't true in a headless server?
> - webkit: I don't actually see an error in the log; we'll investigate
>this closer on our side
Still outstanding.
So yesterday evening we were down to 5 failures, but over night we got
a swath of new test failures. Wading through them now, but some look
quite nonobvious.
nautilus failed on
evaluated: nautilus_file_get_name (file_1)
expected: eazel:///
got: eazel:
It failed on exactly that until January 5, then was working until
yesterday, and now failing again. For such issues I'd be glad if one
of the nautilus maintainers could work this out with me.
libgdata once again failed on the youtube test, as it did in the past.
In general, any test that tries to access remote servers is pretty
much doomed on that machine I'm afraid, as it can only do http and
https through a proxy (it has $http{,s}_proxy set).
I'll look at the other failures now.
Martin
--
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
signature.asc
Description: Digital signature
___
release-team@gnome.org
https://mail.gnome.org/mailman/listinfo/release-team
Release-team lurker? Do NOT participate in discussions.
Re: Announcement/RFC: jhbuild continuous integration testing
Hello Tristan, Tristan Van Berkom [2013-02-14 6:42 +0900]: > Upon reading this particular part (and I noticed before you are > using mostly jhbuild mechanics), it leads me to wonder, how > granular exactly are these rebuilds ? Right now, every 15 minutes. Sometimes longer, when the previous run is still running. > I think ideally it would be great if builds could be triggered by > commit. In other words, commits are serialized chronologically and > each and every commit should trigger an entire rebuild, each rebuild > should build everything in the moduleset "up to the latest commit"... > separately, one after the other. That is indeed the long-term plan, but there's still some work to be done before we can do that. The machine we are running this on has 64 2.7 GHz cores and 64 GB of RAM, that really isn't a bottleneck right now. The main two problems right now are that the "jhbuild update" stage takes some 5 minutes to update all the ~ 160 git trees, and that jhbuild build doesn't parallelize at all, i. e. build modules which don't depend on each other could build in parallel. Once we solve both, and we dramatically reduce the time of one run from "several hours" (which is currently needed if e. g. a glib change happens, which rebuilds pretty much everything) to < 15 minutes. > The way I imagine this works now (and this is a big assumption, > correct me if I'm wrong), is that a commit in a given module triggers > a jhbuild build, which would mean that: > >a.) Several commits could have been made in a given module > by the time jhbuild actually runs... meaning we dont know > which of the given commits in that lapse of time actually > caused the fault. That's right. It's massively better to know that one commit in these 15 minutes triggered it than "something in the past week", but still not perfect as you say. >b.) Module "foo" triggers a rebuild... and while jhbuild builds, > it also pulls in new changes from module "bar", in this > case it's possible that a recent commit in module "bar" > caused another module "baz" to be effected, but in the > end it's module "foo" who is blamed (since module "foo" > essentially /triggered a rebuild/) That's a trickier thing. For most commits, one should actually be able to build them independently, but sometimes those "in between" breaks are inevitable. Say, you make an API change in a library and then update your application to the new API, then in between you will get a build failure. The next iteration should fix it again. We have that problem independently of the frequency we build stuff of course, as we can always hit a "bad time". Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ release-team@gnome.org https://mail.gnome.org/mailman/listinfo/release-team Release-team lurker? Do NOT participate in discussions.
Re: Announcement/RFC: jhbuild continuous integration testing
First, this sounds like really interesting stuff, great news. On Tue, Feb 12, 2013 at 3:43 PM, Martin Pitt wrote: > Hello fellow GNOME developers, > > this already came up as a side issue recently[1], but now we are at a > point where have reasonably stabilized our GNOME jhbuild continuous > builds/integration test server to become actually useful: > > https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/ > > This is building gnome-suites-core-3.8.modules, which currently > consists of 160 modules. Builds are updated every 15 minutes, and > triggered whenever there was a new commit in a module or any of its > dependencies. This mostly uses the smarts of jhbuild, we just have > some extra scripts around to pick the results apart for Jenkins and > drive the whole thing [2]. You can click through all the modules, all > their builds, and get their build logs. > > Right now there are 151 successes (blue), 5 modules fail to build > (red), and 4 modules build but fail in "make check" (yellow). It's > been like that for a week or two now, so I'd say we are doing > reasonably well for now. Some details: > > Build failures: > - colord: recently started depending on libsystemd-login, which we >don't have yet; that's a fault on the Ubuntu side > - e-d-s: calls an undeclared g_cond_timed_wait(), not sure what this >is about > - folks: this started failing very recently, and thus is a perfect >example why this is useful (unqualified ambiguous usage of >HashTable) > - gst-plugins-bad: unknown type GStaticRecMutex; this might be due to >recent changes in streamer? That smells like a case of "broken by >change in dependency, needs updating to new API" > - mutter: worked until Jan 7, now failing on unknown XIBarrierEvent; >that might be a fault in Ubuntu's X.org packages or upstream, I >haven't investigated this yet > > Test failures: > - gst-plugins-good, empathy: one test failure, the other tests work > - realmd: This looks like the test suite is making some assumptions >about the environment which aren't true in a headless server? > - webkit: I don't actually see an error in the log; we'll investigate >this closer on our side > > This was set up by Jean-Baptiste Lallement, I mostly help out with > reviewing the daily status and cleaning up after some build/test > failures which are due to broken checkouts, stale files, new missing > build dependencies, and so on. It's reasonably maintenance intensive, > but that's something which the two of us are willing to do if this > actually gets used. > > The main difference to Colin's ostree builds is that this also runs > "make check", which is one of the main points of this: We want to know > as soon as possible if e. g. a new commit in glib breaks something in > gvfs or evolution-data-server. Where "soon" is measured in minutes > instead of days/weeks, so that the knowledge what got changed and why > is still fresh in the developer's head. That's also why I recently > started to add integration tests to e. g. gvfs or > gnome-settings-daemon, so that over time we can cover more and more > functionality tests in these. > > To make this really useful, we can't rely on developers checking this > every hour or every day, of course; instead we need push notifications > as soon as a module starts failing. That's the bit which needs broader > discussion and consent. > > I see some obvious options here what to do when the status of a module > (OK/fails tests/fails build) changes: > > (1) mail the individual maintainers, as in the DOAP files >(1a) do it for everyone, and let people who don't want this filter >them out on a particular mail header (like "X-GNOME-QA:") >(1b) do this as opt-in > >This most often reaches the people who can do something about the >failure. Of course there are cases where it's not the module's fault, but a >dependency changed/got broken. There is no way we can automatically >determine whether it was e. g. a deliberate API break which modules >need to adjust to, or indeed a bug in the depending library, so we >might actually need to mail both the maintainers of the module that >triggered the rebuild, and the maintainers of the module which now >broke. Upon reading this particular part (and I noticed before you are using mostly jhbuild mechanics), it leads me to wonder, how granular exactly are these rebuilds ? I think ideally it would be great if builds could be triggered by commit. In other words, commits are serialized chronologically and each and every commit should trigger an entire rebuild, each rebuild should build everything in the moduleset "up to the latest commit"... separately, one after the other. I know, it sounds like some CPU will be melting quickly at the rate gnome-wide commits are made... but it would be simply awesome, if we could automatically pull out the exact commit which introduced exactly which failed build report in which module
Re: Announcement/RFC: jhbuild continuous integration testing
On Wed, Feb 13, 2013 at 5:23 AM, Bastien Nocera wrote: > If you file the bugs as soon as they're visible, you'll end up filing > outdated bugs, and severely reducing the good will of the people fixing > those bugs. > > I don't think it would personally take me very long to yell "I KNOW" at > the bugmail and want to opt-out. > > Bugmail should be for long-standing issues. If the problem can be solved > under 10/15 minutes, don't start nagging people watching the bug mail. Indeed, I've cleaned out plenty of 5-10 year old build breakage bugs from gtk bugzilla in the last week. Having failures from this system show up in #testing would be brilliant, on the other hand. ___ release-team@gnome.org https://mail.gnome.org/mailman/listinfo/release-team Release-team lurker? Do NOT participate in discussions.
Re: Announcement/RFC: jhbuild continuous integration testing
On Wed, 2013-02-13 at 10:59 +0100, Olav Vitters wrote: > On Wed, Feb 13, 2013 at 06:53:17AM +0100, Martin Pitt wrote: > > Travis Reitter [2013-02-12 13:21 -0800]: > > > On Tue, 2013-02-12 at 07:43 +0100, Martin Pitt wrote: > > > > To make this really useful, we can't rely on developers checking this > > > > every hour or every day, of course; instead we need push notifications > > > > as soon as a module starts failing. That's the bit which needs broader > > > > discussion and consent. > > > > > > I'd like to offer: > > > > > > (0) auto-file an urgent/critical bug against the module in the case of > > > build breaks (and maybe high/major for test breaks?) > > > > Claudio Saavedra [2013-02-12 23:24 +0200]: > > > (3) Automatically filing a bug in the broken module with the details of > > > the breakage and additionally CC: whoever might be interested in keeping > > > an eye on the continuous integration? > > > > This creates more overhead, but I like this proposal as well. I guess > > one can use python-bzutils and the like to auto-create bugs, and > > remember on the Jenkins side which bug was opened for which module, > > and auto-close it once the build works again. > > > > The main issue that I see with this is that it's much harder to filter > > away/opt out, so it requires some broader consensus that we want to do > > this. We can still add a module blacklist if needed, though. > > I think build issues should be filed as Bugzilla bugs. At most we maybe > want to set some keyword / status whiteboard. But I guess the summary > would be consistent and people will quickly learn of it. They should be filed as Bugzilla bugs *after a timeout*. We already see build failures on ostree popping up on the #testable channel, and those are usually fixed in a timely manner. If you file the bugs as soon as they're visible, you'll end up filing outdated bugs, and severely reducing the good will of the people fixing those bugs. I don't think it would personally take me very long to yell "I KNOW" at the bugmail and want to opt-out. Bugmail should be for long-standing issues. If the problem can be solved under 10/15 minutes, don't start nagging people watching the bug mail. ___ release-team@gnome.org https://mail.gnome.org/mailman/listinfo/release-team Release-team lurker? Do NOT participate in discussions.
Re: Announcement/RFC: jhbuild continuous integration testing
On Wed, Feb 13, 2013 at 11:23:52AM +0100, Bastien Nocera wrote: > On Wed, 2013-02-13 at 10:59 +0100, Olav Vitters wrote: > > On Wed, Feb 13, 2013 at 06:53:17AM +0100, Martin Pitt wrote: > > > The main issue that I see with this is that it's much harder to filter > > > away/opt out, so it requires some broader consensus that we want to do > > > this. We can still add a module blacklist if needed, though. > > > > I think build issues should be filed as Bugzilla bugs. At most we maybe > > want to set some keyword / status whiteboard. But I guess the summary > > would be consistent and people will quickly learn of it. > > They should be filed as Bugzilla bugs *after a timeout*. We already see > build failures on ostree popping up on the #testable channel, and those > are usually fixed in a timely manner. Hmm, makes sense. Maybe start of small? -- Regards, Olav ___ release-team@gnome.org https://mail.gnome.org/mailman/listinfo/release-team Release-team lurker? Do NOT participate in discussions.
Re: Announcement/RFC: jhbuild continuous integration testing
On Wed, Feb 13, 2013 at 06:53:17AM +0100, Martin Pitt wrote: > Travis Reitter [2013-02-12 13:21 -0800]: > > On Tue, 2013-02-12 at 07:43 +0100, Martin Pitt wrote: > > > To make this really useful, we can't rely on developers checking this > > > every hour or every day, of course; instead we need push notifications > > > as soon as a module starts failing. That's the bit which needs broader > > > discussion and consent. > > > > I'd like to offer: > > > > (0) auto-file an urgent/critical bug against the module in the case of > > build breaks (and maybe high/major for test breaks?) > > Claudio Saavedra [2013-02-12 23:24 +0200]: > > (3) Automatically filing a bug in the broken module with the details of > > the breakage and additionally CC: whoever might be interested in keeping > > an eye on the continuous integration? > > This creates more overhead, but I like this proposal as well. I guess > one can use python-bzutils and the like to auto-create bugs, and > remember on the Jenkins side which bug was opened for which module, > and auto-close it once the build works again. > > The main issue that I see with this is that it's much harder to filter > away/opt out, so it requires some broader consensus that we want to do > this. We can still add a module blacklist if needed, though. I think build issues should be filed as Bugzilla bugs. At most we maybe want to set some keyword / status whiteboard. But I guess the summary would be consistent and people will quickly learn of it. You can use information from DOAP files to figure out the Bugzilla product. This is not always needed. Then commit whatever DOAP fixes are needed (the DOAP files are not bug free :P. In case someone wants to opt out, we can add a new gnome-specific option to the DOAP file specifying that any Continuous Integration is not welcome. What I do in ftpadmin (see sysadmin-bin) is: - Check if there is a bug-database with 'bugzilla.gnome.org' For those URL(s): Check if the URL containers product=$SOMETHING - If there is a good product: use that - else: assume tarball = bugzilla product (will fail for jhbuild, often modules are renamed and you don't know the real one) Note: Not all products are on bugzilla.gnome.org. Maybe after GNOME bugzilla, strive for 'bugs.freedesktop.org'? For monitoring these bugs, you can easily watch people in Bugzilla. -- Regards, Olav ___ release-team@gnome.org https://mail.gnome.org/mailman/listinfo/release-team Release-team lurker? Do NOT participate in discussions.
