Re: Broken master and newcomers.
This is Interesting - http://www.3d-architectural-rendering.com -- View this message in context: http://nabble.documentfoundation.org/Broken-master-and-newcomers-tp4180386p4182075.html Sent from the Dev mailing list archive at Nabble.com. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On Thu, Apr 21, 2016 at 6:14 AM, jan iversen wrote: > Sorry for double mail, to this theme, but speaking with a good friend of mine > (who runs a very big ci system), I got an idea, which might solve the problem > for good. > > Whenever jenkins builds a gerrit job, jenkins master knows when all platforms > for that patch is compiled Nope. as I explained earlier.. tb job are per platform/config and are not linked... so there is no garanteed that a given commit will get a tb build on every config one reason among other is the huge disparity in build time... so the setup favor quick turn around per platform/config and maximal coverage rather than big blocking check-all on the same commit setup But ultimately the solution is to concentrate on gerrit build verificatio, ot add the same coverage than tb there.. and leave the tb as they are.. that way if something pass gerrit build verification it _will_ pass tb so always green tb become a 'always gerrit workflow' Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
Sorry for double mail, to this theme, but speaking with a good friend of mine (who runs a very big ci system), I got an idea, which might solve the problem for good. Whenever jenkins builds a gerrit job, jenkins master knows when all platforms for that patch is compiled, so it should be possible to have a post-script on jenkins master (for gerrit builds) that does: 1) check if all platforms build 2) get the master commit the patch is based on 3) if the commit is higher than tag "buildable_master", move tag I am no ci expert, so I leave it to Norbert to tell if the assumptions is wrong for our setup (I was told that jenkins master can be setup very differently). This will allow contributors to build their first patch on a commit that are buildable on all platforms. Outstanding is to refine it, by including the result from our tinderboxes, which seems more difficult. thoughts? rgds jan i. > On 21 Apr 2016, at 10:25, jan iversen wrote: > > > Just had an interesting IRC chat on the infra channel. > > It seems it would be easy to have our ci system maintain a tag pr. platform. > > The idea would be: > when master compiles on platform Foo, set/move tag "buildable_on_". > > Doing that would have a very minimal effect on the build job, but give (at > least) new contributors a better security, that any build problems are local. > > I highly recommend that such a tagging be implemented, and volunteer to > update our "getInvolved" page to reflect this. > > I am aware it does not guarantee that ci can build on all platforms, but it > gives us a push in the right direction. > > thoughts? > > rgds > jan I. > > > > ___ > LibreOffice mailing list > LibreOffice@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/libreoffice ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
Just had an interesting IRC chat on the infra channel. It seems it would be easy to have our ci system maintain a tag pr. platform. The idea would be: when master compiles on platform Foo, set/move tag "buildable_on_". Doing that would have a very minimal effect on the build job, but give (at least) new contributors a better security, that any build problems are local. I highly recommend that such a tagging be implemented, and volunteer to update our "getInvolved" page to reflect this. I am aware it does not guarantee that ci can build on all platforms, but it gives us a push in the right direction. thoughts? rgds jan I. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On 07/04/16 00:17, Norbert Thiebaud wrote: > On Wed, Apr 6, 2016 at 6:14 PM, Anthonys Lists > wrote: >> On 07/04/2016 00:08, Norbert Thiebaud wrote: Imho, IF we do something like this, we should use the git facilities to > create a shallow clone backup, that people can then use ftp or rsync or > some > other interruptible protocol to download. >>> >>> We have been doing that for years: >>> >>> http://dev-www.libreoffice.org/bundles/ >>> >> That should then make this whole discussion moot then? Just modify the >> instructtions to warn newcomers that master *can* be broken, and if they >> want a known-good build to start with, go and get a bundle rather than using >> clone. Once thye've got a good build they can "git pull" and go from there. > > the bundle is not guarantee to build anymore than a random master... > > It is just that we regularly (in fact is a cron iirc) tar-gz core.git > and other repo, precisely for people that have trouble doing a git > clone on a big repo > In that case, rather than firing it on a cron job, is there any chance that we could adjust the build job so that - if the most recent bundle is more than say one day out (dunno what time limit you have on your cron job) - it fires off the tar job after a clean build. And you make it sound like it's just a tar. If it's for people who have problems with git clone, shouldn't we make it a shallow clone? Minimal change (famous last words) for maximal impact, hopefully :-) Cheers, Wol ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On Wed, Apr 6, 2016 at 6:14 PM, Anthonys Lists wrote: > On 07/04/2016 00:08, Norbert Thiebaud wrote: >>> >>> Imho, IF we do something like this, we should use the git facilities to >>> >create a shallow clone backup, that people can then use ftp or rsync or >>> > some >>> >other interruptible protocol to download. >> >> We have been doing that for years: >> >> http://dev-www.libreoffice.org/bundles/ >> > That should then make this whole discussion moot then? Just modify the > instructtions to warn newcomers that master *can* be broken, and if they > want a known-good build to start with, go and get a bundle rather than using > clone. Once thye've got a good build they can "git pull" and go from there. the bundle is not guarantee to build anymore than a random master... It is just that we regularly (in fact is a cron iirc) tar-gz core.git and other repo, precisely for people that have trouble doing a git clone on a big repo Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On 07/04/2016 00:08, Norbert Thiebaud wrote: Imho, IF we do something like this, we should use the git facilities to >create a shallow clone backup, that people can then use ftp or rsync or some >other interruptible protocol to download. We have been doing that for years: http://dev-www.libreoffice.org/bundles/ That should then make this whole discussion moot then? Just modify the instructtions to warn newcomers that master *can* be broken, and if they want a known-good build to start with, go and get a bundle rather than using clone. Once thye've got a good build they can "git pull" and go from there. Cheers, Wol ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On Wed, Apr 6, 2016 at 5:58 PM, Anthonys Lists wrote: > On 06/04/2016 08:21, Bjoern Michaelsen wrote: >>> >>> An alternative (which I have seen used in another project), would be to >>> let >>> >jenkins generate a source tar ball, incl. the .git directory, of the >>> > latest >>> >sane build. >> >> No, thats horrible -- it make onboarding to git even harder. >> > What quite do you mean? If the tarball includes the .git directory, then > it's effectively a clone, so they get a "known good" clone. > > That said, I'd rather not do that ... > > If we're going down that sort of route (and I had this discussion on the > firebird list) then remember that "git clone" does not work well over a > flaky internet connection, and if it bombs you typically have to restart > from scratch. And flaky connections tend to be pay connections, making it an > expensive business :-( > > Imho, IF we do something like this, we should use the git facilities to > create a shallow clone backup, that people can then use ftp or rsync or some > other interruptible protocol to download. We have been doing that for years: http://dev-www.libreoffice.org/bundles/ ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On 06/04/2016 08:21, Bjoern Michaelsen wrote: An alternative (which I have seen used in another project), would be to let >jenkins generate a source tar ball, incl. the .git directory, of the latest >sane build. No, thats horrible -- it make onboarding to git even harder. What quite do you mean? If the tarball includes the .git directory, then it's effectively a clone, so they get a "known good" clone. That said, I'd rather not do that ... If we're going down that sort of route (and I had this discussion on the firebird list) then remember that "git clone" does not work well over a flaky internet connection, and if it bombs you typically have to restart from scratch. And flaky connections tend to be pay connections, making it an expensive business :-( Imho, IF we do something like this, we should use the git facilities to create a shallow clone backup, that people can then use ftp or rsync or some other interruptible protocol to download. Where they go from there is then up to them, but this will also have the advantage of making it easier for people with poor internet to get on board ... :-) Cheers, Wol ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
On Wed, Apr 6, 2016 at 1:04 AM, jan iversen wrote: > Hi. > > I experience quite frequently that new contributors follow the instructions, > but end up with a master that will not compile, these day that is unlucky: here are the stats for the last 14 days: from:Wed Mar 23 21:30:24 2016 master linux rel jobs: 296 ok: 282 ko: 11 fail ratio: 3.72 % break: 9 broken duration: 2.12% master linux dbg jobs: 242 ok: 226 ko: 15 fail ratio: 6.20 % break: 12 broken duration: 2.98% master mac reljobs: 281 ok: 274 ko: 7 fail ratio: 2.49 % break: 4 broken duration: 0.86% master mac dbgjobs: 285 ok: 282 ko: 3 fail ratio: 1.05 % break: 2 broken duration: 1.95% master win reljobs: 222 ok: 212 ko: 10 fail ratio: 4.50 % break: 4 broken duration: 1.00% master win dbgjobs: 229 ok: 215 ko: 14 fail ratio: 6.11 % break: 7 broken duration: 2.64% master win64 dbg jobs: 232 ok: 216 ko: 15 fail ratio: 6.47 % break: 7 broken duration: 3.19% broken duration is giving you the following info: if you randomly pull master at any time in the period considered: what was the odd that it was broken at that time during the last 14 days your odd varied from 1 in a 115 for mac releases build to 1 in 30 for windows64 dbg let's look at windows dbg: the last break right this minute on windows dbg was 1 day 9 hours ago http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5935/ "Device or resource busy" either a parallelism problem in our build system or a bug in Windows. master was not really red because of that. the previous break was http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5915/ 2 days 5 hours ago.. direct push by sberg the one before was 2 hour earlier http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5913/ that one seems attributable to Windows crappy FS. the one before that was http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5878/ 5 days 13 hours ago, this one failed on win-dbg tb but passed Jenkins. the one before that was 6 days 11 hours ago http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5851/ that was due to 41bdaa37cc62f656cc164992c4c7d39bec7e57e2 a direct push by noel the one before that was 7 days 16 hr ago and was not due to the commit.. but some jenkins <-> windows interaction problem Note that these kind of 'break' would not actually impact someone pulling master. iow it is red in ci but not really red so of the win dbg break... 3 where due to windows glitches. 1 was properly vetted by Jenkins but still failed in that tinderbox 2 could have been avoided by using gerrit. but in the end that is 3 real break in 2 weeks for a real red duration of 2 hours 38 minutes that is 1 chance in 127 to get a random red state. so :"I experience quite frequently " seems to be a 'count the his/ignore the misses' kind of thing. IOW: my advice... let's keep nudging people to avail themselves of the CI facility and avoid direct push... that would have change the odd of a random red pull from 1 to 127 to 1 in 305 in this particular case. but let's not forget that not that long ago it was easier to count how often windows was buildable rather than hunt the commit where it was not. I also have another data point: the bibisect build: these are less sensitive since I purposefully build them without Werror and I purposfully do not run the checks (the goal being to have binaries.. not to validate the build quality) here are the result: this are the commit in bibisect that cover more than 1 source commit. the number at the end of each line is the number of unbuildable commit that are folded in that one bibisect commit 1dee09a4aa086524fd7704ad3dd6dc7e9ba201f4:1 1ebeb8fb59bfff5b9bdb3ac6db158c7c13e85e70:1 2a019e2b222ad49446c730bdbdea6c6651e36d91:1 3229e0f25f5d55d566208e42b276339a9c3c63c5:1 4c4af51cf6105e635feb206ea193f150f05e0edf:1 6d6a571cd180ee828ec8c8a5bd21a46d571d47ee:1 81168aee40fd18d7fcc475406f4c58ff7ba63455:1 8a70721672f5a1bce10547849435d7a0e286fb45:1 a54383ef92868298a368bcec871e5f2f2b2343b6:1 a9cca36433706038da0a51d6311da1bb77eb6112:1 e5c3bed99f2508b10833ccada394ce1abc37bf07:1 e72fd72eaea986ba41b8afff31ae9b77177fc685:1 f3ed988856fe00c5826781d97aaf2cae61f1abb7:1 4498e0cb3277ce65738acbc7c1faac43e9066430:2 6d46b870384fbfce0293a8fce41e373087ff681f:2 7b7852a9877e1123c60d5872ecf8ff0d78da45d7:2 ac5a57d70ad872f2dfe9d1bff73baa5318042e0e:2 b89bf27413eaf5cfab9a501a297756b8ef22eaa2:2 c6f1dc4f2407945334adef66c989f3878abc7043:2 cc822d9c6af1e8d6e937e23145471daf52ebd472:2 f73273c14261004eae364d06509b029c3a3883f2:2 45efc63eb004ebd2643cea9ebcdd810d0935bfa6:3 5de2c9cdc2e246e98ab43dde0baf14ca0947dd47:3 90fbb1101724c057d3c269f3b05f5584f701bb2f:3 b1d96f0e8acd28c6536896ed9aa70a88daa3f8fc:3 ba10b24c5f679cfcace557a920c83264b374e054:4 cebf63930f184875b7a04b3c95ad182554df5aa6:4 1d25fbbb38f0973147eda92bde66872b7cc974c0:5 1cb794851d2cbb56563a82b763da6c8a696dd465:6 326d9741e8236d3912ca55b0e9a7d4cee7b5a882:6 9e601b0af6620629a4aaf739551809b122cc55a2:7 f7a38779df3d655b6f19d
Re: Broken master and newcomers.
Hi, On Wed, Apr 06, 2016 at 07:08:06PM +0200, jan iversen wrote: > > http://people.canonical.com/~bjoern/presentations/tb3.odp > > http://people.canonical.com/~bjoern/presentations/tb3-2014.odp > > https://gerrit.libreoffice.org/gitweb?p=tb3-django.git;a=summary > > https://gerrit.libreoffice.org/gitweb?p=tb3-docker.git;a=summary > > > > for some abandoned work I did for that. I gave up on it as with > > gerrit/Jenkins, master > > state seemed 'good enough' to me. > > Interesting. also (just found that in the attic): https://www.youtube.com/watch?v=J2Thx4f_QvI to maybe make some sense of what I was trying to do ... Best, Bjoern ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
> > That idea is old -- and yes, having it directly in as a note, tag or brach git > would be awesome. Personally, I would love to see a "master-ci-verfied" > _branch_ > that is following the master branch up to the point that is known good by CI. > A > branch has the advantage that it would allow newcomers to locally rebase their > commit from one known good state to the next easily. > > However, nothing like that has been implemted yet. So: go go go. The idea of a branch triggered ideas. Having a special "tinderbox" that regulary (e.g. once a day) controls master and if it builds (at least on linux) merges changes to a fix branch, is something easy to do. Just today we saw a new contributor having problems due to master being broken, and some of us used time to identify what the problem was. So IMHO having this feature will save time, meaning I could use time to implement it. > > >> An alternative (which I have seen used in another project), would be to let >> jenkins generate a source tar ball, incl. the .git directory, of the latest >> sane build. > > No, thats horrible -- it make onboarding to git even harder. > >> I know many will say, but master is sometimes broken, you have to live with >> that. That is a statement I agree with, for anything else than the first >> build. > > Nah, the argument was rather: If we have a last 'known good master' everyone > will use that (including the people who broke master) and nobody will use the > HEAD of master anymore (nor will those that broke master). > > My guess is that we need automating bisecting on master to really pin down the > guilty in these cases. See: If we had a "check master tinderbox", it could probably easy do the bibisecting automatically (going from last known stable build). > > http://people.canonical.com/~bjoern/presentations/tb3.odp > http://people.canonical.com/~bjoern/presentations/tb3-2014.odp > https://gerrit.libreoffice.org/gitweb?p=tb3-django.git;a=summary > https://gerrit.libreoffice.org/gitweb?p=tb3-docker.git;a=summary > > for some abandoned work I did for that. I gave up on it as with > gerrit/Jenkins, master > state seemed 'good enough' to me. Interesting. > Heh, hanging those high that break master and didnt verify on Jenkins before > -- > esp. now that it is trivial? New headline for ESC, top master breakers :-) rgds jan i. > > Best, > > Bjoern ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Broken master and newcomers.
Hi, On Wed, Apr 06, 2016 at 08:04:39AM +0200, jan iversen wrote: > I have been thinking for a while, if we could get jenkins to keep a marker of > the last buildable version, store it somwhere easy to grap (preferable in the > repo itself), and then extend our build instructions to do "git checkout" to > that point. That idea is old -- and yes, having it directly in as a note, tag or brach git would be awesome. Personally, I would love to see a "master-ci-verfied" _branch_ that is following the master branch up to the point that is known good by CI. A branch has the advantage that it would allow newcomers to locally rebase their commit from one known good state to the next easily. However, nothing like that has been implemted yet. So: go go go. > An alternative (which I have seen used in another project), would be to let > jenkins generate a source tar ball, incl. the .git directory, of the latest > sane build. No, thats horrible -- it make onboarding to git even harder. > I know many will say, but master is sometimes broken, you have to live with > that. That is a statement I agree with, for anything else than the first > build. Nah, the argument was rather: If we have a last 'known good master' everyone will use that (including the people who broke master) and nobody will use the HEAD of master anymore (nor will those that broke master). My guess is that we need automating bisecting on master to really pin down the guilty in these cases. See: http://people.canonical.com/~bjoern/presentations/tb3.odp http://people.canonical.com/~bjoern/presentations/tb3-2014.odp https://gerrit.libreoffice.org/gitweb?p=tb3-django.git;a=summary https://gerrit.libreoffice.org/gitweb?p=tb3-docker.git;a=summary for some abandoned work I did for that. I gave up on it as with gerrit/Jenkins, master state seemed 'good enough' to me. > Any thought on how we can secure new contributors get a buildable version the > first time ? Heh, hanging those high that break master and didnt verify on Jenkins before -- esp. now that it is trivial? Best, Bjoern ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice
