Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 12:11 PM, William Siegrist wrote: On Feb 26, 2010, at 9:29 AM, Maciej Stachowiak wrote: On Feb 26, 2010, at 9:17 AM, Dimitri Glazkov wrote: To summarize the thread: 1) We're adopting "when in doubt, roll it out" approach to patches that turn tree red. I think it's po

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Ojan Vafai
On Fri, Feb 26, 2010 at 11:53 AM, Maciej Stachowiak wrote: > On Feb 26, 2010, at 11:43 AM, Simon Fraser wrote: > 2. The Mozilla tinderbox page (their buildbot waterfall) had a way for > people to leave comments, by adding a "star" to a particular build with a > comment. This is used as a way to c

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Jeremy Orlow
On Fri, Feb 26, 2010 at 9:00 PM, Jeremy Orlow wrote: > On Fri, Feb 26, 2010 at 8:53 PM, Maciej Stachowiak wrote: > >> >> On Feb 26, 2010, at 11:43 AM, Simon Fraser wrote: >> >> >>> Mozilla has (or at least had when I worked there) two additional "tree >>> rules" that helped keep the tree green:

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread William Siegrist
On Feb 26, 2010, at 9:29 AM, Maciej Stachowiak wrote: > > On Feb 26, 2010, at 9:17 AM, Dimitri Glazkov wrote: > >> To summarize the thread: >> >> 1) We're adopting "when in doubt, roll it out" approach to patches >> that turn tree red. > > I think it's polite, though not mandatory, to make a r

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Jeremy Orlow
On Fri, Feb 26, 2010 at 8:53 PM, Maciej Stachowiak wrote: > > On Feb 26, 2010, at 11:43 AM, Simon Fraser wrote: > > >> Mozilla has (or at least had when I worked there) two additional "tree >> rules" that helped keep the tree green: >> >> 1. A sheriff was appointed at all times, and had the autho

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 11:43 AM, Simon Fraser wrote: Mozilla has (or at least had when I worked there) two additional "tree rules" that helped keep the tree green: 1. A sheriff was appointed at all times, and had the authority to close the tree if there was significant build or test breakag

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Simon Fraser
On Feb 26, 2010, at 11:34 AM, Geoffrey Garen wrote: >> There is a non-trivial cost of this workflow on the rest of the team. >> -keeps the commit-queue from running >> -often results in new test failures going unnoticed because the tree is >> already red >> -we can't generally trust that all the

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 11:34 AM, Geoffrey Garen wrote: There is a non-trivial cost of this workflow on the rest of the team. -keeps the commit-queue from running -often results in new test failures going unnoticed because the tree is already red -we can't generally trust that all the tests shou

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 11:08 AM, Alexey Proskuryakov wrote: On 26.02.2010, at 10:15, Jeremy Orlow wrote: I didn't even know it existed until now. Was there ever an email sent out on this? If so, I missed it. Buildbot used to announce results there, but it was a few years ago. My recolle

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Geoffrey Garen
> There is a non-trivial cost of this workflow on the rest of the team. > -keeps the commit-queue from running > -often results in new test failures going unnoticed because the tree is > already red > -we can't generally trust that all the tests should pass locally I think all of the costs you li

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Ojan Vafai
On Fri, Feb 26, 2010 at 10:40 AM, Geoffrey Garen wrote: > A lot of the proposals on this thread would interfere with this work flow: > > 1. Finish patch and get it working on local machine. > 2. Check in, automatically test for compatibility on other machines and > OS's in parallel, resolving une

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alexey Proskuryakov
On 26.02.2010, at 10:15, Jeremy Orlow wrote: I didn't even know it existed until now. Was there ever an email sent out on this? If so, I missed it. Buildbot used to announce results there, but it was a few years ago. My recollection is that when it worked, about half of active committ

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Enrica Casucci
I didn't know that failing tests would block the commit queue. I saw they were failing yesterday afternoon and I thought it was ok to wait until this morning to fix them. My apologies for the inconvenience. I believe a reasonable approach to handle these situations is to try to contact the perso

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Geoffrey Garen
I think it would be more productive to start with better systems for informing people that they've broken something, and move on to rolling out patches aggressively if informing people doesn't work. It's not surprising that people neglect a red tree when they don't know about it. A lot of the

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Jeremy Orlow
On Fri, Feb 26, 2010 at 7:06 PM, Maciej Stachowiak wrote: > > On Feb 26, 2010, at 9:56 AM, Alexey Proskuryakov wrote: > > >> On 26.02.2010, at 9:29, Maciej Stachowiak wrote: >> >> I'd like it if we had an IRC bot that announced build breakage on >>> #webkit. >>> >> >> >> Perhaps better yet, on #

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 9:56 AM, Alexey Proskuryakov wrote: On 26.02.2010, at 9:29, Maciej Stachowiak wrote: I'd like it if we had an IRC bot that announced build breakage on #webkit. Perhaps better yet, on #webkit-build, as buildbot used to do. In the past, no one ever joined #webkit-buil

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 9:58 AM, Alexey Proskuryakov wrote: On 26.02.2010, at 9:50, Jeremy Orlow wrote: > I would re-write rule one as something like this: > 1. Comment in the bugzilla bug when the build breaks. If there is no > bugzilla bug, comment in #webkit. > 2. 15 minutes after the b

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alexey Proskuryakov
On 26.02.2010, at 9:50, Jeremy Orlow wrote: > I would re-write rule one as something like this: > 1. Comment in the bugzilla bug when the build breaks. If there is no > bugzilla bug, comment in #webkit. > 2. 15 minutes after the break or 10 minutes after the comment, with > no reply from

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alexey Proskuryakov
On 26.02.2010, at 9:29, Maciej Stachowiak wrote: I'd like it if we had an IRC bot that announced build breakage on #webkit. Perhaps better yet, on #webkit-build, as buildbot used to do. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webk

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Jeremy Orlow
On Fri, Feb 26, 2010 at 6:47 PM, Dimitri Glazkov wrote: > On Fri, Feb 26, 2010 at 9:44 AM, Eric Seidel wrote: > > The bots take 15 minutes to cycle. The moment the build is broken, > > thats FIX_TIME + BOT_CYCLE_TIME until their green again. > > > > I think we should cap the fix grace period at

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Dimitri Glazkov
On Fri, Feb 26, 2010 at 9:44 AM, Eric Seidel wrote: > The bots take 15 minutes to cycle.  The moment the build is broken, > thats FIX_TIME + BOT_CYCLE_TIME until their green again. > > I think we should cap the fix grace period at something like 15 > minutes, that means no more than 30 minutes of

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alex Milowski
On Fri, Feb 26, 2010 at 8:55 AM, Adam Barth wrote: > On Fri, Feb 26, 2010 at 8:47 AM, Alex Milowski wrote: >> On Fri, Feb 26, 2010 at 8:19 AM, Adam Barth wrote: >>> On Fri, Feb 26, 2010 at 7:24 AM, Alex Milowski wrote: On Fri, Feb 26, 2010 at 7:17 AM, Eric Seidel wrote: > On Fri, Feb

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Eric Seidel
The bots take 15 minutes to cycle. The moment the build is broken, thats FIX_TIME + BOT_CYCLE_TIME until their green again. I think we should cap the fix grace period at something like 15 minutes, that means no more than 30 minutes of tree redness per break. That might be too aggressive to start

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Chris Jerdonek
On Fri, Feb 26, 2010 at 1:36 AM, Adam Barth wrote: > 2) Require pre-commit vetting of patches.  We have the resources to > > Here's how I would design the life and times of a patch: > > 1) Contributor uploads patch and nominates it for review. > 2) Patch vetted by the EWS on numerous platforms. >

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 1:36 AM, Adam Barth wrote: Not to point fingers, but we've been having trouble keeping build.webkit.org green these past few weeks. As I write this message, every platform is broken, again. As the project scales, polluting the build with brokenness impacts more developers

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Nikolas Zimmermann
Am 26.02.2010 um 18:17 schrieb Dimitri Glazkov: To summarize the thread: 1) We're adopting "when in doubt, roll it out" approach to patches that turn tree red. 2) Need to find a way to run Mac-EWS for non-committers. 3) Enable "build-break" emails to webkit-dev or another opt-in mailing list

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Maciej Stachowiak
On Feb 26, 2010, at 9:17 AM, Dimitri Glazkov wrote: To summarize the thread: 1) We're adopting "when in doubt, roll it out" approach to patches that turn tree red. I think it's polite, though not mandatory, to make a reasonable effort to find the person responsible for the breakage and giv

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Dimitri Glazkov
To summarize the thread: 1) We're adopting "when in doubt, roll it out" approach to patches that turn tree red. 2) Need to find a way to run Mac-EWS for non-committers. 3) Enable "build-break" emails to webkit-dev or another opt-in mailing list What else? :DG< On Fri, Feb 26, 2010 at 9:08 AM, A

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Adam Barth
Well, the total bill is a bit bigger, but yeah. :) Adam On Fri, Feb 26, 2010 at 9:05 AM, Kenneth Christiansen wrote: > That is some of the best 9 cents spend ever! > > Kenneth > > On Fri, Feb 26, 2010 at 1:58 PM, Adam Barth wrote: >> On Fri, Feb 26, 2010 at 8:55 AM, Adam Barth wrote: >>> Ama

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Kenneth Christiansen
That is some of the best 9 cents spend ever! Kenneth On Fri, Feb 26, 2010 at 1:58 PM, Adam Barth wrote: > On Fri, Feb 26, 2010 at 8:55 AM, Adam Barth wrote: >> Amazon tells me that our current bots use about 4 GB/month of download >> bandwidth and 600 MB/month of upload bandwidth.  I presume al

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Adam Barth
On Fri, Feb 26, 2010 at 8:55 AM, Adam Barth wrote: > Amazon tells me that our current bots use about 4 GB/month of download > bandwidth and 600 MB/month of upload bandwidth.  I presume almost all > of the bandwidth is to update the working copies of the four bots > hosted there. In case you're cu

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Adam Barth
On Fri, Feb 26, 2010 at 8:47 AM, Alex Milowski wrote: > On Fri, Feb 26, 2010 at 8:19 AM, Adam Barth wrote: >> On Fri, Feb 26, 2010 at 7:24 AM, Alex Milowski wrote: >>> On Fri, Feb 26, 2010 at 7:17 AM, Eric Seidel wrote: On Fri, Feb 26, 2010 at 7:12 AM, Alex Milowski wrote: >> The only

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alex Milowski
On Fri, Feb 26, 2010 at 8:19 AM, Adam Barth wrote: > On Fri, Feb 26, 2010 at 7:24 AM, Alex Milowski wrote: >> On Fri, Feb 26, 2010 at 7:17 AM, Eric Seidel wrote: >>> On Fri, Feb 26, 2010 at 7:12 AM, Alex Milowski wrote: > The only EWS which requires committer access is Mac-EWS.  All other >

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Adam Barth
On Fri, Feb 26, 2010 at 7:24 AM, Alex Milowski wrote: > On Fri, Feb 26, 2010 at 7:17 AM, Eric Seidel wrote: >> On Fri, Feb 26, 2010 at 7:12 AM, Alex Milowski wrote: The only EWS which requires committer access is Mac-EWS.  All other EWS bots will run any patch. >>> >>> Why is that?   T

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alex Milowski
On Fri, Feb 26, 2010 at 7:17 AM, Eric Seidel wrote: > On Fri, Feb 26, 2010 at 7:12 AM, Alex Milowski wrote: >>> The only EWS which requires committer access is Mac-EWS.  All other >>> EWS bots will run any patch. >> >> Why is that?   That's the platform I'm most interested in see run. > > Various

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Eric Seidel
On Fri, Feb 26, 2010 at 7:12 AM, Alex Milowski wrote: >> The only EWS which requires committer access is Mac-EWS.  All other >> EWS bots will run any patch. > > Why is that?   That's the platform I'm most interested in see run. Various reasons. Mostly due to our current hardware setup. If someo

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Alex Milowski
On Fri, Feb 26, 2010 at 7:09 AM, Eric Seidel wrote: > On Fri, Feb 26, 2010 at 4:14 AM, Kenneth Christiansen > wrote: >>> 1) Contributor uploads patch and nominates it for review. >>> 2) Patch vetted by the EWS on numerous platforms. >> >> When a non-committer uploads a patch, it is not being vet

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Eric Seidel
On Fri, Feb 26, 2010 at 4:14 AM, Kenneth Christiansen wrote: >> 1) Contributor uploads patch and nominates it for review. >> 2) Patch vetted by the EWS on numerous platforms. > > When a non-committer uploads a patch, it is not being vet by EWS. I > know that is due to security issues. It would be

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Kenneth Christiansen
> 1) Contributor uploads patch and nominates it for review. > 2) Patch vetted by the EWS on numerous platforms. When a non-committer uploads a patch, it is not being vet by EWS. I know that is due to security issues. It would be really nice with an option for a reviewer to accept it to run on the

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Xan Lopez
On Fri, Feb 26, 2010 at 11:36 AM, Adam Barth wrote: > Not to point fingers, but we've been having trouble keeping > build.webkit.org green these past few weeks.  As I write this message, > every platform is broken, again.  As the project scales, polluting the > build with brokenness impacts more d

Re: [webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Jeremy Orlow
On Fri, Feb 26, 2010 at 10:36 AM, Adam Barth wrote: > Not to point fingers, but we've been having trouble keeping > build.webkit.org green these past few weeks. As I write this message, > every platform is broken, again. As the project scales, polluting the > build with brokenness impacts more

[webkit-dev] The tree is on fire: a tragedy of the commons

2010-02-26 Thread Adam Barth
Not to point fingers, but we've been having trouble keeping build.webkit.org green these past few weeks. As I write this message, every platform is broken, again. As the project scales, polluting the build with brokenness impacts more developers and drains more productivity. Here are some approa

Re: [webkit-dev] The tree is on fire

2010-01-25 Thread Eric Seidel
I filed a bug about turning back on blame-mails: https://bugs.webkit.org/show_bug.cgi?id=34075 Further discussion specific to the mails can be held there. On Mon, Jan 25, 2010 at 12:50 AM, Eric Seidel wrote: > It seems one way to more-quickly solve these sorts of fires would be > to turn back on

Re: [webkit-dev] The tree is on fire

2010-01-25 Thread Eric Seidel
It seems one way to more-quickly solve these sorts of fires would be to turn back on the buildbot blame mails. We used to have them years ago, and somewhere along the line they got turned off. I looked briefly at how to make the changes to: http://trac.webkit.org/browser/trunk/WebKitTools/BuildSl

Re: [webkit-dev] The tree is on fire

2010-01-22 Thread Steve Falkenburg
Fixed in http://trac.webkit.org/changeset/53743 -steve On Jan 22, 2010, at 5:37 PM, Jian Li wrote: > There is another show stopper caused by > http://trac.webkit.org/changeset/53740. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists

Re: [webkit-dev] The tree is on fire

2010-01-22 Thread Jian Li
Thanks dimich for fixing this debug-only test failure. The buildbot seems to be improving a lot now. Sorry for the failures caused by r53722. I have fixed all related failures on Mac and Qt. The Gtk results seem not be updated for quite some time. So I do not touch them. If there is anything else

Re: [webkit-dev] The tree is on fire

2010-01-22 Thread Eric Seidel
I'm learning to email. On Fri, Jan 22, 2010 at 5:03 PM, Eric Seidel wrote: > https://bugs.webkit.org/show_bug.cgi?id=33948 also broke leopard. > > On Fri, Jan 22, 2010 at 4:34 PM, Maciej Stachowiak wrote: >> >> Ever since this change: >> >> http://trac.webkit.org/changeset/53722 >> >> The Leopar

Re: [webkit-dev] The tree is on fire

2010-01-22 Thread Dmitry Titov
Ok, I believe r53727 and r53738 are going to fix Leopard and SnowLeopard (and improve Windows) test runs. Keeping an eye on it... ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] The tree is on fire

2010-01-22 Thread Maciej Stachowiak
Ever since this change: http://trac.webkit.org/changeset/53722 The Leopard and SnowLeopard tests have been failing fast/dom/Window/window-property-descriptors.html. Ironically, it looks like that patch landed new results in an attempt to fix that test, but it looks to me like the new results