Re: [webkit-dev] Please be careful with webkitpy changes!

[youenn fablet]

Hi,

Do you have any additional information on why the first patch failed?
Can the problem be reproduced?

Thanks
   Y
Le 1 avr. 2015 21:43, "Brent Fulgham"  a écrit :

> Hi Everyone,
>
> We lost Windows EWS coverage for the past 36 hours due to a very
> benign-appearing change to some webkitpy code. I haven’t yet figured out
> why this particular set of changes caused the Windows bots to start
> failing, but it has to do with various differences between the Cygwin
> Python 2.7.8 build and the versions used on our other EWS bots.
>
> This does not seem like something developers SHOULD have to worry about,
> but it’s an unfortunately reality that they really do need to.
>
> To make matters worse, the patch that introduced the problem passed EWS.
> This is because the EWS bots only really begin using changes to webkitpy
> when they restart processing (about once every 10-13 build iterations).
>
> To help combat this problem, I’d like to request that when making changes
> to webkitpy, please keep an eye on the various EWS bots to make sure they
> continue processing. If they do start failing, please roll the patch back
> out and we can work together to resolve the issue.
>
> I apologize for how manual and inconvenient this needs to be (at least for
> now), but keeping the EWS up and running is critical to the smooth function
> of this project.
>
> If you have any questions, please don’t hesitate to e-mail me or look for
> me on IRC.
>
> Thanks!
>
> -Brent
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Osztrogonác Csaba]

Hi,

There are different issues here:
- EWS bots don't run webkitpy tests, except the Mac WK1 and WK2.
(running webkitpy tests is part of run tests) Maybe we should
run webkitpy tests on the non tester EWS bots too.

- poor webkitpy test coverage: Windows buildbots should have
noticed the mentioned failure after the patch landed.
(Or is it possible that Win EWS and buildbot have different config?)

- EWS can't check properly if a patch break its own code, because
it applies the patch and then do the build, test, etc - without
restarting the process. By this time most of the code are in the
memory, changing the code of a running process won't influence
the actual run. It isn't trivial to fix this issue, and I'm not
sure if it is so important.

This kind of issues are very rare and can be catched easily if the port
maintainers / gardeners are monitoring the EWS queues and buildbots
continuously, not only once a day or more rarely. Maybe we could add
a heartbeat feature to webkitbot. It could ping maintainers on IRC
or send an email if a buildbot or EWS is offline or the queue is too 
long for a while.


br,
Ossy

Maciej Stachowiak írta:
You said it did not detect the failure until many builds later. That 
seems bad. People expect EWS validation to happen on their bug, not out 
of band 10-13 builds later. Is there any way to fix this limitation? 
That seems better than asking people to remember exceptions about 
patches that EWS can't validate the normal way.




On Apr 1, 2015, at 9:29 PM, Brent Fulgham > wrote:


The Windows EWS bots process patches fairly quickly. Once I corrected 
the problem today, it managed to process about 97 patches in about an 
hour.


I do think one bottleneck is due to individual EWS bots "locking" 
patches. The first bot to reach a patch locks the patch against other 
bots handling it. If the patch happens to be 'consumed' be a bot with 
some kind of problem (e.g., bad local configuration, a full disk 
drive, etc.), that patch will not be touched again --- even if the other 
eight EWS bots are sitting dormant.


Is there some other processing metric you are concerned about?

? Brent Fulgham - Apple Inc.



On Apr 1, 2015, at 2:26 PM, Maciej Stachowiak > wrote:



Is it possible to make EWS start processing changes more promptly?

On Apr 1, 2015, at 12:42 PM, Brent Fulgham > wrote:


Hi Everyone,

We lost Windows EWS coverage for the past 36 hours due to a very 
benign-appearing change to some webkitpy code. I haven't yet figured 
out why this particular set of changes caused the Windows bots to 
start failing, but it has to do with various differences between the 
Cygwin Python 2.7.8 build and the versions used on our other EWS bots.


This does not seem like something developers SHOULD have to worry 
about, but it's an unfortunately reality that they really do need to.


To make matters worse, the patch that introduced the problem passed 
EWS. This is because the EWS bots only really begin using changes to 
webkitpy when they restart processing (about once every 10-13 build 
iterations).


To help combat this problem, I'd like to request that when making 
changes to webkitpy, please keep an eye on the various EWS bots to 
make sure they continue processing. If they do start failing, please 
roll the patch back out and we can work together to resolve the issue.


I apologize for how manual and inconvenient this needs to be (at 
least for now), but keeping the EWS up and running is critical to 
the smooth function of this project.


If you have any questions, please don't hesitate to e-mail me or 
look for me on IRC.


Thanks!

-Brent

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Ryosuke Niwa]

Right now, EWS restarts after every 20 iterations or so.  We could expedite
the process by doing it more frequently.  However, I can't think of a way
we can test a webkitpy patch that affects EWS on EWS since it's webkitpy
code that's applying the patch and running the tests, etc…

On Thu, Apr 2, 2015 at 1:48 AM, Maciej Stachowiak  wrote:

> You said it did not detect the failure until many builds later. That seems
> bad. People expect EWS validation to happen on their bug, not out of band
> 10-13 builds later. Is there any way to fix this limitation? That seems
> better than asking people to remember exceptions about patches that EWS
> can't validate the normal way.
>
>
>
> On Apr 1, 2015, at 9:29 PM, Brent Fulgham  wrote:
>
> The Windows EWS bots process patches fairly quickly. Once I corrected the
> problem today, it managed to process about 97 patches in about an hour.
>
> I do think one bottleneck is due to individual EWS bots “locking” patches.
> The first bot to reach a patch locks the patch against other bots handling
> it. If the patch happens to be ‘consumed’ be a bot with some kind of
> problem (e.g., bad local configuration, a full disk drive, etc.), that
> patch will not be touched again — even if the other eight EWS bots are
> sitting dormant.
>
> Is there some other processing metric you are concerned about?
>
>  Brent Fulgham - Apple Inc.
>
>
>
> On Apr 1, 2015, at 2:26 PM, Maciej Stachowiak  wrote:
>
>
> Is it possible to make EWS start processing changes more promptly?
>
> On Apr 1, 2015, at 12:42 PM, Brent Fulgham  wrote:
>
> Hi Everyone,
>
> We lost Windows EWS coverage for the past 36 hours due to a very
> benign-appearing change to some webkitpy code. I haven’t yet figured out
> why this particular set of changes caused the Windows bots to start
> failing, but it has to do with various differences between the Cygwin
> Python 2.7.8 build and the versions used on our other EWS bots.
>
> This does not seem like something developers SHOULD have to worry about,
> but it’s an unfortunately reality that they really do need to.
>
> To make matters worse, the patch that introduced the problem passed EWS.
> This is because the EWS bots only really begin using changes to webkitpy
> when they restart processing (about once every 10-13 build iterations).
>
> To help combat this problem, I’d like to request that when making changes
> to webkitpy, please keep an eye on the various EWS bots to make sure they
> continue processing. If they do start failing, please roll the patch back
> out and we can work together to resolve the issue.
>
> I apologize for how manual and inconvenient this needs to be (at least for
> now), but keeping the EWS up and running is critical to the smooth function
> of this project.
>
> If you have any questions, please don’t hesitate to e-mail me or look for
> me on IRC.
>
> Thanks!
>
> -Brent
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Maciej Stachowiak]

You said it did not detect the failure until many builds later. That seems bad. 
People expect EWS validation to happen on their bug, not out of band 10-13 
builds later. Is there any way to fix this limitation? That seems better than 
asking people to remember exceptions about patches that EWS can't validate the 
normal way.



> On Apr 1, 2015, at 9:29 PM, Brent Fulgham  wrote:
> 
> The Windows EWS bots process patches fairly quickly. Once I corrected the 
> problem today, it managed to process about 97 patches in about an hour.
> 
> I do think one bottleneck is due to individual EWS bots “locking” patches. 
> The first bot to reach a patch locks the patch against other bots handling 
> it. If the patch happens to be ‘consumed’ be a bot with some kind of problem 
> (e.g., bad local configuration, a full disk drive, etc.), that patch will not 
> be touched again — even if the other eight EWS bots are sitting dormant.
> 
> Is there some other processing metric you are concerned about?
> 
>  Brent Fulgham - Apple Inc.
> 
> 
> 
>> On Apr 1, 2015, at 2:26 PM, Maciej Stachowiak  wrote:
>> 
>> 
>> Is it possible to make EWS start processing changes more promptly?
>> 
>>> On Apr 1, 2015, at 12:42 PM, Brent Fulgham  wrote:
>>> 
>>> Hi Everyone,
>>> 
>>> We lost Windows EWS coverage for the past 36 hours due to a very 
>>> benign-appearing change to some webkitpy code. I haven’t yet figured out 
>>> why this particular set of changes caused the Windows bots to start 
>>> failing, but it has to do with various differences between the Cygwin 
>>> Python 2.7.8 build and the versions used on our other EWS bots.
>>> 
>>> This does not seem like something developers SHOULD have to worry about, 
>>> but it’s an unfortunately reality that they really do need to.
>>> 
>>> To make matters worse, the patch that introduced the problem passed EWS. 
>>> This is because the EWS bots only really begin using changes to webkitpy 
>>> when they restart processing (about once every 10-13 build iterations).
>>> 
>>> To help combat this problem, I’d like to request that when making changes 
>>> to webkitpy, please keep an eye on the various EWS bots to make sure they 
>>> continue processing. If they do start failing, please roll the patch back 
>>> out and we can work together to resolve the issue.
>>> 
>>> I apologize for how manual and inconvenient this needs to be (at least for 
>>> now), but keeping the EWS up and running is critical to the smooth function 
>>> of this project.
>>> 
>>> If you have any questions, please don’t hesitate to e-mail me or look for 
>>> me on IRC.
>>> 
>>> Thanks!
>>> 
>>> -Brent
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Adam Barth]

On Apr 1, 2015 9:29 PM, "Brent Fulgham"  wrote:
>
> The Windows EWS bots process patches fairly quickly. Once I corrected the
problem today, it managed to process about 97 patches in about an hour.
>
> I do think one bottleneck is due to individual EWS bots “locking”
patches. The first bot to reach a patch locks the patch against other bots
handling it. If the patch happens to be ‘consumed’ be a bot with some kind
of problem (e.g., bad local configuration, a full disk drive, etc.), that
patch will not be touched again — even if the other eight EWS bots are
sitting dormant.

It might be worth adding the ability for a bot to unlock a patch when it
detects a problem with itself.  We didn't have that originally because we
were paranoid about handling the case of an "unclean" bot shutdown and
wanted to make sure we got a lot of experience with that case.  At this
point, it's probably an overly paranoid design.

Adam

> Is there some other processing metric you are concerned about?
>
>  Brent Fulgham - Apple Inc.
>
>
>
>> On Apr 1, 2015, at 2:26 PM, Maciej Stachowiak  wrote:
>>
>>
>> Is it possible to make EWS start processing changes more promptly?
>>
>>> On Apr 1, 2015, at 12:42 PM, Brent Fulgham  wrote:
>>>
>>> Hi Everyone,
>>>
>>> We lost Windows EWS coverage for the past 36 hours due to a very
benign-appearing change to some webkitpy code. I haven’t yet figured out
why this particular set of changes caused the Windows bots to start
failing, but it has to do with various differences between the Cygwin
Python 2.7.8 build and the versions used on our other EWS bots.
>>>
>>> This does not seem like something developers SHOULD have to worry
about, but it’s an unfortunately reality that they really do need to.
>>>
>>> To make matters worse, the patch that introduced the problem passed
EWS. This is because the EWS bots only really begin using changes to
webkitpy when they restart processing (about once every 10-13 build
iterations).
>>>
>>> To help combat this problem, I’d like to request that when making
changes to webkitpy, please keep an eye on the various EWS bots to make
sure they continue processing. If they do start failing, please roll the
patch back out and we can work together to resolve the issue.
>>>
>>> I apologize for how manual and inconvenient this needs to be (at least
for now), but keeping the EWS up and running is critical to the smooth
function of this project.
>>>
>>> If you have any questions, please don’t hesitate to e-mail me or look
for me on IRC.
>>>
>>> Thanks!
>>>
>>> -Brent
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Brent Fulgham]

The Windows EWS bots process patches fairly quickly. Once I corrected the 
problem today, it managed to process about 97 patches in about an hour.

I do think one bottleneck is due to individual EWS bots “locking” patches. The 
first bot to reach a patch locks the patch against other bots handling it. If 
the patch happens to be ‘consumed’ be a bot with some kind of problem (e.g., 
bad local configuration, a full disk drive, etc.), that patch will not be 
touched again — even if the other eight EWS bots are sitting dormant.

Is there some other processing metric you are concerned about?

 Brent Fulgham - Apple Inc.



> On Apr 1, 2015, at 2:26 PM, Maciej Stachowiak  wrote:
> 
> 
> Is it possible to make EWS start processing changes more promptly?
> 
>> On Apr 1, 2015, at 12:42 PM, Brent Fulgham  wrote:
>> 
>> Hi Everyone,
>> 
>> We lost Windows EWS coverage for the past 36 hours due to a very 
>> benign-appearing change to some webkitpy code. I haven’t yet figured out why 
>> this particular set of changes caused the Windows bots to start failing, but 
>> it has to do with various differences between the Cygwin Python 2.7.8 build 
>> and the versions used on our other EWS bots.
>> 
>> This does not seem like something developers SHOULD have to worry about, but 
>> it’s an unfortunately reality that they really do need to.
>> 
>> To make matters worse, the patch that introduced the problem passed EWS. 
>> This is because the EWS bots only really begin using changes to webkitpy 
>> when they restart processing (about once every 10-13 build iterations).
>> 
>> To help combat this problem, I’d like to request that when making changes to 
>> webkitpy, please keep an eye on the various EWS bots to make sure they 
>> continue processing. If they do start failing, please roll the patch back 
>> out and we can work together to resolve the issue.
>> 
>> I apologize for how manual and inconvenient this needs to be (at least for 
>> now), but keeping the EWS up and running is critical to the smooth function 
>> of this project.
>> 
>> If you have any questions, please don’t hesitate to e-mail me or look for me 
>> on IRC.
>> 
>> Thanks!
>> 
>> -Brent
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Please be careful with webkitpy changes!

[Maciej Stachowiak]

Is it possible to make EWS start processing changes more promptly?

> On Apr 1, 2015, at 12:42 PM, Brent Fulgham  wrote:
> 
> Hi Everyone,
> 
> We lost Windows EWS coverage for the past 36 hours due to a very 
> benign-appearing change to some webkitpy code. I haven’t yet figured out why 
> this particular set of changes caused the Windows bots to start failing, but 
> it has to do with various differences between the Cygwin Python 2.7.8 build 
> and the versions used on our other EWS bots.
> 
> This does not seem like something developers SHOULD have to worry about, but 
> it’s an unfortunately reality that they really do need to.
> 
> To make matters worse, the patch that introduced the problem passed EWS. This 
> is because the EWS bots only really begin using changes to webkitpy when they 
> restart processing (about once every 10-13 build iterations).
> 
> To help combat this problem, I’d like to request that when making changes to 
> webkitpy, please keep an eye on the various EWS bots to make sure they 
> continue processing. If they do start failing, please roll the patch back out 
> and we can work together to resolve the issue.
> 
> I apologize for how manual and inconvenient this needs to be (at least for 
> now), but keeping the EWS up and running is critical to the smooth function 
> of this project.
> 
> If you have any questions, please don’t hesitate to e-mail me or look for me 
> on IRC.
> 
> Thanks!
> 
> -Brent
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Please be careful with webkitpy changes!

[Brent Fulgham]

Hi Everyone,

We lost Windows EWS coverage for the past 36 hours due to a very 
benign-appearing change to some webkitpy code. I haven’t yet figured out why 
this particular set of changes caused the Windows bots to start failing, but it 
has to do with various differences between the Cygwin Python 2.7.8 build and 
the versions used on our other EWS bots.

This does not seem like something developers SHOULD have to worry about, but 
it’s an unfortunately reality that they really do need to.

To make matters worse, the patch that introduced the problem passed EWS. This 
is because the EWS bots only really begin using changes to webkitpy when they 
restart processing (about once every 10-13 build iterations).

To help combat this problem, I’d like to request that when making changes to 
webkitpy, please keep an eye on the various EWS bots to make sure they continue 
processing. If they do start failing, please roll the patch back out and we can 
work together to resolve the issue.

I apologize for how manual and inconvenient this needs to be (at least for 
now), but keeping the EWS up and running is critical to the smooth function of 
this project.

If you have any questions, please don’t hesitate to e-mail me or look for me on 
IRC.

Thanks!

-Brent
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev