On 01/16/2014 07:41 PM, Vivien Nicolas wrote:
On 15/01/2014 20:48, Andrew Sutherland wrote:
However, the simplest/most pragmatic thing to do initially is probably
just to use the existing linux perf tool to generate logs of what's
happening during the test runs using "perf schedule" and when we see a
failure due to event loop sadness, we look at that log and decide if it
was a real failure and what we can do about it. There are a great set of
examples at http://lwn.net/Articles/353295/
I want to make bug 606574 works with gonk and integrate something in
devtools. That should provide us a Gecko mechanism to monitor the
event loop.
Right. We use the Gecko built-in mechanism to monitor the event loop.
But as Fabrice was saying, our test-runners are frequently virtual
machines/similar with performance characteristics that are usually not
guaranteed. I was proposing some options we can use so that if the
Gecko mechanism indicates that the event loop appears to have stopped
processing events for a second that we can quickly determine if it was a
false positive or not in order to avoid wasting time investigating the
wrong thing.
As a specific example of this problem, Thunderbird was experiencing a
number of timeouts/test failures on our mozmill UI tests that were hard
to understand. It turned out that the VM (VMware based) was using a
disk image mounted via NFS from another server. Some kind of network
glitch/NFS would result in the VM effectively freezing for a few seconds
at a time. There was nothing wrong with the tests, but timeouts would
of course fire when the wall-time on the VM instantly jumped forward a
few seconds. The error was detected by surfacing time-stamps and
suspicious latencies in our logging, but it was very confusing until
that happened. It sounds like the implementation in bug 606574 would be
equally tricked by this phenomenon or similar phenomenon. For example,
starvation of resources due to other VMs monopolizing the CPUs/I/O
substrates sufficiently, etc.
Andrew
_______________________________________________
dev-b2g mailing list
dev-b2g@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g
