On Tue, Nov 9, 2010 at 3:38 PM, Robert Collins <[email protected]> wrote: > This is obviously related to the discussion about how much value we > get from the tests, but I've not seen any data saying that *windmill* > is our problem. It seems to be hard to use correctly, and we seem to > be failing to use it correctly in at least some places. Which sucks > :(. >
This may be true for the recent failings, and maybe it's generally true. But I don't think this is the whole story. Tests can be correct and pass by themselves but fail when run as a group. There have been efforts to rewrite tests and restructure html to not rely on xpath as well as some other tricks (waits.forElement after page load rather than assertNode, for example). So you could say this is not using Windmill correctly, but "correct" is a sliding term IMHO. In other words, we should be using it correctly as we understand "correct" today, but there's no guarantee this will be correct tomorrow. Windmill is notorious about varying results under system load or other conditions (the 512k file size limitation in the past comes to mind). Some people cannot run them locally well without really increasing the timeout constants used. I have an issue now with getting a lazr-js upgrade branch landed for similar reasons. All tests pass individually, but when run as a group in ec2, 10-15 of them fail at any time. I've been working on this for a week or better myself, rockstar for two weeks prior to that. And yeah, we've made several little changes all along that get us closer to landing every time. Eventually I'll learn that if I dance on one leg and turn around in a circle 3 times while throwing salt over my shoulder and also sacrificing a goat, then the tests will eventually pass consistently. But I'm not sure I feel comfortable calling that incantation the "correct" use of Windmill. Cheers, deryck -- Deryck Hodge https://launchpad.net/~deryck http://www.devurandom.org/ _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
