We've used WebTest for almost 7 years. In general the tests are quite stable. There's one test that over the years randomly failed and we couldn't figure out why. I mean, this went on for years. Finally, we discovered the underlying bug, and it was a bad one, it was too bad we had not investigated the random test failure more thoroughly. WebTest was trying to tell us something, and we ignored it.
That said, sometimes we have backward compatibility issues so it can be a bit unstable in that way. About a month ago we tried to upgrade to the latest release, but it (probably it was really HtmlUnit) stopped being able to click some of our buttons that have Javascript. I asked about it on this list and never got any help with it, so we just stayed at the old version. That bugs me, but I don't know what to do about it, I asked if we should submit a bug to HtmlUnit (I don' follow their list or anything) but didn't get an answer. -- Lisa On Thu, Oct 14, 2010 at 2:51 AM, mascis <[email protected]> wrote: > > Hi, > > This is not a question for a single problem but rather a opening of a > discussion. > I have been working with Canoo Webtest for a half of a year now. Everything > has worked quite well and these automated tests have been a great help. The > only bigger problem is that Webtest don't seem to be very stable. What I > mean is that the results that Webtest change sometimes even if nothing has > changed in the web page I test. > Sometimes Webtest loses cookies used for logging, sometimes it gives > strange > JS errors and next time I rune the same test it passes, sometimes it gives > error for not founding a certain web page even if resulting page shows > correct page etc. > We use Bamboo to run tests in certain times every day and for some reason > this seems to make things harder. Bamboo makes tests run a bit slower so > chance of missing logging cookie is bigger and for some reason those > strange > JS errors ( > http://old.nabble.com/A-strange-Java-Script-error-to29894316.html this > for > example) occur more often. > This is quite annoying since I can't rely that once I make test pass it > will > pass every time I run it. It also takes my time to go trough reports and > look if there is real errors or something that Webtest just gives without > anything being broken. > We have a lot of JS and AJAX etc. "advanced" things in our web page. I > don't > know if that makes those errors appear or what. Does anyone else have same > kind of things or is it just me? > > -- > View this message in context: > http://old.nabble.com/Webtest-and-stability-tp29960485p29960485.html > Sent from the WebTest mailing list archive at Nabble.com. > > _______________________________________________ > WebTest mailing list > [email protected] > http://lists.canoo.com/mailman/listinfo/webtest > -- Lisa Crispin Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers and Agile Teams_ (Addison-Wesley 2009) Contributor to _Beautiful Testing_ (O'Reilly 2009) http://lisacrispin.com @lisacrispin on Twitter
