Well that process is a lot like what I do but there is one proto step I do before anything else. First I run the mobile-spec qunit tests with the currently released version. I make note of how many tests are passing and failing (as well as which tests fail). I use this as a baseline for step 1. If more tests fail with the proposed releases we've taken a step backwards.
Once upon a time I started working on a doc that explicitly stated the results expected of each manual test but I've never been able to finish it. Qunit is a good way to test to make sure all the objects of the API exist but what I believe Becky was referring to and I agree that it does not give us 100% coverage. For instance, the qunit tests can check for a Media object but they can't tell you if the sound really plays or it can check on the video capture API but it can't really grab a video as it requires user interaction. Simon Mac Donald http://hi.im/simonmacdonald On Mon, Feb 27, 2012 at 4:50 PM, Filip Maj <[email protected]> wrote: > Testing process: this is important :) > > Let's hash that out in this thread, and document over at > http://wiki.apache.org/cordova/RunningTests > > Here's my version, please add anything that you think I'm missing. For me > it's as simple as: > > 1. Run the qunit tests in mobile-spec. This is a baseline. I have never > seen it pass 100% but below 50 tests failing (less than 10%) seems to be > what the average across platforms is. Our goal is to reduce this as much > as possible. > 2. Run the "manual" tests in mobile-spec. These are the individual > links/pages showing up when you run mobile-spec. > > I would like to see the manual tests refined a bit more to have expected > behaviour mapped out more obviously. Something like: > - you run the accelerometer manual tests. > - these tests ask you to lie the device flat on a table and hit a > button. We assert that the various x/y/z accel values line up to what we > expect. > - do the same for various other device positions. > - rinse and repeat for all of our APIs. > > I had Becky express concerns that mobile-spec is not a complete test suite > and "needs improvement." Please, if you feel this is the case, it is > *extremely* important to commit patches to the test suite! It is our only > tool that allows us to say with confidence "we are ok to release." > > Moving forward, with cordova-js lined up as dropping in the next few > releases, as well as the embeddable webview + rename changes coming into > the native side on all of our platform implementations, the test suite > needs to be our go-to method for determining what works and what doesn't. > If you feel it is lacking: patch it. > > Let's get this process nailed down in our wiki and in this thread so our > next release goes smoother. > > On 12-02-27 1:35 PM, "Simon MacDonald" <[email protected]> wrote: > > >Well running mobile spec is only part of the regression I usually do. > >Typically I'll run the autotests with the previous version then again with > >the latest version and find the differences. By doing that this release I > >was able to fix the issue with the DirectoryReader on Android. Then I'll > >run through the manual tests in mobile spec to make sure that things are > >still working. Then I start going through the examples on > >docs.phonegap.combecause it is really embarrassing when the sample > >code in our docs doesn't > >run. > > > >I've been doing this for a number of releases now and it helps me get a > >feel for how stable we are. With all the moving parts for 1.5 I don't feel > >like we are as stable as 1.4 and don't think we should tag tomorrow as > >there are some more things I want to look at. > > > >Simon Mac Donald > >http://hi.im/simonmacdonald > > > > > >On Mon, Feb 27, 2012 at 4:30 PM, Joe Bowser <[email protected]> wrote: > > > >> What regression process are you using? I've personally have never seen > >> mobile-spec pass on Android, so I'm curious as to what the criteria is. > >> > >> Joe > >> > >> On Mon, Feb 27, 2012 at 1:24 PM, Simon MacDonald > >> <[email protected]>wrote: > >> > >> > I do not feel comfortable tagging 1.5.0 tomorrow. I'm still finding > >>bugs > >> in > >> > regression. > >> > > >> > Simon Mac Donald > >> > http://hi.im/simonmacdonald > >> > > >> > > >> > On Mon, Feb 27, 2012 at 4:20 PM, Joe Bowser <[email protected]> > wrote: > >> > > >> > > JIRA says that this is supposed to happen tomorrow. > >> > > > >> > > On Mon, Feb 27, 2012 at 1:18 PM, Simon MacDonald > >> > > <[email protected]>wrote: > >> > > > >> > > > Wait, when are we supposed to be releasing 1.5? > >> > > > > >> > > > Simon Mac Donald > >> > > > http://hi.im/simonmacdonald > >> > > > > >> > > > > >> > > > On Mon, Feb 27, 2012 at 4:10 PM, Steven Gill > >><[email protected] > >> > > >> > > > wrote: > >> > > > > >> > > > > I haven't pulled the release candidate together yet, so feel > >>free > >> to > >> > > > retag. > >> > > > > > >> > > > > -Steve > >> > > > > > >> > > > > On Mon, Feb 27, 2012 at 12:58 PM, Becky Gibson < > >> > [email protected] > >> > > > > >wrote: > >> > > > > > >> > > > > > Has anything been done with the build? I would like to add > >>the > >> > > > > > <type>Entry.toURL() api and deprecate to.URI() into the > >>release > >> > > > > candidate. > >> > > > > > Is ready to go, just need to retag if this change is to go > >>in. > >> > > > > > > >> > > > > > -becky > >> > > > > > > >> > > > > > > >> > > > > > On Mon, Feb 27, 2012 at 12:35 PM, Becky Gibson < > >> > > [email protected] > >> > > > > > >wrote: > >> > > > > > > >> > > > > > > iOS is tagged for 1.5.0rc1 > >> > > > > > > > >> > > > > > > > >> > > > > > > On Sat, Feb 25, 2012 at 12:31 AM, Simon MacDonald < > >> > > > > > > [email protected]> wrote: > >> > > > > > > > >> > > > > > >> One test case was using escapeURL when it should have used > >> > > escapeURI > >> > > > > so > >> > > > > > I > >> > > > > > >> fixed it and checked it in. Also, in common-js there was a > >>bug > >> > in > >> > > > the > >> > > > > > >> readEntries method where it was referencing this.fullPath > >>when > >> > it > >> > > > > should > >> > > > > > >> have been this.path. I fixed that as well. > >> > > > > > >> > >> > > > > > >> I've regenerated the cordova.android.js file and checked > >>it in > >> > > along > >> > > > > > with > >> > > > > > >> a > >> > > > > > >> version bump for Android to 1.5.0rc1. > >> > > > > > >> > >> > > > > > >> Simon Mac Donald > >> > > > > > >> http://hi.im/simonmacdonald > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On Fri, Feb 24, 2012 at 6:53 PM, Joe Bowser < > >> [email protected]> > >> > > > > wrote: > >> > > > > > >> > >> > > > > > >> > Which test case? > >> > > > > > >> > > >> > > > > > >> > On Fri, Feb 24, 2012 at 12:23 PM, Simon MacDonald < > >> > > > > > >> > [email protected] > >> > > > > > >> > > wrote: > >> > > > > > >> > > >> > > > > > >> > > I will not be tagging Android today. There are a few > >> > > previously > >> > > > > > >> passing > >> > > > > > >> > > test cases in mobile-spec that are now failing. Will > >>tag > >> > once > >> > > > the > >> > > > > > >> problem > >> > > > > > >> > > is resolved. > >> > > > > > >> > > > >> > > > > > >> > > Simon Mac Donald > >> > > > > > >> > > http://hi.im/simonmacdonald > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > On Fri, Feb 24, 2012 at 1:54 PM, Steve Gill < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > >> > > wrote: > >> > > > > > >> > > > >> > > > > > >> > > > Hey guys, > >> > > > > > >> > > > > >> > > > > > >> > > > Lets tag 1.5.0rc1 today. Post here once your repo is > >> > tagged. > >> > > > > > >> > > > > >> > > > > > >> > > > -Steve > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > On Feb 22, 2012, at 9:31 AM, Brian LeRoux > >><[email protected]> > >> > > wrote: > >> > > > > > >> > > > > >> > > > > > >> > > > > notes from call this am > >> > > > > > >> > > > > > >> > > > > > >> > > > > - aim to release 1.5 on tues > >> > > > > > >> > > > > - cordova js unlikely for wp, ios, blackberry > >> > > > > > >> > > > > - cordova rename from phonegap on track > >> > > > > > >> > > > > - coho release tool now does gpg signing: need > >>input > >> > from > >> > > > > > mentors > >> > > > > > >> on > >> > > > > > >> > > > > cutting official apache release > >> > > > > > >> > > > > - plugins are breaking: create a wiki page about > >> > updating > >> > > > > ---but > >> > > > > > >> > > > > plugins have bigger problems which we aim to > >>address > >> > > > sometime > >> > > > > > >> between > >> > > > > > >> > > > > now and 2.x > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >
