Some test frameworks just have an "expectFailure", so a failed test actually lets the test suite pass, and a passed test makes it fail.
-Michal On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <[email protected]> wrote: > -1 for known failing tests. You need to have them all pass for a clean run. > If the tests don't work, take them out. > > I would support some additional functionality to the test runner to allow > marking tests. > We definitely have tests that are know to not work on a platform, OS > version or device. > Being able to embody that info in the test system would be great. > > Until we get more stuff cleaned up we also have tests that are flakey and > probably should just trigger a rerun if they fail. > My preference is to just fix those though. > > > > > > On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <[email protected] > >wrote: > > > Having a known failure in the tests on wp7 is no biggie, it has always > > been there. Just move on ... > > > > Sent from my iPhone > > > > > On Oct 26, 2013, at 3:24 PM, Michal Mocny <[email protected]> wrote: > > > > > > We have a proposal and prototype on the table right now for re-working > > > tests to ship with plugins, defined according to auto and manual tests. > > > > > > To accomplish what you ask for would require a specialized testing app > > that > > > simply runs both at the same time. (this wouldn't be the default, but > > would > > > be easy to make). > > > > > > Thus, I think the tests shouldn't be modified (its hard to state at > test > > > definition time in which fashion they should be used), the test runner > > > should. This wont solve the problem today, but perhaps in about a > month > > it > > > could. > > > > > > -Micha > > > > > > > > > On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) < > > > [email protected]> wrote: > > > > > >> Hi Michael, > > >> > > >> Agree. But taking into account having a way to run all the tests > > >> (including ones w/ user interaction) is very useful for Windows Phone > I > > >> propose the following > > >> 1. No changes for non-WP platforms > > >> 2. For WP > > >> a) Use the following condition for the tests which require user > > >> interaction > > >> define(..., function(...) { > > >> if (isWP8 && !runAll) return; > > >> expect(...); > > >> ... > > >> }) > > >> b) Current autotests will run w/o runAll option so won't require user > > >> interaction > > >> c) Add 'Run All Tests (Extended)' option specifically for WP8 where > we > > >> will have runAll == true > > >> > > >> Motivation: > > >> 1. I don't think we should move such tests to manual tests for WP only > > to > > >> be consistent with other platforms - we actually test api call and > check > > >> result > > >> 2. By default all tests will run w/o any user interaction > > >> 3. We will have an option to quickly check all api before release via > > Run > > >> All Tests (Extended). In other case we should have special information > > how > > >> to check all the api and not to forget to run such special tests. > > >> > > >> Thx! > > >> Sergey > > >> -----Original Message----- > > >> From: [email protected] [mailto:[email protected]] On Behalf Of > Michal > > >> Mocny > > >> Sent: Saturday, October 26, 2013 4:12 AM > > >> To: dev > > >> Subject: Re: Mobile spec tests and exclusion list > > >> > > >> Auto tests should run automatically without intervention. If user > > actions > > >> is needed for test to pass, we should call that something different > > (manual > > >> tests have been used). > > >> > > >> I think some varient of #3 is fine, this isn't a common problem. I > > >> wouldn't even test for Medic specifically, since I want my auto tests > to > > >> run automatically even when testing by hand. > > >> > > >> define(..., function(...) { > > >> if (isWP8) return; > > >> expect(...); > > >> ... > > >> }) > > >> > > >> -Michal > > >> > > >> > > >> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) < > > >> [email protected]> wrote: > > >> > > >>> Mobile spec autotests include tests which on some platforms require > > >>> user interaction to complete. For example, contact save api on > Windows > > >>> Phone requires user to manually click on save button. This prevents > > >>> the tests to be run as part of Medic test automation since test app > > >>> just hangs on such api calls. > > >>> > > >>> Is Windows Phone special or there are similar problem on other > > platforms? > > >>> > > >>> I'm thinking about the following possible approaches: > > >>> #1 Ad-hoc solution to Medic - replacing some test files as part of > > >>> Medic functionality (some additional wp specific build step). > > >>> #2 Extending mobile spec functionality- adding something like tests > > >>> exclusion config where you can define test ids (or even the whole > api) > > >>> to be skipped. Such exclusion list could be generated on the fly and > > >>> put to the app before starting tests. > > >>> #3 If there are only few such tests we can probably add check for the > > >>> current platform to determine whether to include the test. For > example: > > >>> if(!(window.MedicTestRunner && isWP8)) {testDefinition} Or the same > > >>> way but inside the test to fail gracefully. > > >>> > > >>> Thoughts? > > >>> > > >>> Thx! > > >>> Sergey > > >> > > >
