I grok the theory here but not sure I agree w/ the practice. The working 'master' should always be green. Working branches should be the place where failing tests happen. If I'm looking to collaborate I won't be hacking on any branches w/ failing tests b/c I'll just assume the developers are in progress or they don't know wtf they are doing.
On Mon, Oct 28, 2013 at 8:56 AM, purplecabbage <[email protected]>wrote: > Seeing an expected failure on their device may be the impetus for > Microsoft or any other device maker to change the api. > To me, hiding or not executing the test means we surrender to the fact > that it will never pass. Also, there could be cases where a solution exists > but is not yet known, and seeing the failure could push someone to discover > it. > > Sent from my iPhone > > > On Oct 28, 2013, at 8:19 AM, David Kemp <[email protected]> wrote: > > > > Specifically I am thinking of a test that passes on one platform/device > > but will not pass on another one. > > so maybe 'take them out' was poor language. > > > > If at all possible, the test should be coded in some way to skip it or > > change the expectation for that platform/device, rather than 'just > knowing' > > that test 46 always fails on platform xx. > > > > This could be done in the test itself (inspect cordova-device or > something) > > or by some external file that describes expected results. > > I would generally rather see it done explicitly in the test. > > > > > > > > > >> On Mon, Oct 28, 2013 at 10:10 AM, Michal Mocny <[email protected]> > wrote: > >> > >> Some test frameworks just have an "expectFailure", so a failed test > >> actually lets the test suite pass, and a passed test makes it fail. > >> > >> -Michal > >> > >> > >>> On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <[email protected]> wrote: > >>> > >>> -1 for known failing tests. You need to have them all pass for a clean > >> run. > >>> If the tests don't work, take them out. > >>> > >>> I would support some additional functionality to the test runner to > allow > >>> marking tests. > >>> We definitely have tests that are know to not work on a platform, OS > >>> version or device. > >>> Being able to embody that info in the test system would be great. > >>> > >>> Until we get more stuff cleaned up we also have tests that are flakey > and > >>> probably should just trigger a rerun if they fail. > >>> My preference is to just fix those though. > >>> > >>> > >>> > >>> > >>> > >>> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage < > [email protected] > >>>> wrote: > >>> > >>>> Having a known failure in the tests on wp7 is no biggie, it has always > >>>> been there. Just move on ... > >>>> > >>>> Sent from my iPhone > >>>> > >>>>> On Oct 26, 2013, at 3:24 PM, Michal Mocny <[email protected]> > >> wrote: > >>>>> > >>>>> We have a proposal and prototype on the table right now for > >> re-working > >>>>> tests to ship with plugins, defined according to auto and manual > >> tests. > >>>>> > >>>>> To accomplish what you ask for would require a specialized testing > >> app > >>>> that > >>>>> simply runs both at the same time. (this wouldn't be the default, but > >>>> would > >>>>> be easy to make). > >>>>> > >>>>> Thus, I think the tests shouldn't be modified (its hard to state at > >>> test > >>>>> definition time in which fashion they should be used), the test > >> runner > >>>>> should. This wont solve the problem today, but perhaps in about a > >>> month > >>>> it > >>>>> could. > >>>>> > >>>>> -Micha > >>>>> > >>>>> > >>>>> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> Hi Michael, > >>>>>> > >>>>>> Agree. But taking into account having a way to run all the tests > >>>>>> (including ones w/ user interaction) is very useful for Windows > >> Phone > >>> I > >>>>>> propose the following > >>>>>> 1. No changes for non-WP platforms > >>>>>> 2. For WP > >>>>>> a) Use the following condition for the tests which require user > >>>>>> interaction > >>>>>> define(..., function(...) { > >>>>>> if (isWP8 && !runAll) return; > >>>>>> expect(...); > >>>>>> ... > >>>>>> }) > >>>>>> b) Current autotests will run w/o runAll option so won't require > >> user > >>>>>> interaction > >>>>>> c) Add 'Run All Tests (Extended)' option specifically for WP8 where > >>> we > >>>>>> will have runAll == true > >>>>>> > >>>>>> Motivation: > >>>>>> 1. I don't think we should move such tests to manual tests for WP > >> only > >>>> to > >>>>>> be consistent with other platforms - we actually test api call and > >>> check > >>>>>> result > >>>>>> 2. By default all tests will run w/o any user interaction > >>>>>> 3. We will have an option to quickly check all api before release > >> via > >>>> Run > >>>>>> All Tests (Extended). In other case we should have special > >> information > >>>> how > >>>>>> to check all the api and not to forget to run such special tests. > >>>>>> > >>>>>> Thx! > >>>>>> Sergey > >>>>>> -----Original Message----- > >>>>>> From: [email protected] [mailto:[email protected]] On Behalf Of > >>> Michal > >>>>>> Mocny > >>>>>> Sent: Saturday, October 26, 2013 4:12 AM > >>>>>> To: dev > >>>>>> Subject: Re: Mobile spec tests and exclusion list > >>>>>> > >>>>>> Auto tests should run automatically without intervention. If user > >>>> actions > >>>>>> is needed for test to pass, we should call that something different > >>>> (manual > >>>>>> tests have been used). > >>>>>> > >>>>>> I think some varient of #3 is fine, this isn't a common problem. I > >>>>>> wouldn't even test for Medic specifically, since I want my auto > >> tests > >>> to > >>>>>> run automatically even when testing by hand. > >>>>>> > >>>>>> define(..., function(...) { > >>>>>> if (isWP8) return; > >>>>>> expect(...); > >>>>>> ... > >>>>>> }) > >>>>>> > >>>>>> -Michal > >>>>>> > >>>>>> > >>>>>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> Mobile spec autotests include tests which on some platforms require > >>>>>>> user interaction to complete. For example, contact save api on > >>> Windows > >>>>>>> Phone requires user to manually click on save button. This > >> prevents > >>>>>>> the tests to be run as part of Medic test automation since test > >> app > >>>>>>> just hangs on such api calls. > >>>>>>> > >>>>>>> Is Windows Phone special or there are similar problem on other > >>>> platforms? > >>>>>>> > >>>>>>> I'm thinking about the following possible approaches: > >>>>>>> #1 Ad-hoc solution to Medic - replacing some test files as part of > >>>>>>> Medic functionality (some additional wp specific build step). > >>>>>>> #2 Extending mobile spec functionality- adding something like tests > >>>>>>> exclusion config where you can define test ids (or even the whole > >>> api) > >>>>>>> to be skipped. Such exclusion list could be generated on the fly > >> and > >>>>>>> put to the app before starting tests. > >>>>>>> #3 If there are only few such tests we can probably add check for > >> the > >>>>>>> current platform to determine whether to include the test. For > >>> example: > >>>>>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition} Or the > >> same > >>>>>>> way but inside the test to fail gracefully. > >>>>>>> > >>>>>>> Thoughts? > >>>>>>> > >>>>>>> Thx! > >>>>>>> Sergey > >> >
