Nevermind then. Doesn't really matter considering tests will be broken out into each repo anyway.
@purplecabbage risingj.com On Mon, Oct 28, 2013 at 9:22 AM, Brian LeRoux <[email protected]> wrote: > I grok the theory here but not sure I agree w/ the practice. The working > 'master' should always be green. Working branches should be the place where > failing tests happen. If I'm looking to collaborate I won't be hacking on > any branches w/ failing tests b/c I'll just assume the developers are in > progress or they don't know wtf they are doing. > > > On Mon, Oct 28, 2013 at 8:56 AM, purplecabbage <[email protected] > >wrote: > > > Seeing an expected failure on their device may be the impetus for > > Microsoft or any other device maker to change the api. > > To me, hiding or not executing the test means we surrender to the fact > > that it will never pass. Also, there could be cases where a solution > exists > > but is not yet known, and seeing the failure could push someone to > discover > > it. > > > > Sent from my iPhone > > > > > On Oct 28, 2013, at 8:19 AM, David Kemp <[email protected]> wrote: > > > > > > Specifically I am thinking of a test that passes on one > platform/device > > > but will not pass on another one. > > > so maybe 'take them out' was poor language. > > > > > > If at all possible, the test should be coded in some way to skip it or > > > change the expectation for that platform/device, rather than 'just > > knowing' > > > that test 46 always fails on platform xx. > > > > > > This could be done in the test itself (inspect cordova-device or > > something) > > > or by some external file that describes expected results. > > > I would generally rather see it done explicitly in the test. > > > > > > > > > > > > > > >> On Mon, Oct 28, 2013 at 10:10 AM, Michal Mocny <[email protected]> > > wrote: > > >> > > >> Some test frameworks just have an "expectFailure", so a failed test > > >> actually lets the test suite pass, and a passed test makes it fail. > > >> > > >> -Michal > > >> > > >> > > >>> On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <[email protected]> > wrote: > > >>> > > >>> -1 for known failing tests. You need to have them all pass for a > clean > > >> run. > > >>> If the tests don't work, take them out. > > >>> > > >>> I would support some additional functionality to the test runner to > > allow > > >>> marking tests. > > >>> We definitely have tests that are know to not work on a platform, OS > > >>> version or device. > > >>> Being able to embody that info in the test system would be great. > > >>> > > >>> Until we get more stuff cleaned up we also have tests that are flakey > > and > > >>> probably should just trigger a rerun if they fail. > > >>> My preference is to just fix those though. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage < > > [email protected] > > >>>> wrote: > > >>> > > >>>> Having a known failure in the tests on wp7 is no biggie, it has > always > > >>>> been there. Just move on ... > > >>>> > > >>>> Sent from my iPhone > > >>>> > > >>>>> On Oct 26, 2013, at 3:24 PM, Michal Mocny <[email protected]> > > >> wrote: > > >>>>> > > >>>>> We have a proposal and prototype on the table right now for > > >> re-working > > >>>>> tests to ship with plugins, defined according to auto and manual > > >> tests. > > >>>>> > > >>>>> To accomplish what you ask for would require a specialized testing > > >> app > > >>>> that > > >>>>> simply runs both at the same time. (this wouldn't be the default, > but > > >>>> would > > >>>>> be easy to make). > > >>>>> > > >>>>> Thus, I think the tests shouldn't be modified (its hard to state at > > >>> test > > >>>>> definition time in which fashion they should be used), the test > > >> runner > > >>>>> should. This wont solve the problem today, but perhaps in about a > > >>> month > > >>>> it > > >>>>> could. > > >>>>> > > >>>>> -Micha > > >>>>> > > >>>>> > > >>>>> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) < > > >>>>> [email protected]> wrote: > > >>>>> > > >>>>>> Hi Michael, > > >>>>>> > > >>>>>> Agree. But taking into account having a way to run all the tests > > >>>>>> (including ones w/ user interaction) is very useful for Windows > > >> Phone > > >>> I > > >>>>>> propose the following > > >>>>>> 1. No changes for non-WP platforms > > >>>>>> 2. For WP > > >>>>>> a) Use the following condition for the tests which require user > > >>>>>> interaction > > >>>>>> define(..., function(...) { > > >>>>>> if (isWP8 && !runAll) return; > > >>>>>> expect(...); > > >>>>>> ... > > >>>>>> }) > > >>>>>> b) Current autotests will run w/o runAll option so won't require > > >> user > > >>>>>> interaction > > >>>>>> c) Add 'Run All Tests (Extended)' option specifically for WP8 > where > > >>> we > > >>>>>> will have runAll == true > > >>>>>> > > >>>>>> Motivation: > > >>>>>> 1. I don't think we should move such tests to manual tests for WP > > >> only > > >>>> to > > >>>>>> be consistent with other platforms - we actually test api call and > > >>> check > > >>>>>> result > > >>>>>> 2. By default all tests will run w/o any user interaction > > >>>>>> 3. We will have an option to quickly check all api before release > > >> via > > >>>> Run > > >>>>>> All Tests (Extended). In other case we should have special > > >> information > > >>>> how > > >>>>>> to check all the api and not to forget to run such special tests. > > >>>>>> > > >>>>>> Thx! > > >>>>>> Sergey > > >>>>>> -----Original Message----- > > >>>>>> From: [email protected] [mailto:[email protected]] On Behalf Of > > >>> Michal > > >>>>>> Mocny > > >>>>>> Sent: Saturday, October 26, 2013 4:12 AM > > >>>>>> To: dev > > >>>>>> Subject: Re: Mobile spec tests and exclusion list > > >>>>>> > > >>>>>> Auto tests should run automatically without intervention. If user > > >>>> actions > > >>>>>> is needed for test to pass, we should call that something > different > > >>>> (manual > > >>>>>> tests have been used). > > >>>>>> > > >>>>>> I think some varient of #3 is fine, this isn't a common problem. > I > > >>>>>> wouldn't even test for Medic specifically, since I want my auto > > >> tests > > >>> to > > >>>>>> run automatically even when testing by hand. > > >>>>>> > > >>>>>> define(..., function(...) { > > >>>>>> if (isWP8) return; > > >>>>>> expect(...); > > >>>>>> ... > > >>>>>> }) > > >>>>>> > > >>>>>> -Michal > > >>>>>> > > >>>>>> > > >>>>>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) < > > >>>>>> [email protected]> wrote: > > >>>>>> > > >>>>>>> Mobile spec autotests include tests which on some platforms > require > > >>>>>>> user interaction to complete. For example, contact save api on > > >>> Windows > > >>>>>>> Phone requires user to manually click on save button. This > > >> prevents > > >>>>>>> the tests to be run as part of Medic test automation since test > > >> app > > >>>>>>> just hangs on such api calls. > > >>>>>>> > > >>>>>>> Is Windows Phone special or there are similar problem on other > > >>>> platforms? > > >>>>>>> > > >>>>>>> I'm thinking about the following possible approaches: > > >>>>>>> #1 Ad-hoc solution to Medic - replacing some test files as part > of > > >>>>>>> Medic functionality (some additional wp specific build step). > > >>>>>>> #2 Extending mobile spec functionality- adding something like > tests > > >>>>>>> exclusion config where you can define test ids (or even the whole > > >>> api) > > >>>>>>> to be skipped. Such exclusion list could be generated on the fly > > >> and > > >>>>>>> put to the app before starting tests. > > >>>>>>> #3 If there are only few such tests we can probably add check for > > >> the > > >>>>>>> current platform to determine whether to include the test. For > > >>> example: > > >>>>>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition} Or the > > >> same > > >>>>>>> way but inside the test to fail gracefully. > > >>>>>>> > > >>>>>>> Thoughts? > > >>>>>>> > > >>>>>>> Thx! > > >>>>>>> Sergey > > >> > > >
