Hi Everyone, It is true that mobile spec test results for Windows are currently not reliable. But our experience allows us to understand which automated tests are expected to fail on Windows / WP8 platforms and which are true fails.
At the same time, we are fixing plugin automated tests for Windows platform and mark unsupported feature tests as pending (e.g. https://github.com/apache/cordova-plugin-contacts/pull/49, https://github.com/apache/cordova-plugin-file/pull/88). We have the same plans for WP8 platform. To ensure Cordova plugins quality on Windows, we also make complete run for manual plugin tests in addition to mobile spec tests. Regards, Dmitriy -----Original Message----- From: Toda, Shingo [mailto:[email protected]] Sent: Friday, October 31, 2014 4:57 AM To: [email protected] Cc: Toda, Shingo Subject: RE: [mobile-spec] consistency and reliability It's great that this topic has been raised as we have some questions on mobile spec. We need to test Cordova and core plugins to assure ourselves of the quality so we are running mobile spec and add tests if there are APIs for which we couldn't find tests in mobile spec but I'm interested in how you test Cordova on various platforms. On Android and iOS, mobile spec runs nicely but on Windows 8 mobile spec there are many crashes due to unsupported API, alert undefined, file paths etc. I am wondering how you test Cordova for Windows 8. If you don't mind, I would like to know if you are using (new/old) mobilespec or other test suite for testing Windows 8. Is there any strategy for testing platform-specific APIs which not all platforms support? Regards, Shingo -----Original Message----- From: Brian LeRoux [mailto:[email protected]] Sent: Friday, October 31, 2014 11:09 AM To: [email protected] Subject: Re: [mobile-spec] consistency and reliability Nope. On Thu, Oct 30, 2014, 5:04 PM Jesse <[email protected]> wrote: > I would much rather we fix things, than continually rewrite + discard, > which seems to be the norm these days. > tape/tap would require us to throw away thousands of jasmine2 based > tests wouldn't it? > > > @purplecabbage > risingj.com > > On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <[email protected]> wrote: > > > tape (and or tap) has no globals and simple / shallow callback > > solution which fixes that > > > > we will have fresh intern blood to sacrifice to the test lib gods > > alter soon.. > > > > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <[email protected]> wrote: > > > > > I've seen that in the past and it usually happens due to misuse of > > jasmine, > > > which seems unfortunately quite brittle. Specifically, if done() > > > isn't called, or an async event fires after a test has already > > > failed / timed > > out > > > and calls done(), or registers new tests with it() etc, weird > > > things > will > > > happen. > > > > > > Usually its a sign that a test is not handling failures > > > appropriately, > > but > > > it is a bit unfortunate that jasmine is so brittle. > > > > > > -Michal > > > > > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo < > > [email protected] > > > > > > > wrote: > > > > > > > This is exactly what I got too. > > > > > > > > Regards, > > > > Shingo > > > > > > > > From: Shazron [mailto:[email protected]] > > > > Sent: Friday, October 31, 2014 9:02 AM > > > > To: [email protected] > > > > Cc: Toda, Shingo > > > > Subject: Re: [mobile-spec] consistency and reliability > > > > > > > > Seems to be exactly what I am seeing. > > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1 > > > > > > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo < > > > [email protected] > > > > <mailto:[email protected]>> wrote: > > > > Hi Shazron > > > > > > > > Our team are running mobilespec for Android, iOS and Windows. I > cannot > > > see > > > > the attachment but we also have been seeing what you’ve gotten > > > > since > > new > > > > style mobilespec was introduced. It seems that each plugin test > > > > run > > works > > > > fine but if all tests get run at once I am likely to get more > > > > than > one > > > > report in a run reporting that there are a number of specs like > > > > 900 > > specs > > > > (I don’t think specs are so much). > > > > > > > > I noticed that some of the specs were called more than once when > > > > I > got > > > > multiple reports. I usually disable FileTransfer, Globalization > > > > and > > Media > > > > specs at first then run test for them individually because I saw > > > > some > > of > > > > those tests got called more than once. This could be just my > > > > case so > it > > > > might not be yours. > > > > > > > > Regards, > > > > Shingo > > > > > > > > From: Shazron > > > > [mailto:[email protected]<mailto:[email protected]>] > > > > Sent: Friday, October 31, 2014 7:54 AM > > > > To: [email protected]<mailto:[email protected]> > > > > Subject: [mobile-spec] consistency and reliability > > > > > > > > > > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios > > > > --globalplugins` > > > > > > > > Has anyone gotten mobile-spec to work consistently? I get wildly > > > different > > > > results on each run. Sometimes there is more than one report in > > > > a > run. > > > See: > > > > > > > > [Inline image 1] > > > > > > > > > > > > > >
