+1 I agree, this would be awesome. New question, should this merely be the "standard" we adhere to for core plugins, or should we actively make it difficult for plugin devs to not ship tests directly with plugins? (Not sure how we could accomplish that, so I hope its just a convention that applies to our work).
-Michal On Thu, Jun 19, 2014 at 7:48 PM, Jesse <purplecabb...@gmail.com> wrote: > My ultimate would be this: > > cordova create TestFilePlugin > cd TestFilePlugin > cordova platform add android > cordova plugin add > > http://git-wip-us.apache.org/repos/asf/cordova-labs.git#cdvtest:cordova-plugin-test-framework > cordova plugin add ../cordova-plugin-file/ > cordova plugin add ../cordova-plugin-file/test/ > cordova run android --start cdvtests/index.html > > Then do this for each plugin, and for each platform > Then do this for all combinations of plugins > ... > > Note the run --start does not yet exist, but this would be awesome! > > > @purplecabbage > risingj.com > > > On Thu, Jun 19, 2014 at 4:15 PM, Jesse <purplecabb...@gmail.com> wrote: > > > Option a) was what I suggested a ways back, and I still stand by it. > > I think it provides the greatest transparency, and simplicity, yet it is > > still very flexible. > > I don't think it would be hard to accomplish either. This is the small > > re-org I was hinting at, you've already done the hard part. > > > > > > @purplecabbage > > risingj.com > > > > > > On Thu, Jun 19, 2014 at 3:45 PM, Michal Mocny <mmo...@chromium.org> > wrote: > > > >> Andrew has raised that concern as well. My gut says that the bundling > of > >> a > >> few shorts scripts that get parsed but not run as long as they don't get > >> require() will not affect applications negatively (there are probably > many > >> more significant overheads we live with today in cordova) -- but I > >> understand why that may not be useful default. > >> > >> In that case, some ideas: (I recall these were proposed previously but > not > >> sure by whom) > >> (a) Bundle tests as a plugin-within-the-plugin as such: > >> myplugin/ > >> - plugin.xml > >> - src/... > >> - www/... > >> - tests/ > >> - plugin.xml > >> - www/... > >> > >> Which basically means the plugin tests live in the same repo/branch, and > >> are fetched as part of "plugin add", but are not moved into platforms/ > on > >> cordova prepare by default, thus don't end up in your application (disk > >> and > >> network are cheap, application startup and size costs are not, right?). > >> Then, to run tests, we basically need to iterate all plugins looking > for a > >> nested tests/plugin.xml, and install those. This can be added to > >> CLI/Plugman, or just be a CLI hook even. > >> > >> (b) add a <js-test-module> or <js-module type="test"> that is only used > if > >> you run prepare with --test. Similar to the above, but I think requires > >> more CLI/config file changes, which I'm not a fan of. > >> (c) Just ship tests as a second plugin in a second repo, and document > how > >> to install tests. Can then perhaps have a <dependency type=tests>. I > >> don't like this as much since its basically back to mobile-spec. > >> > >> WDYT? > >> > >> -Michal > >> > >> > >> On Thu, Jun 19, 2014 at 4:53 PM, Jesse <purplecabb...@gmail.com> wrote: > >> > >> > re: > >> > > >> > Q: What do I do if my plugin tests must have very large assets? > >> > - A: Don't bundle those assets with your plugin. If you can, have > >> your > >> > ... > >> > > >> > > >> > My concern is I do not want to see tests added to every project that > >> uses a > >> > plugin, even if the assets are not large, there are implications to > >> > including the test framework + all the tests because they get loaded > and > >> > processed with all of the plugins and will impact load time even if > >> never > >> > run. > >> > > >> > 99.9% of the time the plugin tests will be used by us the plugin > >> > developers, and not the people who use the plugin in there apps. > >> > > >> > I agree, having the tester install the test harness plugin dependency > >> > themselves is probably a better option, as I see you have wrapped all > >> tests > >> > inside a exports.defineAutoTests so we don't have to worry about > >> > describe/it/expects not being defined. > >> > > >> > > >> > @purplecabbage > >> > risingj.com > >> > > >> > > >> > On Thu, Jun 19, 2014 at 1:27 PM, Michal Mocny <mmo...@chromium.org> > >> wrote: > >> > > >> > > Jesee, the branch is NOT a requirement (I don't think I documented > it > >> as > >> > > such, except in the examples for installing plugins for initial > look). > >> > > Actually, we should delete those stale branches now that we are > >> moving > >> > > up-to-date tests into master. It was just for early experimentation > >> on > >> > the > >> > > feature. > >> > > > >> > > Jesse, I'm not seeing the benefit of using plugins-tests.xml or the > >> > > dependency on the test plugin yet, may you elaborate your thoughts? > >> > > > >> > > My hope was that tests are just always installed alongside plugins. > >> If > >> > > that is not a good idea for some particular plugin, say because it > >> uses > >> > > huge assets, I elaborated my answer in the plugin FAQ ( > >> > > > >> > > > >> > > >> > https://github.com/apache/cordova-labs/blob/cdvtest/cordova-plugin-test-framework/README.md#faq > >> > > ): > >> > > FAQ > >> > > > >> > > - > >> > > > >> > > Q: Should I add org.apache.cordova.test-harness as a <dependancy> > >> of > >> > my > >> > > plugin? > >> > > - A: No. The end-user should decide if they want to install the > >> test > >> > > harness, not your plugin (most users won't). > >> > > - > >> > > > >> > > Q: What do I do if my plugin tests must have very large assets? > >> > > - A: Don't bundle those assets with your plugin. If you can, have > >> your > >> > > tests fail gracefully if those assets don't don't exist > (perhaps > >> > log > >> > > a > >> > > warning, perhaps fail a single asset-checking test, and skip > the > >> > > rest). > >> > > Then, ideally download those assets automatically into local > >> > storage > >> > > the > >> > > first time tests run. Or create a manual test step to download > >> > > and install > >> > > assets. As a final alternative, split those test assets into a > >> > > separate > >> > > plugin, and instruct users to install that plugin to run your > >> full > >> > > test > >> > > suite. > >> > > - > >> > > > >> > > Q: Should I ship my app with the test harness plugin installed? > >> > > - A: Not likely. If you want, you can. Then your app could even > >> embed > >> > a > >> > > link to the test page (cdvtests/index.html) from a help > section > >> of > >> > > your app, to give end users a way to run your test suite out > in > >> > > the feild. > >> > > That may help diagnose causes of issues within your app. > Maybe. > >> > > > >> > > > >> > > ============= > >> > > > >> > > > >> > > Feel free the debate those answers -- now is certainly the time -- > >> but I > >> > > put a lot of effort to make it super flexible and to not require > >> > depending > >> > > on changes to CLI :P > >> > > > >> > > -Michal > >> > > > >> > > > >> > > On Thu, Jun 19, 2014 at 3:11 PM, Jesse <purplecabb...@gmail.com> > >> wrote: > >> > > > >> > > > Sorry I missed providing feedback on this earlier ... > >> > > > Having a deeper look at this, I am not feeling great about the > extra > >> > > > requirement that every plugin have an additional branch. > >> > > > > >> > > > Several concerns arise : > >> > > > - test branch can be out of sync with master > >> > > > - how do we test a specific version? > >> > > > - tests are not immediately visible when looking at master > >> > > > - differing versions of plugin.xml depending on the branch > >> > > > > >> > > > The majority of the work has been done (thanks Michal!), and > mostly > >> any > >> > > > suggestions I make will just require moving code and changing a > few > >> > > > conventions. > >> > > > > >> > > > What if we? : > >> > > > 1. add a plugin-test.xml file which has the exact format of > >> plugin.xml > >> > > > 2. keep tests/ and plugin-test.xml file in master branch > >> > > > 3. have plugman/cli support an additional flag --test so we can > >> install > >> > > > like this: > >> > > > cordova plugin add > >> > > > http://git-wip-us.apache.org/repos/asf/cordova-plugin-device.git > >> > --test > >> > > > This would mean that in addition to processing the plugin.xml of > the > >> > > > plugin, we would also process the plugin-test.xml file ( identical > >> > > > processing logic ) > >> > > > 4. have all plugin-test.xml files declare a dependency on > >> > > > cordova-plugin-test-framework > >> > > > > >> > > > The above suggestions could also be used in conjuction with the > >> cordova > >> > > run > >> > > > --tests platform mentioned by Michal, but without the need to > manage > >> > the > >> > > > switching of branches. > >> > > > > >> > > > > >> > > > @purplecabbage > >> > > > risingj.com > >> > > > > >> > > > > >> > > > On Tue, Jun 17, 2014 at 2:16 PM, Michal Mocny < > mmo...@chromium.org> > >> > > wrote: > >> > > > > >> > > > > Piotr: Actually I'm not sure how running tests in the harness > >> would > >> > > work, > >> > > > > since the path to the resource may be different. However, in > >> > general, > >> > > > with > >> > > > > development using the harness you aren't making any changes to > >> > plugins. > >> > > > > The whole point is for app developers who want to modify only > web > >> > > > > application bits and not deal with native compiles. > >> > > > > > >> > > > > In theory the app harness could support working on the > js-modules > >> of > >> > > > > plugins, but that sounds like a really niche idea. I'd not be > >> > opposed > >> > > to > >> > > > > someone working on it but I'm not sure you'll have luck finding > >> > > > volunteers. > >> > > > > > >> > > > > -Michal > >> > > > > > >> > > > > > >> > > > > On Tue, Jun 17, 2014 at 5:13 PM, Michal Mocny < > >> mmo...@chromium.org> > >> > > > wrote: > >> > > > > > >> > > > > > At the time I went through my design iterations I just didn't > >> want > >> > to > >> > > > > > necessarily depend on cordova tooling changes / documentation. > >> In > >> > > > other > >> > > > > > words, someone else may have a different strategy for > testing.. > >> > > > > > > >> > > > > > My personal opinion would be have the test plugin ship with a > >> > plugin > >> > > > hook > >> > > > > > (those are in, right? or at least on their way), that will > >> > > > automatically > >> > > > > > update the start page if you pass a flag to run command. Ie, > >> in an > >> > > > ideal > >> > > > > > world: `cordova run --tests` runs a plugin hook passing in > >> --tests > >> > > > flag > >> > > > > > which changes the start page, in a way that isn't overwritten > by > >> > the > >> > > > > > top-level config.xml. > >> > > > > > > >> > > > > > My 2 cents, since I don't want "our way" of testing mobile > spec > >> to > >> > be > >> > > > > "the > >> > > > > > only way" to test. Frameworks and opinions on testing > change. > >> > > > > > > >> > > > > > -Michal > >> > > > > > > >> > > > > > > >> > > > > > On Tue, Jun 17, 2014 at 4:33 PM, Piotr Zalewa < > >> pzal...@mozilla.com > >> > > > >> > > > > wrote: > >> > > > > > > >> > > > > >> One thing more - it would be great if user could create a > test > >> > using > >> > > > > test > >> > > > > >> harness app as well. Is it also considered? > >> > > > > >> > >> > > > > >> Dnia Tue Jun 17 13:27:22 2014 Martin Gonzalez pisze: > >> > > > > >> > >> > > > > >> It would be a nice to have in the cli, aimed to just setup > the > >> > > right > >> > > > > path > >> > > > > >>> in the config.xml, maybe along with an another argument to > >> build, > >> > > > > >>> run/emulate as well. > >> > > > > >>> It sounds great. > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> 2014-06-17 15:21 GMT-05:00 Piotr Zalewa < > pzal...@mozilla.com > >> >: > >> > > > > >>> > >> > > > > >>> Thanks Martin, > >> > > > > >>>> > >> > > > > >>>> Has it been considered to create a separate command > >> "testrun" or > >> > > > > similar > >> > > > > >>>> which would remove the need to edit the config.xml? > >> > > > > >>>> > >> > > > > >>>> Dnia Tue Jun 17 11:58:33 2014 Michal Mocny pisze: > >> > > > > >>>> > >> > > > > >>>> Martin, thanks for posting those links. > >> > > > > >>>> > >> > > > > >>>>> > >> > > > > >>>>> And I'll look into the INFRA tickets I need to file to set > >> up a > >> > > > repo > >> > > > > >>>>> for > >> > > > > >>>>> that plugin, since its ready to come out of labs. > >> > > > > >>>>> > >> > > > > >>>>> > >> > > > > >>>>> On Tue, Jun 17, 2014 at 2:06 PM, Martin Gonzalez < > >> > > > > >>>>> martin.c.glez.g...@gmail.com> wrote: > >> > > > > >>>>> > >> > > > > >>>>> This is the Cordova Plugin Test Framework readme.md, > you > >> can > >> > > > catch > >> > > > > >>>>> up > >> > > > > >>>>> > >> > > > > >>>>>> with > >> > > > > >>>>>> the functionality by reading some of the content: > >> > > > > >>>>>> > >> > > > > >>>>>> Repository: > >> > > > > >>>>>> https://github.com/apache/cordova-labs > >> > > > > >>>>>> > >> > > > > >>>>>> Docs: > >> > > > > >>>>>> > >> https://github.com/apache/cordova-labs/blob/master/README.md > >> > > > > >>>>>> > >> > > > > >>>>>> https://github.com/apache/cordova-labs/blob/cdvtest/ > >> > > > > >>>>>> cordova-plugin-test-framework/README.md > >> > > > > >>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> 2014-06-17 12:56 GMT-05:00 Piotr Zalewa < > >> pzal...@mozilla.com > >> > >: > >> > > > > >>>>>> > >> > > > > >>>>>> a documentation explaining how it's gonna work > >> > > > > >>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>>> Dnia Tue Jun 17 10:51:58 2014 Michal Mocny pisze: > >> > > > > >>>>>>> > >> > > > > >>>>>>> What do you mean? > >> > > > > >>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> On Tue, Jun 17, 2014 at 1:50 PM, Piotr Zalewa < > >> > > > > pzal...@mozilla.com> > >> > > > > >>>>>>>> wrote: > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> Is there any predev document? > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>>>> Dnia Mon Jun 16 18:30:46 2014 Andrew Grieve pisze: > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> Yeah, really exciting. Thanks for taking this on. > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>>> On Mon, Jun 16, 2014 at 3:42 PM, Michal Mocny < > >> > > > > >>>>>>>>>> mmo...@chromium.org> > >> > > > > >>>>>>>>>> wrote: > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> Fantastic! > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> I'll try to keep an eye out on the PR's, and please > >> ping > >> > me > >> > > > if > >> > > > > >>>>>>>>>>> you > >> > > > > >>>>>>>>>>> would > >> > > > > >>>>>>>>>>> like any help. > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> -Michal > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> On Mon, Jun 16, 2014 at 3:25 PM, Marcel Kinard < > >> > > > > >>>>>>>>>>> cmarc...@gmail.com> > >> > > > > >>>>>>>>>>> wrote: > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> Hi, after some discussions here with IBM > >> management, > >> > > > we’re > >> > > > > >>>>>>>>>>> going > >> > > > > >>>>>>>>>>> to > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> bring > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>>> in a couple extra interns for a week to jumpstart > the > >> > > > > migration > >> > > > > >>>>>>>>>>>> of > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> the > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>> tests out of mobile-spec into the new plugin test > >> framework. > >> > > > Staci > >> > > > > >>>>>>> > >> > > > > >>>>>>>> Cooper > >> > > > > >>>>>>>>>>>> will be leading this effort, and Martin Gonzalez > will > >> > be a > >> > > > > part > >> > > > > >>>>>>>>>>>> of > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> it. > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>>> So if you see a bunch of pull requests, this is what it > >> is > >> > > for. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> We’ll > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>> get > >> > > > > >>>>>>> > >> > > > > >>>>>>>> the interns to submit an ICLA asap. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>> -- > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> Piotr Zalewa > >> > > > > >>>>>>>>> Mozilla > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> -- > >> > > > > >>>>>>>> > >> > > > > >>>>>>> Piotr Zalewa > >> > > > > >>>>>>> Mozilla > >> > > > > >>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> -- > >> > > > > >>>>>> Regards, > >> > > > > >>>>>> Martin Gonzalez > >> > > > > >>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> > >> > > > > >>>>> -- > >> > > > > >>>> Piotr Zalewa > >> > > > > >>>> Mozilla > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> > >> > > > > >> -- > >> > > > > >> Piotr Zalewa > >> > > > > >> Mozilla > >> > > > > >> > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
