On Fri, Oct 11, 2013 at 9:08 PM, Andrew Grieve <agri...@chromium.org> wrote:
> The eval of the jasmine interface deserves mention. Is the motivation
> there that tests can choose to use another testing framework? That's why
> you don't just make jasmine functions globals?
>
I was hoping the plugins would need to depend on anything but CDVTest, and
not expect any globals. I guess, though, that CDVTest still expects the
app to provide to a test framework and some other stuff, so in the end its
no different. I was hedging on being able to update CDVTest in the future
for whatever we need, and all 3rdparty plugins would not need updating.
eval() could be used to do all sorts of clever things if we needed..
>
> One nit pick just from reading your email is that this will cause the test
> js-modules to be injected into apps that use the plugins. I think probably
> we will want to update the tools to recognize a <js-test-module>. We
> *could* work around it by adding the tests to new plugins that depend on
> the thing they are testing, but I think changing the tools would be nicer.
>
I also mentioned splitting tests into second plugin but thats overkill
except in extreme circumstances. Note that the tests aren't actually
loaded unless you require them, so its just a matter of larger binaries
which could be filtered out manually.
My person preference would be to support conditional build types, which
have come up before. ie, cordova prepare debug, cordova prepare release,
cordova prepare test -- and plugin.xml could specify a different set of
js-module for either.
A specific <js-test-module> would be fine, but isnt "0 new tooling".
Also, I forgot to mention, but we do need to add support for getting the
full list of plugins installed, which should be trivial to add to
modulemapper (maybe there is already a way to reach in, but I don't think
we have a documented api for it).
> Another nit is that it would be nice if the CordovaTests app didn't depend
> on any plugins. e.g., don't have it depend on device plugin.
>
CordovaTests doesn't explicitly depend on device plugin, except that at the
moment I have it printing the same info that MobileSpec does at startup.
This could be wrapped in a try{}catch in case the require fails, but its
good info to have in the common case.
>
> Overall, really like the approach!
>
Thanks!
>
>
> On Fri, Oct 11, 2013 at 5:17 PM, Michal Mocny <mmo...@chromium.org> wrote:
>
>> TLDR; I've implemented a plugin testing strategy that requires 0 new
>> tooling features, can support non-core plugins, and hopefully even support
>> a variety of methods for running/reporting the test results. Super alpha
>> preview, but take a look if you like the direction!
>>
>>
>> NEW: CDVTest Plugin: https://github.com/mmocny/CDVTest
>>
>> NEW: CordovaTest App: https://github.com/mmocny/CordovaTests
>>
>> UPDATED: Converted three existing plugins to this "new style". Code is on
>> feature branches of respective repos (cdvtest branch):
>> org.apache.cordova.device
>> org.apache.cordova.device-motion
>> org.chromium.storage
>> (must clone locally, switch to branch, and plugin add from local path)
>>
>>
>> Now, any plugin that wants to join in on the fun needs to provide a
>> <js-module
>> src="..." name="test" /> and use this template:
>>
>> exports.init = function() {
>>
>> eval(require('org.apache.cordova.test.test').injectJasmineInterface(this,
>> 'this'));
>>
>> // TESTS GO HERE
>> describe(..., function() {
>> it(...);
>> });
>> };
>> (The eval is optional but super useful; it will inject the jasmine
>> interface into your scope, so you don't have to type jasmine.describe,
>> jasmine.it, etc. Not sure of any cleaner way to do this.)
>>
>>
>> STEPS:
>> 1. create a new cordova project
>> 2. import the CordovaTest app into your www
>> 3. add any or all of the above plugins
>> 4. give it a run, and try out the auto tests (all pass on ipad, some still
>> fail on android)
>>
>> Lots of work left to do, but hopefully good enough to whet your appetite.
>>
>> One thing to note, I've attempted to make CDVTest and plugin tests unaware
>> of the specific jasmine version the app is hosting (so that it can be
>> swapped without changing all plugins), but it must support the new style
>> interface for async tests (ie, accept a done callback). This is the style
>> that node-jasmine uses, mocha uses, and jasmine-2.0 is going to use (I'm
>> using jasmine 2.0 rc3). This means that core plugin tests require some
>> code transformations, but the net effect is cleaner tests and more common
>> style with our node tools' tests.
>>
>> Manual tests are not implemented yet.
>>
>> -Michal
>>
>> On Fri, Oct 11, 2013 at 12:54 PM, Michal Mocny <mmo...@chromium.org>
>> wrote:
>>
>> > I'm throwing something together right now, actually. I'll post my
>> current
>> > progress today so you can take a look.
>> >
>> >
>> > On Fri, Oct 11, 2013 at 12:41 PM, Brian LeRoux <b...@brian.io> wrote:
>> >
>> >> Sorry keep meaning to respond. I like Michal's first step but growing
>> to a
>> >> full suite of tools. Are you currently tackling this Braden? I feel
>> like
>> >> it
>> >> is related to the Medic stuff and maybe we should throw one of our
>> guys on
>> >> the problem fully.
>> >>
>> >>
>> >> On Sep 27, 2013 5:10 PM, "Braden Shepherdson" <bra...@chromium.org>
>> >> wrote:
>> >>
>> >> > Which one?
>> >> >
>> >> >
>> >> > On Fri, Sep 27, 2013 at 10:09 AM, Brian LeRoux <b...@brian.io> wrote:
>> >> >
>> >> > > I really like your proposal as a starting point. Very simple but
>> would
>> >> > > allow for in-app testing as well as on the cmd line if we so wish.
>> >> > >
>> >> > >
>> >> > > On Fri, Sep 27, 2013 at 3:28 PM, Michal Mocny <mmo...@chromium.org
>> >
>> >> > wrote:
>> >> > >
>> >> > > > I was looking over some old emails from this list on plugin
>> testing,
>> >> > and
>> >> > > an
>> >> > > > idea that was proposed way back was to ship plugin tests as a
>> second
>> >> > > > plugin. That way, you can chose to install tests, or not, and
>> know
>> >> > > > explicitly if they are being copied into your final project.
>> >> > > >
>> >> > > > An alternative would be to support build targets a la
>> >> "release/debug"
>> >> > and
>> >> > > > have target-specific plugin.xml tags (assets, js-modules,
>> >> > source-file..).
>> >> > > >
>> >> > > > -Michal
>> >> > > >
>> >> > > >
>> >> > > > On Fri, Sep 27, 2013 at 4:52 AM, Brian LeRoux <b...@brian.io>
>> wrote:
>> >> > > >
>> >> > > > > I think this is basically what we've been proposing for a while
>> >> now.
>> >> > > > >
>> >> > > > >
>> >> > > > > On Thu, Sep 26, 2013 at 8:29 PM, Michal Mocny <
>> >> mmo...@chromium.org>
>> >> > > > wrote:
>> >> > > > >
>> >> > > > > > I would suggest perhaps a simpler approach, which doesn't add
>> >> > > anything
>> >> > > > > new
>> >> > > > > > to cordova-cli/plugman:
>> >> > > > > >
>> >> > > > > > - Each plugin ships with a "tests" js-module, and we
>> document a
>> >> > > > > convention
>> >> > > > > > of where they should live, and what signature it should have
>> >> (i.e.,
>> >> > > > > > cordova.require('plugin.name.Tests').forEach(...) ).
>> >> > > > > > - Will need a common way to describe/report results (others
>> >> have
>> >> > > > > > mentioned TAP).
>> >> > > > > > - Any app is free to run those plugin tests in any which way,
>> >> but
>> >> > we
>> >> > > > > ship a
>> >> > > > > > mobile-spec app which is one opinionated way to do so.
>> >> > > > > > - It attempts to require the test module for each installed
>> >> > plugin,
>> >> > > > > runs
>> >> > > > > > them, and aggregates results.
>> >> > > > > > - It could report results to some shared server, allow
>> >> toggling
>> >> > of
>> >> > > > > tests,
>> >> > > > > > etc, but no plugin should know or care about those features.
>> >> > > > > >
>> >> > > > > > Using that as a generic base:
>> >> > > > > >
>> >> > > > > > - We ship a "CDVTests" (or whatever) plugin which has a
>> bunch of
>> >> > > > library
>> >> > > > > > code for creating tests, and plugins can use it to register
>> >> their
>> >> > > > tests.
>> >> > > > > > - This makes it easier to register manual tests in a common
>> >> format
>> >> > > for
>> >> > > > > core
>> >> > > > > > plugins, and prevents code duplication for core auto tests.
>> >> > > > > > - External plugins can chose to use our testing library, or
>> not.
>> >> > > > > >
>> >> > > > > > -Michal
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Thu, Sep 26, 2013 at 10:34 AM, Braden Shepherdson <
>> >> > > > > bra...@chromium.org
>> >> > > > > > >wrote:
>> >> > > > > >
>> >> > > > > > > Here's an off-the-top-of-my-head sketch of how we might do
>> >> > Voltron
>> >> > > > > tests:
>> >> > > > > > >
>> >> > > > > > > - Add a tag to plugin.xml that names each test file:
>> >> > > > > > > <test type="automatic" src="spec/foo.js" name="Foo
>> >> Automated"
>> >> > > />
>> >> > > > > > > <test type="manual" src="spec/bar.js" name="Foo
>> Manual" />
>> >> > > > > > > - Add a new command, cordova test (maybe prepare-test),
>> that:
>> >> > > > > > > - Ignores the top-level www.
>> >> > > > > > > - Instead copies in a basic testing index.html similar
>> to
>> >> the
>> >> > > > > current
>> >> > > > > > > mobile-spec's
>> >> > > > > > > - That index reads a file akin to cordova_plugins.js
>> >> > > > > > (cordova_tests.js,
>> >> > > > > > > maybe?) generated by the CLI, containing the info from the
>> >> <test>
>> >> > > > tags.
>> >> > > > > > > - It has navigation similar to the current mobile-spec,
>> >> with
>> >> > > > > buttons
>> >> > > > > > > for the automatic and manual sections. Auto has "All" and
>> then
>> >> > each
>> >> > > > > > module,
>> >> > > > > > > manual just has the list of modules.
>> >> > > > > > >
>> >> > > > > > > Thoughts?
>> >> > > > > > >
>> >> > > > > > > Braden
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Thu, Sep 26, 2013 at 6:33 AM, Carlos Santana <
>> >> > > > csantan...@gmail.com
>> >> > > > > > > >wrote:
>> >> > > > > > >
>> >> > > > > > > > I like the idea can we call mobilespec now
>> cordova-voltron
>> >> and
>> >> > be
>> >> > > > DRY
>> >> > > > > > and
>> >> > > > > > > > use the tests form the plugins.
>> >> > > > > > > >
>> >> > > > > > > > Voltron by itself creates an App that tests only core,
>> but
>> >> as
>> >> > you
>> >> > > > > > > > use plugman to add plugins to voltron it has more test
>> >> cases.
>> >> > > > > > > >
>> >> > > > > > > > It would not be a bad idea to enhance plugin.xml in the
>> >> future
>> >> > to
>> >> > > > > > include
>> >> > > > > > > > information about testing (i.e. Directory containing
>> tests
>> >> > files,
>> >> > > > > test
>> >> > > > > > > > command, etc..)
>> >> > > > > > > >
>> >> > > > > > > > --Carlos
>> >> > > > > > > >
>> >> > > > > > > > On Thursday, September 26, 2013, Anis KADRI wrote:
>> >> > > > > > > >
>> >> > > > > > > > > What's the challenge of having us use the tests that
>> come
>> >> > with
>> >> > > > the
>> >> > > > > > > > > individual plugins ?
>> >> > > > > > > > >
>> >> > > > > > > > > On Thu, Sep 26, 2013 at 8:13 AM, David Kemp <
>> >> > drk...@google.com
>> >> > > > > > > > <javascript:;>>
>> >> > > > > > > > > wrote:
>> >> > > > > > > > > > Currently, the automated test system that we have
>> >> running
>> >> > > > > (derived
>> >> > > > > > > from
>> >> > > > > > > > > > Medic) uses only the mobilespec tests. It does not
>> yet
>> >> use
>> >> > > > tests
>> >> > > > > > > > > collected
>> >> > > > > > > > > > from the plugins. Its been talked about, but not gone
>> >> > > anywhere.
>> >> > > > > > > > > >
>> >> > > > > > > > > > David Kemp
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Wed, Sep 25, 2013 at 7:58 PM, Jesse <
>> >> > > > purplecabb...@gmail.com
>> >> > > > > > > > <javascript:;>>
>> >> > > > > > > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > >> Yeah, I have pushed some changes to mobile-spec, and
>> >> when
>> >> > I
>> >> > > > did
>> >> > > > > I
>> >> > > > > > > also
>> >> > > > > > > > > >> copied the tests into the plugin involved.
>> >> > > > > > > > > >> Until we get the magic test runner happening, I
>> think
>> >> we
>> >> > > just
>> >> > > > > keep
>> >> > > > > > > > > >> duplicating.
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> @purplecabbage
>> >> > > > > > > > > >> risingj.com
>> >> > > > > > > > > >>
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> On Wed, Sep 25, 2013 at 4:38 PM, Steven Gill <
>> >> > > > > > > stevengil...@gmail.com
>> >> > > > > > > > <javascript:;>
>> >> > > > > > > > > >
>> >> > > > > > > > > >> wrote:
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> > We copied over tests into plugins when we first
>> broke
>> >> > them
>> >> > > > > out,
>> >> > > > > > > but
>> >> > > > > > > > I
>> >> > > > > > > > > >> don't
>> >> > > > > > > > > >> > believe they have been updated.
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > I would say for now to just add the tests to
>> mobile
>> >> > spec,
>> >> > > > and
>> >> > > > > > > > > possibly in
>> >> > > > > > > > > >> > the future we go all voltron to build mobile spec
>> and
>> >> > keep
>> >> > > > > tests
>> >> > > > > > > > with
>> >> > > > > > > > > >> their
>> >> > > > > > > > > >> > corresponding plugins.
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > On Wed, Sep 25, 2013 at 4:22 PM, Joe Bowser <
>> >> > > > > bows...@gmail.com
>> >> > > > > > > > <javascript:;>>
>> >> > > > > > > > > wrote:
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > > Hey
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > Right now, I'm working on a weird file issue
>> that
>> >> > > requires
>> >> > > > > me
>> >> > > > > > to
>> >> > > > > > > > > >> > > update mobile-spec, but I'm wondering where the
>> >> tests
>> >> > > > should
>> >> > > > > > > live.
>> >> > > > > > > > > >> > > Should it all keep living in mobile-spec, or is
>> it
>> >> > with
>> >> > > > the
>> >> > > > > > > > plugins.
>> >> > > > > > > > > >> > > And if it's with the plugins, will there be
>> >> scripts to
>> >> > > > > > assemble
>> >> > > > > > > > > >> > > mobile-spec all Voltron style?
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > This came up earlier, but I haven't found any
>> fix
>> >> that
>> >> > > > > needed
>> >> > > > > > a
>> >> > > > > > > > > >> > > mobile-spec test. (Many that need native
>> testing,
>> >> > like
>> >> > > > > > > recursive
>> >> > > > > > > > > file
>> >> > > > > > > > > >> > > copy, etc). Any thoughts?
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > Joe
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > --
>> >> > > > > > > > Carlos Santana
>> >> > > > > > > > <csantan...@gmail.com>
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
