This reiterates some of the conclusions Braden and Mark have already reached, but generally testing should be split into two categories: unit tests and integration tests.
There's debate about whether things like the file system should be mocked or not, but I think it's wrong to call our current tests bad, they are just incomplete. 1. Unit tests should run fast, each test should run in milliseconds. 2. integration tests include end-to-end communication, and speed is not important +1 for expanding current unit tests for improved coverage I suggest including a separate suite of tests (re: run independently of unit tests). The idea is that the unit tests can be run as part of development, integration tests run once before you push a change. On Thu, Nov 7, 2013 at 11:57 AM, Anis KADRI <anis.ka...@gmail.com> wrote: > The reason why every FS call is mocked is speed but speed is > subjective in my opinion. Given the features of CLI, my opinion is > that anything < 1 minute is acceptable. When I run tests, I am not > actively watching them execute. > I think the only calls that should be mocked are network calls because > you should be able to run tests offline. > I think we should convert (trash?) existing tests back to what they > were instead of adding more end-to-end tests. We should also pay more > attention de Windows. > > On Thu, Nov 7, 2013 at 10:31 AM, Braden Shepherdson <bra...@chromium.org> > wrote: > > Alright, Mark and I have discussed this further, and we will be beginning > > the effort with some end-to-end tests that will supplement the existing > > tests. > > > > To some extent this is duplicating things that go on in the CI, since it > > checks out various plugins using the tools. But we think it's still a > > worthwhile effort since it will make it much easier to catch any problems > > at commit time. > > > > The main pain point in the existing tests that we'll want to fix is their > > fragility. The worst offenders here are the platform parsers, especially > > wp7+8. So Steve, if you're looking for ways to help, I'd suggest looking > at > > those (wp in particular, but all of them). These are some of the worst, > > most vacuous tests we have. They're not providing sample inputs and > > checking the outputs, they are checking that the right functions get > > called. I propose that they should be operating on a real, checked-in, > > example project, in the spec/fixtures directory. The tests should run the > > parsers, and make sure all the files are in the right places and have the > > right contents. > > > > Braden > > > > > > On Thu, Nov 7, 2013 at 1:02 PM, Steve Gill <stevengil...@gmail.com> > wrote: > > > >> I don't think we should scrap the current tests. I am totally in favor > of > >> having new end to end tests. We need to catch regressions better. > >> > >> Braden, let me know how I can help. > >> > >> > >> > >> > On Nov 7, 2013, at 9:02 AM, Michal Mocny <mmo...@chromium.org> wrote: > >> > > >> > Discussing locally with Braden.. these tests seem to be testing > internal > >> > details instead of expected functionality. Its quite common for valid > >> > patches to break the tests and invalid patches to leave the tests > >> passing. > >> > There have been several occurrences recently where things landed even > >> > though "cordova create" or "plugin add" were hosed, which seems fairly > >> > fundamental to keep working ;) > >> > > >> > > >> >> On Thu, Nov 7, 2013 at 11:57 AM, Michal Mocny <mmo...@chromium.org> > >> wrote: > >> >> > >> >> +1 to testing end-to-end, but I thought thats what BuildBot/Medic > does? > >> >> Likely we want to ship those end-to-end test scripts so users can > test > >> >> changes locally, but I think the tests are are inside CLI now are > meant > >> to > >> >> be unit tests, which yes, have very limited usefulness in isolation, > but > >> >> perhaps shouldn't be entirely replaced? If they really are useless > (not > >> >> sure) then they should be fixed/removed. > >> >> > >> >> > >> >> On Thu, Nov 7, 2013 at 11:40 AM, Braden Shepherdson < > >> bra...@chromium.org>wrote: > >> >> > >> >>> The CLI tests are bad. I propose making them better. > >> >>> > >> >>> The tests are bad for two reasons: > >> >>> 1. They're fragile because the tests depend on exactly the right > >> functions > >> >>> being called, sometimes in the right order. > >> >>> 2. They don't test what we really want, which is that projects get > >> created > >> >>> and all the files are in the right places. > >> >>> > >> >>> I propose letting the tests actually run filesystem and related > calls, > >> >>> instead of always mocking them out. In the simplest form, that means > >> >>> running them on the real filesystem. If that's too slow, we can > >> >>> investigate > >> >>> other alternatives, like using a ramdisk, or using that emulated fs > >> that > >> >>> runs everything in RAM inside Node. > >> >>> > >> >>> Mark and I are planning to work on this, starting with the CLI > tests. > >> The > >> >>> Plugman tests are better in this regard, but could probably stand > to be > >> >>> really called as well. > >> >>> > >> >>> Any thoughts or objections? > >> >>> > >> >>> Braden > >> >> > >> >> > >> >
