I don't think we should scrap the current tests. I am totally in favor of having new end to end tests. We need to catch regressions better.
Braden, let me know how I can help. > On Nov 7, 2013, at 9:02 AM, Michal Mocny <mmo...@chromium.org> wrote: > > Discussing locally with Braden.. these tests seem to be testing internal > details instead of expected functionality. Its quite common for valid > patches to break the tests and invalid patches to leave the tests passing. > There have been several occurrences recently where things landed even > though "cordova create" or "plugin add" were hosed, which seems fairly > fundamental to keep working ;) > > >> On Thu, Nov 7, 2013 at 11:57 AM, Michal Mocny <mmo...@chromium.org> wrote: >> >> +1 to testing end-to-end, but I thought thats what BuildBot/Medic does? >> Likely we want to ship those end-to-end test scripts so users can test >> changes locally, but I think the tests are are inside CLI now are meant to >> be unit tests, which yes, have very limited usefulness in isolation, but >> perhaps shouldn't be entirely replaced? If they really are useless (not >> sure) then they should be fixed/removed. >> >> >> On Thu, Nov 7, 2013 at 11:40 AM, Braden Shepherdson >> <bra...@chromium.org>wrote: >> >>> The CLI tests are bad. I propose making them better. >>> >>> The tests are bad for two reasons: >>> 1. They're fragile because the tests depend on exactly the right functions >>> being called, sometimes in the right order. >>> 2. They don't test what we really want, which is that projects get created >>> and all the files are in the right places. >>> >>> I propose letting the tests actually run filesystem and related calls, >>> instead of always mocking them out. In the simplest form, that means >>> running them on the real filesystem. If that's too slow, we can >>> investigate >>> other alternatives, like using a ramdisk, or using that emulated fs that >>> runs everything in RAM inside Node. >>> >>> Mark and I are planning to work on this, starting with the CLI tests. The >>> Plugman tests are better in this regard, but could probably stand to be >>> really called as well. >>> >>> Any thoughts or objections? >>> >>> Braden >> >>
