On Thu, Dec 10, 2009 at 10:57 PM, Dirk Pranke <dpra...@chromium.org> wrote:
> We could do this, but we'd have to add logic to track when directories > were "done", and arbitrarily delay printing results about "other" > directories (hence delaying and serializing results). This might end up causing weirdly irregular bursts of output. The irregular bursts of output isn't that bad. (I had a version of run-webkit-test that did this. Unfortunately, perl is not a fun language for me at least, and I have to admit that the perl code I had would have been hard to maintain/fragile.) Worst case, since we > intentionally run the "http" tests first, and they're the long pole in > the run, this might delay printing all the other directories until > near the end. Not a big deal either. My version did this as well. (I started this behavior in my webkit version and talked to Ojan about doing it.) > I'm not sure what the real benefit of this would be. > The benefit is working in a community and understanding how they do things and adapting to that as opposed to trying to push something very different on them. > (A) Have you looked at the new output yet? > (B) Is getting output by directory really that useful? > I understood your description. Having run the webkit version, I much prefer it due to knowing when certain directories are done and knowing what test(s) failed in those directories as the test goes along (even in the parallel version where the failures may be slightly delayed). The output by directory also adapts better to the buildbot output instead of the huge test-by-test list that chromium buildbots have (which takes a while to download when you click the link for stdio). dave > -- Dirk > > On Thu, Dec 10, 2009 at 10:10 PM, David Levin <le...@chromium.org> wrote: > > Actually, you can have a similar output even with the multi-threading. > > You can display the results for one only directory (even though multiple > > directories are being processed at the same time) and when that directory > is > > done, display the results for the next directory until it is done, > etc. The > > ordering of the directories may be different but the output is very > similar > > to what they have now. > > The effect is quite satisfying and clear. > > dave > > On Thu, Dec 10, 2009 at 10:04 PM, Dirk Pranke <dpra...@chromium.org> > wrote: > >> > >> Yes, I did consider that. The fatal flaw in that plan is that the > >> webkit test script is single-threaded and runs through the tests in > >> order. Ours doesn't, and so we can't easily guarantee the same sort of > >> output they have. Eric and I will probably work through this as we > >> upstream the code. I'm actually hoping to get them to adopt my output, > >> but we'll see. > >> > >> -- Dirk > >> > >> On Thu, Dec 10, 2009 at 7:45 PM, David Levin <le...@chromium.org> > wrote: > >> > Have you considered making the output closer to that of WebKit's > >> > run-webkit-tests? > >> > It seems that would ease the hopeful transition to this version > >> > upstream. > >> > dave > >> > On Thu, Dec 10, 2009 at 7:23 PM, Dirk Pranke <dpra...@chromium.org> > >> > wrote: > >> >> > >> >> Hi all, > >> >> > >> >> If you never run the webkit layout tests, you can stop reading. > >> >> > >> >> Otherwise, earlier today I checked in a patch that should make the > >> >> output much less verbose in the normal case. From the CL: > >> >> > >> >> First, a number of log messages have had their levels changed (mostly > >> >> to > >> >> make them quieter). > >> >> > >> >> Second, the script outputs a "meter" that shows progress through the > >> >> test run, which is a one line summary of where it's at current > >> >> (e.g. "parsing expectations", "gathering files". During the actual > test > >> >> execution, the meter displays "%d tests completed as expected, %d > >> >> didn't, > >> >> %d remain". The meter uses carriage returns but no linefeeds, so the > >> >> output > >> >> is overwritten as it progresses. The meter is disabled if --verbose > is > >> >> specified, to avoid unnecessary confusion. > >> >> > >> >> Third, I removed the --find-baselines option. I think I was the only > >> >> one > >> >> using it, and --sources is good enough (but added the baseline for > >> >> the checksum as well as the .png when using --sources). > >> >> > >> >> Fourth, there is a new "--log" option that can be used to provide > finer > >> >> granularity of logging. It accepts a comma-separated list of options, > >> >> like: > >> >> --log 'actual,expected,timing': > >> >> > >> >> "actual": the actual test results (# of failures by type and > timeline) > >> >> "config": the test settings (results dir, platform, etc.) > >> >> "expected": the results we expected by type and timeline > >> >> "timing": test timing results (slow files, total execution, etc.) > >> >> > >> >> All of this information is logged at the logging.info level (if the > >> >> appropriate option is enabled). > >> >> > >> >> Using the --verbose switch will cause all of options to be logged, as > >> >> well > >> >> as the normal verbose output. In addition, the verbose output will > >> >> disable > >> >> the meter (as mentioned above). Note that the "actual" results will > be > >> >> logged > >> >> to stdout, not stderr, for compatibility with the buildbot log > parser. > >> >> > >> >> Finally, the list of unexpected results (if any) will be logged to > >> >> stdout, > >> >> along with a one-line summary of the test run. > >> >> > >> >> The net result is that when run with no command line options (and > when > >> >> no > >> >> tests fail), only one line of output will be produced. > >> >> > >> >> Feedback / problems / questions to me. > >> >> > >> >> Pam, sorry for making all of your examples in your tech talk > >> >> immediately out of date :) > >> >> > >> >> -- Dirk > >> >> > >> >> -- > >> >> Chromium Developers mailing list: chromium-dev@googlegroups.com > >> >> View archives, change email options, or unsubscribe: > >> >> http://groups.google.com/group/chromium-dev > >> > > >> > > > > > > -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
