On 13.Dec.2004 09:49AM -0600, Andy Lester wrote:
> I'm in love with (well, maybe just a crush on) the idea of
> having a continual prove in a window while you're running
> tests. I'm just trying to think of how best to do it.
There is an MIT study on using test-driven development in the
classroom, available at:
http://pag.lcs.mit.edu/~mernst/pubs/ct-user-study.pdf
One of the sections (pasted below) describes their tools, which
you might find interesting. Having something like their Emacs
plugin running in a separate window might be fantastic, complete
with diagnostics and (of course) fruity color output.
Also, check out Virginia Tech's Web-CAT (screenshot on page
three of http://xrl.us/ecw3 PDF) for an example testing system
that provides feedback to students on assignments. (The report
is similar to JUnit's output.)
-----------------------------------------------------------------
[excerpt from "An experimental evaluation of continuous testing
during development" by David Saff and Michael D. Ernst]
2 Tools
We have implemented a continuous testing infrastructure for the
Java JUnit testing framework and for the Eclipse and Emacs
development environments. Our JUnit extensions (used by both
plug-ins) persistently record whether a test has ever succeeded
in the past. This permits it to both change the order in which
tests are run and the order in which results are printed. For
instance, regression errors, which are typically more serious,
can be prioritized over unimplemented tests. We have built
continuous testing plug-ins for both Eclipse and Emacs. We focus
here on the Emacs plug-in, which was used in our experiment. We
describe how the user is notified of problems in his or her code
and how the plug-in decides when to run tests. We conclude this
section with a comparison to pre-existing features in Eclipse
and Emacs. Because Emacs does not have a standard notification
mechanism we indicated compilation and test errors in the mode
line. The mode line is the last line of each Emacs text window;
it typically indicates the name of the underlying buffer,
whether the buffer has unsaved modifications, and what Emacs
modes are in use. The Emacs plug-in (a "minor mode" in Emacs
parlance) uses some of the empty space in the mode line. When
there are no errors to report, that space remains blank, but
when there are errors, then the mode line contains text such as
"Compile-errors" or "Regressions:3". The mode line indicates
whether the code cannot be compiled; regression errors have been
introduced (tests that used to give correct answers no longer
do); or some tests are unimplemented (the tests have never
completed correctly). Because space in the mode line is at
a premium, no further details (beyond the number of failing
tests) are provided, but the user can click on the mode line
notification in order to see details about each error. Clicking
shows the errors in a separate window and places the cursor on
the line corresponding to the first error. Additional clicks
navigate to lines corresponding to additional errors and/or to
different frames within a backtrace. The Emacs plug-in performs
testing whenever there is a sufficiently long pause; it does not
require the user to save the code. The Emacs plug-in indicates
errors that would occur if the developer were to save all
modified buffers, 2
then compiled and tested the on-disk version of the code. In
other words, the Emacs plug-in indicates problems with the
developer's current view of the code. The Emacs plug-in
implements testing of modified buffers by transparently saving
them to a separate shadow directory that contains a copy of the
software under development, then performing compilation and
testing in the shadow directory. This approach has the advantage
of providing earlier notification of problems. Otherwise, the
developer would have no possibility of learning of problems
until the next save, which might not occur for a long period.
Notification of inconsistent intermediate states can be positive
if it reinforces that the developer has made an intended change,
or negative if it distracts the developer; see Section 5. The
continuous testing tool represents an incremental advance over
existing technology in Emacs and Eclipse. By default, Emacs
indirectly indicates syntactic problems in code via its
automatic indentation, fontification (coloring of different
syntactic entities in different colors), indication of matching
parentheses, and similar mechanisms. Eclipse provides more
feedback during editing (though less than a full compiler can),
automatically compiles when the user saves a buffer, and
indicates compilation problems both in the text editor window
and in the task list. Our Emacs plugin provides complete
compilation feedback in real time, and both of our plug-ins
provide notification of test errors.
-----------------------------------------------------------------
--
Ian Langworth
Project Guerrilla
Northeastern University
College of Computer and Information Science
