I was thinking about what metrics one *might* use on Scheme
implementations.
This was happening during my fitness class this morning, so we're not
exactly
talking profound thinking. Nonetheless, I came up with the following
list.
A. Academic contribution
- number of scholarly publications in journals and conferences
- number of technical reports and other unrefereed publications
- other systems (Scheme or not) that use ideas introduced in this
implementation
B. Scope
- conformance to standards (RnRS)
- number of useful[1] SRFIs implemented
- support for internationalization
- additional useful features
- support for an object system
C. Development facilities
- presence of an IDE[2]
- editor integration
- quality of error reports
- presence and quality of debugger and profiler
D. Platform support
- FFI powerful enough to support all of (say) OpenGL and a
relational
DBMS
- access to all OS, GUI, etc. facilities without having to write
custom
code
- threads, either provided by the implementation or native
- ability to generate standalone executables
E. Implementation quality
- speed on a standardized benchmark[3]
- number of optimizations performed during compilation, and the
effectiveness
of each
- memory usage on the same benchmark
- ability to pass one or more test suites regarding the items in B
above
- number of platforms supported
- ability to run in an embedded environment, especially one with
limited
resources
- ability to run for extended periods of time without degradation,
e.g.,
as a server
- for implementations whose core is written in C: ability to pass
a static
checker such as PC-Lint, and a runtime analysis tool such as
Purify;
for implementations written in Java or C#, ability to pass
similar tools;
implementations written in Scheme get a pass on this one :)
- number of external dependencies, ease of building from source
- overall design and coding practices; ease of adding new
primitives (not
necessarily the same as FFI)
F. Usability
- documentation quality; including organization, completeness,
examples, etc.
- ease of installation (not quite the same as `building from
source' above)
- release policy; attention paid to backwards compatibility in new
releases
G. Social issues
- availability of application libraries
- size and general activity of user community
I have no doubt omitted your favorite criterion, and for that you are
of course
free to flame me. However, my point is really the obvious one, namely
that no
single figure of merit makes any sense. One person might reject an
implementation
because it runs too slowly, while another might love it because of a
very powerful
debugger (I'm not saying those two features are at all opposed, merely
that it is
possible to imagine implementations that have one but not both
features). Similarly,
an implementation might have been very influential in the past, but
now moribund.
Another implementation might add absolutely nothing new, but still be
extremely
well-engineered, and hence efficient and great to use.
Notes:
[1] No TCP/IP stack can be complete without an implementation of
RFC 1149, the
carrier pigeon protocol.
[2] I'd probably regard the availability of an IDE as a negative.
[3] Of course we'd have to AGREE on a standard benchmark first!
-- v
_______________________________________________
r6rs-discuss mailing list
[email protected]
http://lists.r6rs.org/cgi-bin/mailman/listinfo/r6rs-discuss
