We are pleased to announce the beta release of the forthcoming major
release of the DaCapo benchmark suite. We apologize if you receive
this email more than once.
Please note that this BETA release is NOT SUITABLE AS A RESEARCH
TOOL. The purpose of the release is to invite researchers
(particularly JVM developers) to evaluate the it and give us feedback
(see links below for mailing list and bug trackers). Please be sure
to note known bugs and limitations (see end of release notes) before
sending in feedback.
This will be the first of a number of beta releases. Please check
the web site for updates.
We welcome your feedback and thank many of you for the contributions
you've made already.
Best wishes,
Steve Blackburn, ANU
John Zigman, ANU
Kathryn McKinley, UT Austin
(On behalf of the DaCapo project).
-----
Key information:
1. Source and binary downloads are available here:
https://sourceforge.net/projects/dacapobench/files/dacapobench/9.10-beta0
2. Release notes are available below and here:
http://www.dacapo-bench.org/RELEASE_NOTES.txt
3. Bug tracker and feature requests here:
http://sourceforge.net/projects/dacapobench/develop
http://sourceforge.net/tracker/?group_id=172498&atid=861957
http://sourceforge.net/tracker/?group_id=172498&atid=861960
4. Mailing list here:
https://sourceforge.net/mail/?group_id=172498
5. Regression results here:
http://dacapo.anu.edu.au/regression/sanity/latest/index.html
http://dacapo.anu.edu.au/regression/perf/head.html
-----
dacapo-9.10-beta0
RELEASE NOTES 2009-09-10
************************************************************************
* IMPORTANT: This beta release is NOT SUITABLE AS A RESEARCH TOOL. *
* *
* Please wait for the full release before using the suite for research *
* purposes. The benchmarks in this beta release are: *
* *
* 1) not fully tuned, *
* 2) not fully evaluated, *
* 3) known to have various limitations and bugs, and *
* 4) subject to change without notice. *
* *
************************************************************************
This is the first beta release of the anticipated 9.10 release of the
DaCapo benchmark suite. These notes are structured as follows:
1. Purpose
2. Overview
3. Usage
4. Changes
5. Known problems and limitations
1. Purpose
----------
This beta release exists specifically (and only) to allow community
feedback and contribution to the upcoming major release, and to allow
JVM vendors to test against the suite prior to the final release. We
strongly encourage the community to explore and evaluate these
benchmarks and to send feedback to the dacapo research group via the
mailing list and/or bug tracker. Please check the list of known issues
(below) before sending feedback.
mailing list: dacapobench-research...@lists.sourceforge.net
(subscribe via: http://sourceforge.net/mail/?group_id=172498)
bug tracker:
http://sourceforge.net/tracker/?group_id=172498&atid=861957
or navigate to it from
http://sourceforge.net/projects/dacapobench/develop
We are particularly interested in feedback on the following:
o The decision to add benchmarks "avrora", "batik", "derby",
"sunflow", "tomcat", "tradebeans", and "tradesoap" (see 4.1
below).
o The decision to "retire" benchmarks "antlr", "bloat", "chart",
and "hsqldb" (see 4.2 below).
o The tuning / running time of each of the benchmarks in their
default settings.
o The structure and behavior of the tradebeans, tradesoap and
tomcat client/server benchmarks.
o Any issues associated with building, running and usability of
the suite.
2. Overview
-----------
The DaCapo benchmark suite is slated to be updated every few years.
The 9.10 release will be the first major update of the suite, and is
strictly incompatible with previous releases: new benchmarks have been
added, old benchmarks have been removed, all other benchmarks have
been substantially updated. It is for this reason that in any
published use of the suite, the version of the suite must be
explicitly stated.
The release sees the retirement of a number of single-threaded
benchmarks (antlr, bloat and chart), the replacement of hsqldb by
derby, the addition of six completely new benchmarks, and the upgrade
of all other benchmarks to reflect the current release state of the
applications from which the benchmarks were derived. These changes are
consistent with the original goals of the DaCapo project, which
include the desire for the suite to remain relevant and reflect the
current state of deployed Java applications.
Each of these benchmarks is tested for both performance* and
correctness nightly. Results are available here:
o performance: http://dacapo.anu.edu.au/regression/perf/head.html
o sanity: http://dacapo.anu.edu.au/regression/sanity/latest/
* tradebeans and tradesoap are not yet running performance tests
3. Usage
--------
3.1 Downloading
o Download the binary jar and/or source zip from:
https://sourceforge.net/projects/dacapobench/files/
o Access the source from subversion via
svn co https://dacapobench.svn.sourceforge.net/svnroot/dacapobench
dacapobench
3.2 Running
o Run a benchmark:
java -jar <dacapo-jar-name>.jar <benchmark>
o For usage information, run with no arguments.
3.3 Building
o You must have a working, recent version of ant installed. Change
to the benchmarks directory and then run:
ant -p
for instructions on how to build.
4. Changes
----------
4.1. Additions
avrora: AVRORA is a set of simulation and analysis tools in a
framework for AVR microcontrollers. The benchmark
exhibits a great deal of fine-grained concurrency. The
benchmark is courtesy of Ben Titzer (Sun Microsystems)
and was developed at UCLA.
batik: Batik is an SVG toolkit produced by the Apache
foundation.
The benchmark renders a number of svg files.
derby: Derby is an in-memory database benchmark, using the
Derby database produced by the Apache foundation, and
executing the pseudojdbc workload previously used by
hsqldb. Derby replaces hsqldb.
sunflow: Sunflow is a rendering system for photo-realistic images
based on raytracing engine.
tomcat: Tomcat uses the Apache Tomcat servelet container to run
some sample web applications.
tradebeans: Tradebeans runs the Apache daytrader workload "directly"
(via EJB) within a Geronimo application server.
Daytrader
is derived from the IBM Trade6 benchmark.
tradesoap: Tradesoap is identical to the tradebeans workload, except
that client/server communications is via soap protocols
(and the workloads are reduced in size to compensate the
substantially higher overhead).
Tradebeans and tradesoap were added as a pair specifically to allow
researchers to analyze overheads associated with the widely used soap
protocol.
4.2. Deletions
antlr: Antlr is single threaded and highly repetitive. The
most recent version of jython uses antlr; so antlr
remains represented within the DaCapo suite.
bloat: Bloat is not as widely used as our other workloads
and the code exhibited some pathologies that were
arguably not representive or desirable in a suite that
was to be representive of modern Java applications.
chart: Chart was repetitive and used a framework that appears
not to be as widely used as most of the other DaCapo
benchmarks. The Batik workload has some similarities
with chart (both are render vector graphics), but is
part of a larger heavyly used framework from Apache.
hsqldb: Hsqldb has been replaced by derby, which runs the same
workload but uses a much more widely used database
engine.
4.3. Updates
All other benchmarks have been updated to reflect the latest release
of the underlying application (with the exception of eclipse, where
the update to 3.5 is underway).
4.4. Other Notable Changes
The packaging of the DaCapo suite has been completely re-worked and
the source code is entirely re-organized.
5. Known Issues
---------------
Please consult the bug tracker for a complete and up to date list of
known issues (http://sourceforge.net/tracker/?group_id=172498&atid=861957
).
DaCapo is an open source community project. We welcome all assistance
in addressing bugs and shortcomings in the suite.
A few notable high priority issues are listed here:
5.1 Concurrency
We intend to conduct a concurrency audit, analyzing and documenting
the level of concurrency in each of the workloads. The per-benchmark
documentation available at the commandline (with the -i switch) should
report the level of threading for each benchmark. Although most
benchmarks exhibit significant concurrency, we have made the conscious
decision to continue to include a few single-threaded benchmarks. We
do this because as long as there exist a significant number of single
threaded applications in popular use, the single-threaded performance
of a JVM is important.
5.2 Trade
The trade benchmarks currently do not reliably run beyond one or two
iterations (and consequently don't appear in our peformance regression
results since those regressions perform 10 iterations). The benchmarks
are known to suffer from memory leaks and database deadlocks.
5.3 Eclipse
We are in the process of updating eclipse, but have not yet completed
this, so at present eclipse runs the same version as it did in the
2006 release.
5.4 Derby
We intend to move to using a TCPC workload rather than PseudoJDBC for
derby. We also wish to remove derby's build-time dependency on a 1.4
JVM.
5.5 Tomcat
We intend to make the tomcat workload more interesting. Performacne
results show that tomcat currently has a remarkably flat warm-up curve
when compared to other benchmarks.
