Sergey Kuksenko wrote:
On 11/28/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:
Mikhail Loenko wrote:
> And we need to make sure that a patch that improves DaCapo by 0.1%
> does not lower down the SPEC by 10%
Absolutely :)
>
> We need to agree on a target function to optimize, like some weighted
mean.
It would be useful to understand what SPECjbb is actually testing - can
someone describe it?
First of all I wish to cite spec.org:
*--------------------*
*SPECjbb2005 (Java Server Benchmark)* is SPEC's benchmark for evaluating
the
performance of server side Java. SPECjbb2005 evaluates the performance of
server side Java by emulating a three-tier client/server system (with
emphasis on the middle tier). The benchmark exercises the
implementations of
the JVM (Java Virtual Machine), JIT (Just-In-Time) compiler, garbage
collection, threads and some aspects of the operating system.
No I/O? Concurrency?
It also
measures the performance of CPUs, caches, memory hierarchy and the
scalability of shared memory processors (SMPs). SPECjbb2005 provides a new
enhanced workload, implemented in a more object-oriented manner to reflect
how real-world applications are designed and introduces new features
such as
XML processing and BigDecimal computations to make the benchmark a more
realistic reflection of today's applications.
Any I/O?
SPECjbb2005 Benchmark Highlights
- Emulates a 3-tier system, the most common type of server-side Java
application today.
- Business logic and object manipulation, the work of the middle tier,
predominate.
- Clients are replaced by driver threads, database storage by binary
trees of objects.
That means the clients are in the same VM. No sockets or such?
- Increasing amounts of workload are applied, providing a graphical
view of scalability.
-------------------------------------------<end of citation>
Really, SPECjbb emulates a set of transactions to several warehouses like
"order something, payment, stocks, etc.."
SPECjbb measures performance sequentially on 1,2,3,.... threads that allows
to evaluate scalability.
without I/O
Also I may comment some details:
- SPECjbb doesn't use any database. It's emulated using java.util API. Such
implementation measures pure Java environment.
Ah. No I/O
- For example - java.math.BigDecimal class is de facto standard
representation for financial calculations. Moreover, JDBC maps DECIMAL
(SQL)
data to BigDecimal class. SPECjbb uses BigDecimal extensively.
- Also SPECjbb uses XML processing for emulation logging like it may be
done
in real enterprise system.
Do you have an idea of what %-age of workload processing in the test
suite is XML?
geir
