On 11/28/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:
Mikhail Loenko wrote: > And we need to make sure that a patch that improves DaCapo by 0.1% > does not lower down the SPEC by 10% Absolutely :) > > We need to agree on a target function to optimize, like some weighted mean. It would be useful to understand what SPECjbb is actually testing - can someone describe it?
First of all I wish to cite spec.org: *--------------------* *SPECjbb2005 (Java Server Benchmark)* is SPEC's benchmark for evaluating the performance of server side Java. SPECjbb2005 evaluates the performance of server side Java by emulating a three-tier client/server system (with emphasis on the middle tier). The benchmark exercises the implementations of the JVM (Java Virtual Machine), JIT (Just-In-Time) compiler, garbage collection, threads and some aspects of the operating system. It also measures the performance of CPUs, caches, memory hierarchy and the scalability of shared memory processors (SMPs). SPECjbb2005 provides a new enhanced workload, implemented in a more object-oriented manner to reflect how real-world applications are designed and introduces new features such as XML processing and BigDecimal computations to make the benchmark a more realistic reflection of today's applications. SPECjbb2005 Benchmark Highlights - Emulates a 3-tier system, the most common type of server-side Java application today. - Business logic and object manipulation, the work of the middle tier, predominate. - Clients are replaced by driver threads, database storage by binary trees of objects. - Increasing amounts of workload are applied, providing a graphical view of scalability. -------------------------------------------<end of citation> Really, SPECjbb emulates a set of transactions to several warehouses like "order something, payment, stocks, etc.." SPECjbb measures performance sequentially on 1,2,3,.... threads that allows to evaluate scalability. Also I may comment some details: - SPECjbb doesn't use any database. It's emulated using java.util API. Such implementation measures pure Java environment. - For example - java.math.BigDecimal class is de facto standard representation for financial calculations. Moreover, JDBC maps DECIMAL (SQL) data to BigDecimal class. SPECjbb uses BigDecimal extensively. - Also SPECjbb uses XML processing for emulation logging like it may be done in real enterprise system. Best regards, --- Sergey Kuksenko. Intel Enterprise Solutions Software Division.
