Hi,
Cees has found a bug in our benchmark. We were using rtrim in our select
statement while doing the benchmark and this was forcing postgres to perform
a table scan.
We've corrected the code for this bug. We are reposting results without
rtrim (creating the tables with varchar).
In fact with postgres even ignores the trailing blanks with 'char' datatype.
When we ran these benchmarks at 40 seconds, we could not make out any
difference in the results.
Instead we increased the number of selects to 2000. Here are the new results
between only postgres(6.5.3) and mldbm
postgres
18 wallclock secs ( 1.88 usr + 0.18 sys = 2.06 CPU)
mldbm
3 wallclock secs ( 1.77 usr + 0.21 sys = 1.98 CPU)
Results still compare favourably towards MLDBM.
Summary of our learning from the benchmarks and these discussions
a) We have to use PG 7.1. It is a major improvement over 6.5.3
b) When we need a completely read-only high-performance data structure,
MLDBM is a good option against postgres. This is provided we are able to
cast our database in MLDBM style datastructures.
c) Generic benchmarks may not be useful for most applications we need to
device our own benchmark which represents critical processing requirements
and try out various options.
Thanks to all of you who have contributed to this thread.
Regards,
V Murali S Muthu Ganesh
Differentiated Software Solutions Pvt. Ltd.,
90, 3rd Cross,2nd Main,
Ganga Nagar,
Bangalore - 560 032
Phone : 91 80 3631445, 3431470
Visit us at www.diffsoft.com
- Original Message -
From: Cees Hek [EMAIL PROTECTED]
To: Murali V [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Friday, April 20, 2001 1:45 AM
Subject: [OT] Re: Fast DB access
On Thu, 19 Apr 2001, Murali V wrote:
Hi,
If you read the code more deeply, you'll find that the timeit is only
wrapped around select and not around insert.
We've written the insert code so that in the first round you can
populate
the database.
You comment out the insert code after the first round and run the
benchmark
several times. This would only do select and time select.
Hi Murali,
OK, to start off, I was not specifically aiming my rant at you, I was
replying to someone who had modified your code and was now comparing MySQL
and PostgreSQL, and he was implying that the timings were for inserts and
selects. I took this at face value, and didn't check the code close
enough which I really should have done in the first place.
Connecting this error to an axiom that "Benchmarks are useless" is bad
indeed. Shouldn't we be ironing out errors and runing benchmarks which
are
good.
Perhaps I should have said published benchmarks. In your case, you are
using benchmarks for exactly what they are intended for... Creating a
system that closely resembles your application and putting it through it's
paces. What I find dangerous about publishing benchmarks, is that they
are almost always heavily swayed to a specific application, and most of
the time they show what the user wants them to show.
In your original message, you clain to have a bias against Postgres, and
your benchmark shows that bias. I however am a happy user of postgres,
and am therefore biased towards it. I modified your benchmark script
slightly, and I got the following results (I have include a diff of my
changes at the bottom):
postgres
0 wallclock secs ( 0.02 usr + 0.01 sys = 0.03 CPU)
postgres
0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)
Whereas if I run it with your version I get the following:
postgres
27 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
postgres
27 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)
So what does that tell you about the benchmark? that the postgres part of
this benchmark is useless... It may have given you the answer that you
wanted, but it is misleading to anyone else out there.
This is why there are always flame wars about benchmarking databases (by
the way I think this whole thread has been very civilized and i hope is
stays that way). Invariably the benchmark has missed some critical idea
or optimization which drastically skew the results.
Your recommendation is to pick a DB best suited to your app. But How ??
a) Either by hiring a guru who has seen all kinds of apps with different
DBs
who can give you the answer with which we can run
b) Run a benchmark on critical programs which represent you app across
databases and find what performs best.
I've read too much literature on DB features. All DBs have all features
(except MySQL which does not have commit )
You can't make a thing out of DB literature.
What I would recommend is exactly what you have done in this case. Get
access to any and all the systems that you feel may do the job for you ,
and try them out. Browse the web for other users experiences, but don't
use other peoples benchmarks, because the odds are good that they are
wrong... Create your own, or modify an existing one,
