1. Performance degradation due to being Disk IO bound
In this test the benchmark script and CouchDB were running on the same
machine, as the machine is running Windows XP, it's not surprising that it
was Disk IO bound, the default Windows XP disk cache is quite small. I
could not change this setting as it is my work desktop PC. Also being a 32
bit OS it is not able to address all the memory in the computer.
The system info:
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
OS Name Microsoft Windows XP Professional
Version 5.1.2600 Service Pack 3 Build 2600
OS Manufacturer Microsoft Corporation
System Manufacturer Hewlett-Packard
System Model HP Compaq 8100 Elite SFF PC
System Type X86-based PC
Processor x86 Family 6 Model 37 Stepping 5 GenuineIntel ~3192 Mhz
BIOS Version/Date Hewlett-Packard 786H1 v01.05, 9/06/2010
SMBIOS Version 2.6
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume1
Hardware Abstraction Layer Version = "5.1.2600.5512 (xpsp.080413-2111)"
Time Zone E. Australia Standard Time
Total Physical Memory 4,100.00 MB
Page File Space 5.27 GB
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
and the benchmark test output:
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
D:\Python27\Scripts\KVBench>python KVBench.py
Started DataPrime(100) 10/21/15 14:48:53 E. Australia Standard Time
Finished DataPrime(100) 10/21/15 14:58:36 E. Australia Standard Time
DataPrime took 583.640000105 Seconds
Started Phase 1(50) 10/21/15 14:58:36 E. Australia Standard Time
............................................................................................................................................
............................................................................................................................................
............................................................................................................................................
.................................................................................
Finished Phase 1(50) 10/21/15 15:18:26 E. Australia Standard Time
Phase 1 took 1189.6099999 Seconds
Started Phase 2(50) 10/21/15 15:18:26 E. Australia Standard Time
............................................................................................................................................
............................................................................................................................................
............................................................................................................................................
.................................................................................
Finished Phase 2(50) 10/21/15 15:58:32 E. Australia Standard Time
Phase 2 took 2406.04700017 Seconds
Started Phase 3(50) 10/21/15 15:58:32 E. Australia Standard Time
............................................................................................................................................
............................................................................................................................................
............................................................................................................................................
.................................................................................
Finished Phase 3(50) 10/21/15 16:15:37 E. Australia Standard Time
Phase 3 took 1024.60899997 Seconds
Phase 1
All : 5.08275422186
Query 3.1 : 67.6884480004
Query 3.2 : 21.9682700028
Query 3.3 : 0.220407997608
Read : 0.110005330563
Write : 0.106869500081
Phase 2
All : 23.6991666674
Query 3.1 : 69.0734859996
Query 3.2 : 24.866571332
Query 3.3 : 0.182826003551
Read : 0.121418002129
Write : 0.120152003765
Phase 3
All : 5.63298042818
Query 3.1 : 56.6922459965
Query 3.2 : 20.8026160007
Query 3.3 : 0.159700000763
Read : 0.110064198971
Write : 0.106522006989
KV Bench v1.0.3 score: 11.4716337725
------------------------------------------------------------------
------------------------------------------------------------------
----------------------
On Tue, Oct 27, 2015 at 8:18 PM, Garren Smith <[email protected]> wrote:
> Hi Dave,
>
> This is very cool. Do you have the results and the scripts you used to
> benchmarch CouchDB?
>
> Cheers
> Garren
>
> On Thu, Oct 22, 2015 at 3:13 PM, Dave Amies <[email protected]> wrote:
>
> > Hi All,
> >
> > I'm sure by now most of you will have read at least some parts of this
> > guide:
> >
> > http://guide.couchdb.org/draft/performance.html
> >
> > I was reading it the other day and noticed the "Call to Arms" section at
> > the bottom of the page. I don't know if there are already any
> benchmarking
> > tools out there, but I decided to try writing one. Hopefully the one I
> have
> > written will be useful.
> >
> > About my background, for my day job i am a performance tester, usually
> > specialising in Loadrunner, so this project was something to keep my mind
> > occupied while waiting for my test system to be rebuilt. Given this I
> have
> > only spent a few hours on it and so there is probably still room for
> > improvement, this email is about finding out if there is interest or if
> > this will be useful to the CouchDB community, so really should I continue
> > developing this tool, or am I wasting my time?
> >
> > In designing this benchmarking utility I reflected on all the systems I
> > have tested and tried to come up with some common areas where database
> > systems suffer in performance. Then bearing in mind the fundamental
> > differences between traditional databases and NoSQL databases
> (particularly
> > CouchDB) I tried to construct some some common database usage scenarios.
> >
> > The 3 scenarios I came up with are:
> >
> > 1. Write heavy (each user performs 12 writes, 6 reads and 3 searches /
> > index queries)
> > 2. Index / Query / Search heavy (each user performs 1 write, 2 reads
> and
> > 6 searches / index queries)
> > 3. Read Heavy (each user performs 1 writes, 10 reads and 3 searches /
> > index queries)
> >
> > I have tried out my benchmarking tool on a couple of machines so far, in
> > these tests I managed to cause CouchDB to encounter the following
> > situations:
> >
> > 1. Performance degradation due to being Disk IO bound
> > 2. Performance degradation due to being Memory bound
> > 3. Performance degradation due to being CPU bound
> > 4. Couch DB crashed
> > 5. Benchmarking completed successfully and produce a performance score
> >
> > Based on these results I believe I have created an effective tool for
> > benchmarking, so I decided the best next step was to release the tool as
> an
> > open source project, so I created a github project which can be found
> here:
> > https://github.com/damies13/kvbench. Here you will the readme file
> > describes the 3 scenarios in more detail, the benchmark definition or
> > design and also the pre benchmark data priming. You will also find here
> the
> > python script that is the benchmarking tool and some instructions for
> > setting up a couch db database for the benchmarking process.
> >
> > As this is getting long i'll wrap up by noting that I deliberately did
> not
> > use the python couchdb libraries but instead I used the requests library
> > (standard http) and json library because I wanted to keep the code as
> > generic as possible, the intention is that this benchmarking tool should
> be
> > able to be used to benchmarking any key / value store, whether that be a
> > document based NoSQL, and Key Value based NoSQL database or some other
> Rest
> > API / engine (e.g. backed by a traditional database).
> >
> > I look forward to some feed back, hopefully I have created something
> > useful.
> >
> > Sincerely,
> >
> > Dave.
> >
>
