One more report. I got suspicious of the rather short runtimes, so I picked the default_doc and ran it at 500k:
bench_R14B04_1.1.1_default_doc.tpl.log 2m19.139s bench_R14B04_1.2.x_default_doc.tpl.log 2m18.875s It seems to me that we need more variation in what we test, more OSs, larger ddocs, like the one Stefan linked to. Can anyone help providing this? Cheers Jan -- On Feb 29, 2012, at 00:23 , Jan Lehnardt wrote: > For Robert Newson, avoiding bulk inserts to populate the dbs: > > bench_R14B04_1.1.1_default_doc.tpl.log 0m19.692s > bench_R14B04_1.2.x_default_doc.tpl.log 0m17.033s > > bench_R14B04_1.1.1_nested_6k.tpl.log 1m31.393s > bench_R14B04_1.2.x_nested_6k.tpl.log 0m42.010s > > bench_R14B04_1.1.1_small_doc.tpl.log 0m8.103s > bench_R14B04_1.2.x_small_doc.tpl.log 0m10.597s > > bench_R14B04_1.1.1_wow.tpl.log 0m33.944s > bench_R14B04_1.2.x_wow.tpl.log 0m27.087s > > (Just R14B04, full logs available on demand) > > Cheers > Jan > -- > > > On Feb 28, 2012, at 23:09 , Jan Lehnardt wrote: > >> Same story, but spinning disk, 5400rpm: >> >> bench_R14B04_1.1.1_default_doc.tpl.log 0m19.175s >> bench_R14B04_1.2.x_default_doc.tpl.log 0m16.821s >> bench_R15B_1.1.1_default_doc.tpl.log 0m13.050s >> bench_R15B_1.2.x_default_doc.tpl.log 0m13.292s >> >> bench_R14B04_1.1.1_nested_6k.tpl.log 1m26.941s >> bench_R14B04_1.2.x_nested_6k.tpl.log 0m39.178s >> bench_R15B_1.1.1_nested_6k.tpl.log 0m47.766s >> bench_R15B_1.2.x_nested_6k.tpl.log 0m31.697s >> >> bench_R14B04_1.1.1_small_doc.tpl.log 1m19.851s >> bench_R14B04_1.2.x_small_doc.tpl.log 1m43.057s >> bench_R15B_1.1.1_small_doc.tpl.log 0m52.249s >> bench_R15B_1.2.x_small_doc.tpl.log 1m8.195s >> >> bench_R14B04_1.1.1_wow.tpl.log 0m29.589s >> bench_R14B04_1.2.x_wow.tpl.log 0m24.867s >> bench_R15B_1.1.1_wow.tpl.log 0m20.171s >> bench_R15B_1.2.x_wow.tpl.log 0m18.800s >> >> Full logs at http://jan.prima.de/slow_couch/rust/ >> >> Cheers >> Jan >> -- >> >> >> On Feb 28, 2012, at 21:22 , Jan Lehnardt wrote: >> >>> >>> # tl;dr: >>> >>> bench_R14B04_1.1.1_default_doc.tpl.log 0m18.749s >>> bench_R14B04_1.2.x_default_doc.tpl.log 0m16.304s >>> bench_R15B_1.1.1_default_doc.tpl.log 0m12.946s >>> bench_R15B_1.2.x_default_doc.tpl.log 0m13.616s >>> >>> bench_R14B04_1.1.1_nested_6k.tpl.log 1m27.267s >>> bench_R14B04_1.2.x_nested_6k.tpl.log 0m37.910s >>> bench_R15B_1.1.1_nested_6k.tpl.log 0m46.963s >>> bench_R15B_1.2.x_nested_6k.tpl.log 0m33.011s >>> >>> bench_R14B04_1.1.1_small_doc.tpl.log 1m17.212s >>> bench_R14B04_1.2.x_small_doc.tpl.log 1m41.383s >>> bench_R15B_1.1.1_small_doc.tpl.log 0m52.858s >>> bench_R15B_1.2.x_small_doc.tpl.log 1m9.043s >>> >>> bench_R14B04_1.1.1_wow.tpl.log 0m29.842s >>> bench_R14B04_1.2.x_wow.tpl.log 0m24.178s >>> bench_R15B_1.1.1_wow.tpl.log 0m20.493s >>> bench_R15B_1.2.x_wow.tpl.log 0m19.584s >>> >>> (Full logs at [5]) >>> >>> >>> # Description >>> >>> All of these are on Mac OS X 10.7.3 on an SSD. >>> >>> I'll be running the same set on spinning disk and then Robert N asked >>> me to populate the DBs not using builk docs. Since that's gonna take >>> a while, I'll probably run this overnight. >>> >>> All of the results are generated by my fork of Jason's slow_couchdb[1] >>> and Filipe's seatoncouch[2]. >>> >>> The changes I've made is have the small_doc test run with 500k instead >>> of 50k docs, added .view files to match the tpl files in >>> seatoncouch/templates/* so we can have similar views use the different >>> doc structures. >>> >>> I also added two scripts to orchestrate the above testing in a more >>> automated fashion. It also allows you to run the full matrix yourself. >>> All you need is set up homebrew allow `brew switch erlang R14B04` and >>> R15B (which is controlled in matrix.sh[3]) and have a git checkout of the >>> CouchDB sources that allow you to do `git checkout 1.1.1` or `1.2.x` >>> (which is controlled in runner.sh[4], adjust the path to the git checkout >>> there as well). >>> >>> matrix.sh also allows you to specify which docs to run. >>> >>> Please shout if you need any more info about this test run or how to >>> run this yourself. >>> >>> >>> # Analysis >>> >>> Inconclusive, I'l like to run this on larger dbs in general to see if >>> there are more differences that shake out and I've yet have to run this >>> on a spinning disk let alone another OS* or more complex view functions >>> or larger design docs (like the one Stefan had). >>> >>> * It shouldn't be too much work to port slow_couchdb to other OSs, I'll >>> definitely be looking into that, but we can do with every bit of help :) >>> >>> So far, I'm happy to conclude that while there are definitely provable >>> differences, that we can live with them. >>> >>> Cheers >>> Jan >>> -- >>> >>> >>> [1]: https://github.com/janl/slow_couchdb >>> [2]: https://github.com/janl/seatoncouch >>> [3]: https://github.com/janl/slow_couchdb/blob/master/matrix.sh >>> [4]: https://github.com/janl/slow_couchdb/blob/master/runner.sh >>> [5]: http://jan.prima.de/slow_couch/ssd/ >>> >>> >>> On Feb 28, 2012, at 18:53 , Filipe David Manana wrote: >>> >>>> Jason, repeated my last test with the 1Kb docs ( >>>> https://gist.github.com/1930804, map function >>>> http://friendpaste.com/5C99aqXocN6N6H1BAYIigs ) to cover branch 1.1.x >>>> as well. Here are the full results (also in >>>> https://gist.github.com/1930807): >>>> >>>> >>>> Before COUCHDB-1186 >>>> >>>> fdmanana 23:21:05 ~/git/hub/slow_couchdb (master)> docs=500000 >>>> batch=5000 ./bench.sh wow.tpl >>>> Server: CouchDB/1.2.0a-a68a792-git (Erlang OTP/R14B03) >>>> {"couchdb":"Welcome","version":"1.2.0a-a68a792-git"} >>>> >>>> [INFO] Created DB named `db1' >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> (....) >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> Building view. >>>> {"total_rows":500000,"offset":0,"rows":[ >>>> {"id":"00051ef7-d735-48d7-9ba8-5a21a86e8d57","key":["dwarf","assassin"],"value":[{"x":31227.35,"y":31529.73},{"x":116667.85,"y":82008.25},{"x":224.11,"y":36652.41},{"x":128565.95,"y":6780.2},{"x":165230.43,"y":176208.63}]} >>>> ]} >>>> >>>> real 5m6.676s >>>> user 0m0.009s >>>> sys 0m0.010s >>>> >>>> >>>> After COUCHDB-1186 >>>> >>>> fdmanana 23:50:07 ~/git/hub/slow_couchdb (master)> docs=500000 >>>> batch=5000 ./bench.sh wow.tpl >>>> Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03) >>>> {"couchdb":"Welcome","version":"1.2.0a-f023052-git"} >>>> >>>> [INFO] Created DB named `db1' >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> (....) >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> Building view. >>>> {"total_rows":500000,"offset":0,"rows":[ >>>> {"id":"00051ef7-d735-48d7-9ba8-5a21a86e8d57","key":["dwarf","assassin"],"value":[{"x":31227.35,"y":31529.73},{"x":116667.85,"y":82008.25},{"x":224.11,"y":36652.41},{"x":128565.95,"y":6780.2},{"x":165230.43,"y":176208.63}]} >>>> ]} >>>> >>>> real 5m1.395s >>>> user 0m0.008s >>>> sys 0m0.010s >>>> >>>> >>>> After COUCHDB-1186 + better queueing patch >>>> (http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w) >>>> >>>> fdmanana 00:14:25 ~/git/hub/slow_couchdb (master)> docs=500000 >>>> batch=5000 ./bench.sh wow.tpl >>>> Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03) >>>> {"couchdb":"Welcome","version":"1.2.0a-f023052-git"} >>>> >>>> [INFO] Created DB named `db1' >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> (....) >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> Building view. >>>> {"total_rows":500000,"offset":0,"rows":[ >>>> {"id":"00051ef7-d735-48d7-9ba8-5a21a86e8d57","key":["dwarf","assassin"],"value":[{"x":31227.35,"y":31529.73},{"x":116667.85,"y":82008.25},{"x":224.11,"y":36652.41},{"x":128565.95,"y":6780.2},{"x":165230.43,"y":176208.63}]} >>>> ]} >>>> >>>> real 4m48.175s >>>> user 0m0.008s >>>> sys 0m0.009s >>>> >>>> >>>> CouchDB branch 1.1.x >>>> >>>> fdmanana 08:16:58 ~/git/hub/slow_couchdb (master)> docs=500000 >>>> batch=5000 ./bench.sh wow.tpl >>>> Server: CouchDB/1.1.2a785d32f-git (Erlang OTP/R14B03) >>>> {"couchdb":"Welcome","version":"1.1.2a785d32f-git"} >>>> >>>> [INFO] Created DB named `db1' >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> (....) >>>> [INFO] Uploaded 5000 documents via _bulk_docs >>>> Building view. >>>> {"total_rows":500000,"offset":0,"rows":[ >>>> {"id":"0001c0a1-edcb-4dbc-aa9d-533c73d980cb","key":["dwarf","assassin"],"value":[{"x":62038.32,"y":105825.29},{"x":90713.13,"y":128570.97},{"x":43836.37,"y":80517.12},{"x":71610.97,"y":143739.99},{"x":86038.39,"y":84731.8}]} >>>> ]} >>>> >>>> real 5m44.374s >>>> user 0m0.008s >>>> sys 0m0.010s >>>> >>>> >>>> Disk model APPLE SSD TS128C, quad core machine, 8Gb of ram. >>>> >>>> >>>> >>>> On Tue, Feb 28, 2012 at 5:17 AM, Jason Smith <j...@apache.org> wrote: >>>>> Forgive the clean new thread. Hopefully it will not remain so. >>>>> >>>>> If you can, would you please clone https://github.com/jhs/slow_couchdb >>>>> >>>>> And build whatever Erlangs and CouchDB checkouts you see fit, and run >>>>> the test. For example: >>>>> >>>>> docs=500000 ./bench.sh small_doc.tpl >>>>> >>>>> That should run the test and, God willing, upload the results to a >>>>> couch in the cloud. We should be able to use that information to >>>>> identify who you are, whether you are on SSD, what Erlang and Couch >>>>> build, and how fast it ran. Modulo bugs. >>>> >>>> >>>> >>>> -- >>>> Filipe David Manana, >>>> >>>> "Reasonable men adapt themselves to the world. >>>> Unreasonable men adapt the world to themselves. >>>> That's why all progress depends on unreasonable men." >>> >> >
