Re: Slow to add 1 million items

2014-02-10 Thread Brent Bice
On 02/07/2014 02:40 PM, Christopher Wood wrote: On Fri, Feb 07, 2014 at 10:49:29AM -0700, Brent Bice wrote: (SNIP) I've got a few OpenLDAP instances that I use for writing log data to, so write performance is critical, but since I'm building it from log data, absitively, posolutely,

Re: Re: Re: Slow to add 1 million items

2014-02-07 Thread Andrew Eross
Thanks, Ulrich. That made me do a bit searching, there's a nice summary of some of the ext4 performance options here: http://blog.smartlogicsolutions.com/2009/06/04/mount-options-to-improve-ext4-file-system-performance/ Definitely cool ideas (but more academic for us specifically) - I'd be a bit

Re: Slow to add 1 million items

2014-02-07 Thread Brent Bice
On 02/05/2014 11:45 AM, Quanah Gibson-Mount wrote: Ok. I had some wierd perf issues with Ubuntu10 and Ubuntu12. I was able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel (currently 3.10.11-031011-generic on my Ubuntu12 box). Also, it isn't necessary to use ext4 (I use ext2 for

Re: Slow to add 1 million items

2014-02-07 Thread Brent Bice
On 02/04/2014 02:58 AM, Andrew Eross wrote: Hello all, I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list. I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB.

Re: Slow to add 1 million items

2014-02-07 Thread Howard Chu
Brent Bice wrote: On 02/05/2014 11:45 AM, Quanah Gibson-Mount wrote: Ok. I had some wierd perf issues with Ubuntu10 and Ubuntu12. I was able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel (currently 3.10.11-031011-generic on my Ubuntu12 box). Also, it isn't necessary to use

Re: Slow to add 1 million items

2014-02-07 Thread Christopher Wood
On Fri, Feb 07, 2014 at 10:49:29AM -0700, Brent Bice wrote: (SNIP) I've got a few OpenLDAP instances that I use for writing log data to, so write performance is critical, but since I'm building it from log data, absitively, posolutely, guaranteed perfect DB consistency isn't. I can always

Re: Re: Slow to add 1 million items

2014-02-06 Thread Andrew Eross
Hi guys, Ulrich - thanks for the suggestions - btrfs in particular is certainly worth a shot. Quanah - very cool to hear about the 12.04 kernel and ext2 suggestions. thanks! I've just run some new tests on a similar machine with 12.04.4 LTS and a newly installed 3.11.0-15-generic x86_64 kernel.

Antw: Re: Re: Slow to add 1 million items

2014-02-06 Thread Ulrich Windl
Hi! Another suggestion: Dependingon your secondary storage you could experiment with mount option barrier=0: If you have a RAID controller with battery backed-up cache (that guarantees that any data confirmed written to the host will acually be written in the end) it makes no sense to force

Antw: Re: Slow to add 1 million items

2014-02-05 Thread Ulrich Windl
Hi! It's complete nonsense, but did you try to put a database in a RAM disk? This would show the delay that actual secondary storage access costs in addition to primary storage access. Regards, Ulrich Andrew Eross er...@locatrix.com schrieb am 04.02.2014 um 21:52 in Nachricht

Re: Slow to add 1 million items

2014-02-05 Thread Quanah Gibson-Mount
--On Tuesday, February 04, 2014 6:52 PM -0200 Andrew Eross er...@locatrix.com wrote: Thanks, Dieter, Quanah. I've been doing some experimenting with those mdb options. I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:

Re: Slow to add 1 million items

2014-02-05 Thread Andrew Eross
Hi Quanah, Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and the actual file system is ext4 It's a fairly stock 10.04 system, no special config/kernel

Re: Slow to add 1 million items

2014-02-05 Thread Quanah Gibson-Mount
--On Wednesday, February 05, 2014 1:30 PM -0200 Andrew Eross er...@locatrix.com wrote: Hi Quanah, Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux The latest OpenLDAP 2.4.39 All of those tests done with the mdb backend, of course, and

Antw: Re: Slow to add 1 million items

2014-02-05 Thread Ulrich Windl
Andrew Eross er...@locatrix.com schrieb am 05.02.2014 um 16:30 in Nachricht CAL_tfFf2qW5BcT=Xs4uFOSUO=wL0AN=9cyfs+d-xyplitmz...@mail.gmail.com: Hi Quanah, Ubuntu 10.04 LTS Linux 2.6.32-43-generic-pae #97-Ubuntu SMP Wed Sep 5 16:59:17 UTC 2012 i686 GNU/Linux The latest OpenLDAP 2.4.39

Slow to add 1 million items

2014-02-04 Thread Andrew Eross
Hello all, I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list. I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB. I'm using Ubuntu 10.04, and I've tried both the

RE: Slow to add 1 million items

2014-02-04 Thread Chris Card
Hello all, I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list. I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB.

Re: Slow to add 1 million items

2014-02-04 Thread Andrew Eross
Thanks, Chris. Yeah, I hear you on that, but sorry, to be more specific, I was running this test to get an idea of performance for regular LDAP use, and slapadd is a purely offline solution. It would be helpful for a restore, of course, but not equivalent to when our application will be

Re: Slow to add 1 million items

2014-02-04 Thread Dieter Klünter
Am Tue, 4 Feb 2014 10:25:15 + schrieb Chris Card ctc...@hotmail.com: Hello all, I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list. I've been experimenting

Re: Slow to add 1 million items

2014-02-04 Thread Andrew Eross
Hi Dieter, I definitely read the man entries - for hdb it's more complicated, of course, but with mdb, my simple config came directly from the online docs. My understanding is that there's really nothing needed/suggested for tuning parameters for mdb. I've seen there's a 'checkpoint' option

Re: Slow to add 1 million items

2014-02-04 Thread Dieter Klünter
Am Tue, 4 Feb 2014 10:26:40 -0200 schrieb Andrew Eross er...@locatrix.com: Hi Dieter, I definitely read the man entries - for hdb it's more complicated, of course, but with mdb, my simple config came directly from the online docs. My understanding is that there's really nothing

Re: Slow to add 1 million items

2014-02-04 Thread Quanah Gibson-Mount
--On Tuesday, February 04, 2014 10:26 AM -0200 Andrew Eross er...@locatrix.com wrote: I've seen there's a 'checkpoint' option along with 'dbnosync' which can perhaps increase performance at the expense of reliability, but I wouldn't want to use that in production. The db env flags for mdb

Re: Slow to add 1 million items

2014-02-04 Thread Andrew Eross
Thanks, Dieter, Quanah. I've been doing some experimenting with those mdb options. I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time: Base-line, no extra options: 4m8s With writemap enabled: 8m55s With writemap+mapasync