On 8/28/07, Howard Chu <[EMAIL PROTECTED]> wrote: > Quanah Gibson-Mount wrote: > > --On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder > > <[EMAIL PROTECTED]> wrote: > > > >> That's an unfortunate development from oracle considering the > >> presentation that was released showing bdb4.6 to be very fast. > > > > Yup. I believe Howard is working with Sleepycat to find out what was done > > to 4.6 to make it suddenly be 3-4 times slower instead of significantly > > faster as it was in the early releases. > > Yes, it's been a long conversation so far. While their new memory manager > (inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in OpenLDAP) > is a great improvement, their new lock manager is not. > > This may not be an issue on all platforms. On x86 though, they have a hybrid > mutex mechanism which is enabled by default. It uses both assembly language > spinlocks and pthread mutexes, first spinning in the assembly language lock > some number of times before falling back onto the mutex. They claimed that > this > improved performance in their tests, because pthread mutexes can be very > expensive on some platforms. In my tests on x86_64 Linux however, it just > forced CPU usage to 100% (200% actually, dualcore) and slowed down overall > throughput. > > You can give an explicit "--with-mutex=POSIX/pthread" argument when > configuring > BerkeleyDB to avoid the hybrid mutex scheme, in which case performance of BDB > 4.6.19 seems to match what we obtained with BDB 4.6.3. > > If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will > work well for you. The only difference between 4.5.20 and 4.6.1 was in the > memory manager. 4.6.2 just tweaked some portability issues to support BREW > (Qualcomm's cellphone programming environment). 4.6.3 added the ability to > specify different cache priorities per database operation. (I didn't track > what > changed between 4.6.3 and 4.6.18.) > > And for anyone curious - you can read ITS#3851 > http://www.openldap.org/its/index.cgi/Development?id=3851 for the background > on > the problems in BerkeleyDB's memory allocator. While there was a fair amount > of > debate as to whether Jong's proposed solution was of any benefit, it was > pretty > clear that the existing code in BerkeleyDB could be improved. But the changes > in the memory allocator may not visibly affect you if you're running a > small-to-medium sized database. It's only apparent when the total volume of > data is much larger than the BerkeleyDB cache, because that's the condition > that leads to memory fragmentation, and it's the fragmentation that causes the > allocator's performance to degrade. > > The same kind of problem affects the slapd entry cache, when the number of > active entries is much larger than the entry cache, and you're using the glibc > malloc. (At least, for glibc 2.3 and older. I haven't retested with glibc 2.5 > yet.) > > To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the > performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6. > Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a > drop-in replacement for 4.5.20 though.
What do you think your chances are at getting --with-mutex=POSIX/pthread made the default? If not very good, it might be worth mentioning in openldap docs if it's that much of a performance gain.
