Re: ldapsearch performance degradation
Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a caching problem, a disk problem or a network problem. As far as openldap related, there are four caches to watch: 1. the bdb/hdb database (DB_CONFIG, cachesize) 2. the DN cache (dncachesize) 2. the cache of searched and indexed attribute types (idlcachesize) 3. the frontside cache of search results (cachesize) please check slapd.conf whether appropriate sizes are configured, see man slapd-bdb(5) and slapd.conf(5) for more information. But I must admit, a misconfiguration of any of this caches would not lead to such a degrading in presenting search results. An other approach would be to check the caching behaviour of clients, to check the network cache and the disk cache. -Dieter -- -- Tim Dyce Research Computing EPP Group The School of Physics The University of Melbourne +61 3 8344 5462 +61 431 485 166
Re: ldapsearch performance degradation
Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a caching problem, a disk problem or a network problem. As far as openldap related, there are four caches to watch: 1. the bdb/hdb database (DB_CONFIG, cachesize) 2. the DN cache (dncachesize) 2. the cache of searched and indexed attribute types (idlcachesize) 3. the frontside cache of search results (cachesize) please check slapd.conf whether appropriate sizes are configured, see man slapd-bdb(5) and slapd.conf(5) for more information. But I must admit, a misconfiguration of any of this caches would not lead to such a degrading in presenting search results. An other approach would be to check the caching behaviour of clients, to check the network cache and the disk cache. -Dieter -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: ldapsearch performance degradation
Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Thanks Tim On 11/11/10 21:38, Howard Chu wrote: Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a caching problem, a disk problem or a network problem. As far as openldap related, there are four caches to watch: 1. the bdb/hdb database (DB_CONFIG, cachesize) 2. the DN cache (dncachesize) 2. the cache of searched and indexed attribute types (idlcachesize) 3. the frontside cache of search results (cachesize) please check slapd.conf whether appropriate sizes are configured, see man slapd-bdb(5) and slapd.conf(5) for more information. But I must admit, a misconfiguration of any of this caches would not lead to such a degrading in presenting search results. An other approach would be to check the caching behaviour of clients, to check the network cache and the disk cache. -Dieter -- -- Tim Dyce Research Computing EPP Group The School of Physics The University of Melbourne +61 3 8344 5462 +61 431 485 166
Re: ldapsearch performance degradation
Tim Dyce wrote: Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Ah, there's a bug in your script, it's creating the 2nd-level entries with the wrong DN so the DB never had more than 250 entries. Now I've fixed that and run again and can see the behavior you're talking about. It's actually due to a particular design choice: Ordinarily at each level of the tree we keep an index tallying all of the children beneath that point. In back-bdb this index is used for subtree searches and for onelevel searches. (In back-hdb it's only used for onelevel.) However, as a write optimization, for the root entry of the DB, we don't bother to maintain this index, it's simply set to All entries. (Otherwise in the back-bdb case, there is far too much write overhead to maintain this index.) The problem is that All entries is actually a range 1 to N where N is the ID of the last entry in the DB. (And 1 is the ID of the root entry.) As you add entries, N keeps increasing, but 1 stays constant. When you do a subtree search, every entryID in the corresponding index slot is checked. In this case, with a subtree search starting at the root entry, you will always be iterating through every ID from 1 thru N, even though many of those IDs have been deleted, and it takes time for BDB to return no such object for all the deleted IDs. If you do all of your operations under a child entry instead of the database root entry, the performance will be constant. I've already verified this with a modified copy of your test. I can post it if you wish. Thanks Tim On 11/11/10 21:38, Howard Chu wrote: Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a caching problem, a disk problem or a network problem. As far as openldap related, there are four caches to watch: 1. the bdb/hdb database (DB_CONFIG, cachesize) 2. the DN cache (dncachesize) 2. the cache of searched and indexed attribute types (idlcachesize) 3. the frontside cache of search results (cachesize) please check slapd.conf whether appropriate sizes are configured, see man slapd-bdb(5) and slapd.conf(5) for more information. But I must admit, a misconfiguration of any of this caches would not lead to such a
Re: ldapsearch performance degradation
Hi Howard Thank you very much! This explains a *lot* :D For the moment however we have 370 facilities using this information system, and sadly a whole bunch of scripts which will do thier ops from the base. Is there anything else you can suggest we do to work around this? Thanks Tim On 11/11/10 22:45, Howard Chu wrote: Tim Dyce wrote: Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Ah, there's a bug in your script, it's creating the 2nd-level entries with the wrong DN so the DB never had more than 250 entries. Now I've fixed that and run again and can see the behavior you're talking about. It's actually due to a particular design choice: Ordinarily at each level of the tree we keep an index tallying all of the children beneath that point. In back-bdb this index is used for subtree searches and for onelevel searches. (In back-hdb it's only used for onelevel.) However, as a write optimization, for the root entry of the DB, we don't bother to maintain this index, it's simply set to All entries. (Otherwise in the back-bdb case, there is far too much write overhead to maintain this index.) The problem is that All entries is actually a range 1 to N where N is the ID of the last entry in the DB. (And 1 is the ID of the root entry.) As you add entries, N keeps increasing, but 1 stays constant. When you do a subtree search, every entryID in the corresponding index slot is checked. In this case, with a subtree search starting at the root entry, you will always be iterating through every ID from 1 thru N, even though many of those IDs have been deleted, and it takes time for BDB to return no such object for all the deleted IDs. If you do all of your operations under a child entry instead of the database root entry, the performance will be constant. I've already verified this with a modified copy of your test. I can post it if you wish. Thanks Tim On 11/11/10 21:38, Howard Chu wrote: Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a caching problem, a disk problem or a network problem. As far as openldap related, there are four caches to watch: 1. the bdb/hdb database (DB_CONFIG, cachesize) 2. the DN cache
Re: ldapsearch performance degradation
Howard Chu wrote: Tim Dyce wrote: Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Ah, there's a bug in your script, it's creating the 2nd-level entries with the wrong DN so the DB never had more than 250 entries. Now I've fixed that and run again and can see the behavior you're talking about. It's actually due to a particular design choice: Ordinarily at each level of the tree we keep an index tallying all of the children beneath that point. In back-bdb this index is used for subtree searches and for onelevel searches. (In back-hdb it's only used for onelevel.) However, as a write optimization, for the root entry of the DB, we don't bother to maintain this index, it's simply set to All entries. (Otherwise in the back-bdb case, there is far too much write overhead to maintain this index.) The problem is that All entries is actually a range 1 to N where N is the ID of the last entry in the DB. (And 1 is the ID of the root entry.) As you add entries, N keeps increasing, but 1 stays constant. When you do a subtree search, every entryID in the corresponding index slot is checked. In this case, with a subtree search starting at the root entry, you will always be iterating through every ID from 1 thru N, even though many of those IDs have been deleted, and it takes time for BDB to return no such object for all the deleted IDs. If you do all of your operations under a child entry instead of the database root entry, the performance will be constant. I've already verified this with a modified copy of your test. I can post it if you wish. Just for sake of comparison with your original test setup from http://www.openldap.org/lists/openldap-technical/201010/msg00237.html I tweaked the DB_CONFIG parameters to speed up the ldapadds, otherwise it takes too long to run the test. Your cache sizes etc. were all too small... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ newtest.tar.gz Description: GNU Zip compressed data
Re: ldapsearch performance degradation
Tim Dyce wrote: Hi Howard Thank you very much! This explains a *lot* :D For the moment however we have 370 facilities using this information system, and sadly a whole bunch of scripts which will do thier ops from the base. Is there anything else you can suggest we do to work around this? The cheapest workaround is to change your DB config to have an empty suffix. Then the DN is the DB root entry, and your ou=test will be the first child entry. (You'll have to reload the DB of course.) Thanks Tim On 11/11/10 22:45, Howard Chu wrote: Tim Dyce wrote: Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Ah, there's a bug in your script, it's creating the 2nd-level entries with the wrong DN so the DB never had more than 250 entries. Now I've fixed that and run again and can see the behavior you're talking about. It's actually due to a particular design choice: Ordinarily at each level of the tree we keep an index tallying all of the children beneath that point. In back-bdb this index is used for subtree searches and for onelevel searches. (In back-hdb it's only used for onelevel.) However, as a write optimization, for the root entry of the DB, we don't bother to maintain this index, it's simply set to All entries. (Otherwise in the back-bdb case, there is far too much write overhead to maintain this index.) The problem is that All entries is actually a range 1 to N where N is the ID of the last entry in the DB. (And 1 is the ID of the root entry.) As you add entries, N keeps increasing, but 1 stays constant. When you do a subtree search, every entryID in the corresponding index slot is checked. In this case, with a subtree search starting at the root entry, you will always be iterating through every ID from 1 thru N, even though many of those IDs have been deleted, and it takes time for BDB to return no such object for all the deleted IDs. If you do all of your operations under a child entry instead of the database root entry, the performance will be constant. I've already verified this with a modified copy of your test. I can post it if you wish. Thanks Tim On 11/11/10 21:38, Howard Chu wrote: Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have no clue, all i can do is guessing. First let's define the problem: you have measured the presentation of search Results the client side, and you observered an increase of time required to present the results. Mostlikely it is either a
Re: ldapsearch performance degradation
Hi Howard On 11/11/10 23:10, Howard Chu wrote: Tim Dyce wrote: Hi Howard Thank you very much! This explains a *lot* :D For the moment however we have 370 facilities using this information system, and sadly a whole bunch of scripts which will do thier ops from the base. Is there anything else you can suggest we do to work around this? The cheapest workaround is to change your DB config to have an empty suffix. Then the DN is the DB root entry, and your ou=test will be the first child entry. (You'll have to reload the DB of course.) That looks like it will do it :D I've run the test with our production data, and it looks like search times remain static. Thanks very much for all the help! Cheers Tim Thanks Tim On 11/11/10 22:45, Howard Chu wrote: Tim Dyce wrote: Hi Howard, Thanks for the help :D We have been testing in ramdisk as well, to make sure that disk thrashing is not the root cause. If your searches are not running long enough to show up for profiling, increase the number of second level entries until you get something you can profile. Ah, there's a bug in your script, it's creating the 2nd-level entries with the wrong DN so the DB never had more than 250 entries. Now I've fixed that and run again and can see the behavior you're talking about. It's actually due to a particular design choice: Ordinarily at each level of the tree we keep an index tallying all of the children beneath that point. In back-bdb this index is used for subtree searches and for onelevel searches. (In back-hdb it's only used for onelevel.) However, as a write optimization, for the root entry of the DB, we don't bother to maintain this index, it's simply set to All entries. (Otherwise in the back-bdb case, there is far too much write overhead to maintain this index.) The problem is that All entries is actually a range 1 to N where N is the ID of the last entry in the DB. (And 1 is the ID of the root entry.) As you add entries, N keeps increasing, but 1 stays constant. When you do a subtree search, every entryID in the corresponding index slot is checked. In this case, with a subtree search starting at the root entry, you will always be iterating through every ID from 1 thru N, even though many of those IDs have been deleted, and it takes time for BDB to return no such object for all the deleted IDs. If you do all of your operations under a child entry instead of the database root entry, the performance will be constant. I've already verified this with a modified copy of your test. I can post it if you wish. Thanks Tim On 11/11/10 21:38, Howard Chu wrote: Tim Dyce wrote: Hi Dieter, Thanks for the tips on tuning, sadly the problem is still haunting us :( Andrey Kiryanov at CERN has been doing a lot of work on this performance degradation problem as well. He has tried BDB 4.8.30 and OpenLDAP 2.4.23 but the problem is still apparent. I've run the test setup you provided here http://www.openldap.org/lists/openldap-technical/201010/msg00237.html but so far I'm seeing constant (0:00.0 second) results from ldapsearch. Some differences - I used back-hdb, which is going to be superior for a heavy add/delete workload. Also my test DB is running on a tmpfs (RAMdisk). The basic test we are running (sent earlier) creates 100 ou entries in the root, each with 250 child ou entries, then deletes 20-35% of these and re-adds them. For each deletion cycle the ldapsearch performance degrades, taking longer to complete the search each time. The performance is consistent, across restarts of slapd, and tied to the current state of the database. I have tried rsyncing out the database, and returning it later, and the performance is consistent with the number of deletion cycles the database has undergone. The only clue I have is that when dumping the databases which db_dump it's clear that the ordering of the database becomes increasingly less aligned with the order of the output data when doing a full tree search as we are. Which suggests that the database is writing frequently accessed entires too often instead of holding them in cache? I have run cachegrind against the server at 2, 20 and 1000 deletion iterations and the results are very different - http://www.ph.unimelb.edu.au/~tjdyce/callgrind.tar.gz The number of fetches grows massively over time. Anything you guys can suggest would be much appreciated, it's started to affect quite a number of our grid sites. Cheers, Tim On 04/11/10 02:56, Dieter Kluenter wrote: Hi Dieter, I've done some more testing with openldap 2.3 and 2.4, on Redhat and Ubuntu. I even went as far as placing the BDB database directory in a ramdisk. But the performance still seems to degrade over time as data is added then deleted repeatedly from the ldap server. It looks like the BDB database starts to fragment or lose structure over time? I've tried a few DB options that seem to have some impact. Any ideas on what I can do from here? Quite frankly, I have
Re: Asynchronicity
Shankar Anand R wrote: On Mon, Nov 8, 2010 at 5:57 PM, Howard Chu h...@symas.com mailto:h...@symas.com wrote: Shankar Anand R wrote: Hi, Is there any workaround way by which we will be able to do a DIGEST-MD5 - SASL LDAP bind asynchronously? Howard: Is there a plan to have an asynchronous ldap_sasl_interactive_bind in the near future? Any time lines for that? No future plan. It's in CVS as of October 13, so it's already in the past. Great! Thanks for the information. I understand that the openldap project does not follow strict release dates. But I see that there is an average of 3 months between 2.4.x and 2.4.x+1. Can you help me by giving an approximate date when 2.4.24 will be released (with this fix)? No idea. You can help by actually using/testing the code and giving feedback on how it worked out for you. If it still has problems, then it cannot be released sooner. If no one is actually going to use it, then it will not be released. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: olcDbURI error
Quoting Jaap Winius jwin...@umrk.nl: Quoting Pierangelo Masarati masar...@aero.polimi.it: That patch was never committed because the reporter of ITS#6540 said his initial report was not actually relevant for the real issue he was suffering from. Please try that patch and report about its effect; I'll be glad to commit it if it is useful. I'd be happy to do that, but regardless of whether your patch is applied or not, the compile process errors out like this: Oops, my mistake. It has now compiled successfully, with your patch. Result: the error is gone. I suggest you commit your patch ASAP. Thanks, Jaap
Chaining not working
Hi folks,
While testing the current Debian squeeze version of OpenLDAP,
v2.4.23-6, in a provider/consumer syncprov/syncrepl
(refreshAndPersist) configuration, using a patch(1) written by
Pierangelo, I have not been able to get chaining to work.
The consumer, ldaps2, was configured with a referral(2) to the
provider, ldaps1, as well as a chaining configuration(3). A couple of
authzTo rules(4) were added to its entry in the DIT, which immediately
replicated to the consumer, and the provider was configured with an
olcAuthzPolicy directive for to(5). So far, so good.
However, when using ldapmodify on the consumer to test that an entry
in the DIT could actually be modified (the description attr of the
consumer's entry) from there as a result, I got this response:
modifying entry cn=ldaps2,dc=example,dc=com
ldap_modify: Referral (10)
referrals:
ldap://ldaps.example.com/cn=ldaps2,dc=example,dc=com
I know ldapmodify doesn't understand referrals; this is where chaining
should have worked instead. So, I removed the referral from the
consumer's configuration to see what would then happen with the same
command:
modifying entry cn=ldaps2,dc=example,dc=com
ldap_modify: Server is unwilling to perform (53)
additional info: shadow context; no update referral
(shadow context?). In both cases, this shows up in the syslog as a result:
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 fd=19 ACCEPT from
IP=127.0.1.1:43982 (IP=0.0.0.0:389)
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 BIND
dn=cn=admin,dc=example,dc=com method=128
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 BIND
dn=cn=admin,dc=example,dc=com mech=SIMPLE ssf=0
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 RESULT tag=97 err=0 text=
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 MOD
dn=cn=ldaps2,dc=example,dc=com
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 MOD attr=description
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 RESULT tag=103
err=53 text=shadow context; no update referral
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=2 UNBIND
Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 fd=19 closed
Have I made a mistake somewhere, or could this be another bug?
Thanks,
Jaap
1)
ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch
2) LDIF applied to ldaps2 (the consumer) to create the referral to
ldaps1 (the provider) via an alias (ldaps):
-
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcUpdateref
olcUpdateref: ldap://ldaps.example.com
-
3) LDIF applied to ldaps2 to create the chaining configuration:
-
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: {1}back_ldap
dn: olcOverlay={0}chain,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {0}chain
olcChainReturnError: TRUE
dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={1}hdb,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
olcDbURI: ldap://ldaps.example.com
olcDbRebindAsUser: TRUE
olcDbIDAssertBind: bindmethod=simple
binddn=cn=ldaps2,dc=example,dc=com
credentials=bilineatus
mode=self
-
4) LDIF to create a couple of authzTo rules for the consumer:
-
dn: cn=ldaps2,dc=example,dc=com
changetype: modify
add: authzTo
authzTo: {0}dn.regex:^uid=[^,]+,ou=people,dc=example,dc=com$
authzTo: {1}dn.exact:cn=admin,dc=example,dc=com
-
5) LDIF to add an olcAuthzPolicy directive to the provider, ldaps1:
-
dn: cn=config
changetype: modify
add: olcAuthzPolicy
olcAuthzPolicy: to
-
Re: Chaining not working
Jaap Winius wrote: Hi folks, While testing the current Debian squeeze version of OpenLDAP, v2.4.23-6, in a provider/consumer syncprov/syncrepl (refreshAndPersist) configuration, using a patch(1) written by Pierangelo, I have not been able to get chaining to work. The consumer, ldaps2, was configured with a referral(2) to the provider, ldaps1, as well as a chaining configuration(3). A couple of authzTo rules(4) were added to its entry in the DIT, which immediately replicated to the consumer, and the provider was configured with an olcAuthzPolicy directive for to(5). So far, so good. However, when using ldapmodify on the consumer to test that an entry in the DIT could actually be modified (the description attr of the consumer's entry) from there as a result, I got this response: modifying entry cn=ldaps2,dc=example,dc=com ldap_modify: Referral (10) referrals: ldap://ldaps.example.com/cn=ldaps2,dc=example,dc=com I know ldapmodify doesn't understand referrals; this is where chaining should have worked instead. So, I removed the referral from the consumer's configuration to see what would then happen with the same command: modifying entry cn=ldaps2,dc=example,dc=com ldap_modify: Server is unwilling to perform (53) additional info: shadow context; no update referral (shadow context?). In both cases, this shows up in the syslog as a result: Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 fd=19 ACCEPT from IP=127.0.1.1:43982 (IP=0.0.0.0:389) Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 BIND dn=cn=admin,dc=example,dc=com method=128 Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 BIND dn=cn=admin,dc=example,dc=com mech=SIMPLE ssf=0 Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=0 RESULT tag=97 err=0 text= Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 MOD dn=cn=ldaps2,dc=example,dc=com Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 MOD attr=description Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=1 RESULT tag=103 err=53 text=shadow context; no update referral Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 op=2 UNBIND Nov 12 00:48:54 ldaps2 slapd[23862]: conn=1002 fd=19 closed Have I made a mistake somewhere, or could this be another bug? The chain overlay needs to be configured on the frontendDB in order to catch these update referrals. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: German Umlauts are converted and also not accepted
Hallo, Am 10.11.10 15:47, schrieb Christian Manal: Hi, As my name contains a german umlaut I do get an error adding 'Götz Reinicke' for the gecko. Do you mean 'gecos' with that? The 'gecos' attribute from the NIS-schema is an IA5 string [1], which is equal to ASCII. You won't be able to use special characters with that one. jep 'gecos', typo on my side. I was thinking of something like that. Thanks for making that clear to me :-) Also for 'displayName', 'cn' and 'givenName' the 'ö' is automatically replaced by 'oe'. If I change the entry in the ldap by hand in the apache DS, the 'ö' is accepted accept for the geckos. An 'ü' in 'o' is manually accepted to. How is your ldif file encoded? 'cn', 'givenName' and so on are defined as SUB to 'name', which is defined with Directory Sting syntax [2] aka UTF-8. It's from what my text editor says UTF-8. But now I do have also the phenomenon, that in the ldap the umlaut is replaced with a wired character and not replaced with 'oe' I think, that the AD Studio makes something 'funny'. May be. cheers . Götz -- Götz Reinicke IT-Koordinator Tel. +49 7141 969 420 Fax +49 7141 969 55 420 E-Mail goetz.reini...@filmakademie.de Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzende des Aufsichtsrats: Prof. Dr. Claudia Hübner Geschäftsführer: Prof. Thomas Schadt smime.p7s Description: S/MIME Cryptographic Signature
