Re: OpenLDAP performance and slapindex

[A. Schulze]

Am 15.12.2016 um 21:27 schrieb Real, Elizabeth (392K):

> Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (uid) not indexed
> Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (memberUid) not 
> indexed

I would build an index if these lines are logged.

Maybe the "bdb_equality_candidates" could tell you what kind of index you need.
That's what I don't know...

Andreas

Re: OpenLDAP performance and slapindex

[Hallvard Breien Furuseth]

On 14/12/16 22:05, Real, Elizabeth (392K) wrote:

However, we noticed that while on the ldap client and as a regular user
we type “w” to see who is logged on this machine, it takes a while
before we get a response with the results. If we type “w” again the
response is very fast due to caching.


One way to avoid slow responses is to index whatever you will support
searches for, and then set the "unchecked" limit.  That's the limit
for how many candidate entries a search is willing to examine after
consulting indexes, scope etc.  If there are more, the search fails
immediatly with adminSizeExceeded.

That way you get a quick failure instead of a uselessly large result
slowly - provided you take care to index the attributes the user may
search for.  "Uselessly large" as in - if the user got 100 entries in
response to the "w", he'd likely rather type in another character than
then browse through the list of results.

Re: OpenLDAP performance and slapindex

[Michael Ströder]

On 12/15/2016 10:01 PM, Quanah Gibson-Mount wrote:
> --On Thursday, December 15, 2016 8:27 PM + "Real, Elizabeth (392K)"
>  wrote:
> 
>> # less /var/log/ldap.log | grep index
>> Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (uid) not
>> indexed
> 
> As I noted previously, you need to look at the actual search being
> executed.  This is because, as Michael noted, indexing is not
> necessarily always the best solution performance wise.  I'd guess that
> you likely need to add indices on those attributes, but I can't say for
> certain w/o the corresponding log lines that caused those warnings to be
> generated.  And without those log lines, it is not possible to determine
> what /type/ of indexing should be used if it is proven necessary.

Despite my own cautious words recommending careful examination of logs I
would always add these indexes for NSS-LDAP clients with classic RFC
2307 schema:

index uid eq
index memberUid eq

Ciao, Michael.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: OpenLDAP performance and slapindex

[Quanah Gibson-Mount]

--On Thursday, December 15, 2016 8:27 PM + "Real, Elizabeth (392K)" 
 wrote:



# less /var/log/ldap.log | grep index



Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (uid) not
indexed



Hi Liz,

As I noted previously, you need to look at the actual search being 
executed.  This is because, as Michael noted, indexing is not necessarily 
always the best solution performance wise.  I'd guess that you likely need 
to add indices on those attributes, but I can't say for certain w/o the 
corresponding log lines that caused those warnings to be generated.  And 
without those log lines, it is not possible to determine what /type/ of 
indexing should be used if it is proven necessary.


Regards,
Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: OpenLDAP performance and slapindex

[Real, Elizabeth (392K)]

Quanah,

I found these entries on both ldap server logs. So do these entries mean that 
slapindex needs to be executed?

# less /var/log/ldap.log | grep index

Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (uid) not indexed
Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (memberUid) not 
indexed
Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (uid) not indexed
Dec 15 12:22:01  slapd[27852]: <= bdb_equality_candidates: (memberUid) not 
indexed

Thank you,
Liz

From: Quanah Gibson-Mount <qua...@symas.com>
Reply-To: Quanah Gibson-Mount <qua...@symas.com>
Date: Wednesday, December 14, 2016 at 1:21 PM
To: "Real, Elizabeth (392K)" <elizabeth.r...@jpl.nasa.gov>, 
"openldap-technical@openldap.org" <openldap-technical@openldap.org>
Subject: Re: OpenLDAP performance and slapindex

--On Wednesday, December 14, 2016 9:05 PM + "Real, Elizabeth (392K)"
<elizabeth.r...@jpl.nasa.gov<mailto:elizabeth.r...@jpl.nasa.gov>> wrote:



Hello,



I'm running OpenLDAP 2.4.40

Upgrade immediately to 2.4.44.  The 2.4.40 release was seriously flawed and
had numerous critical bugs around replication.

In looking at ways to improve performance of ldap, is slapindex a tool to
use? Meaning after importing users accounts from an older ldap, should
slapindex be ran? Or is this tool is used when you want to modify/add
entries found in the slapd (cn=config.ldif) and database
((olcDatabase={2}hdb.ldif) configuration files not when users are added
to the ldap database.

Stop using back-hdb.  Switch to back-mdb.

<https://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/>

If you already had indices defined for the database when you did the
import, there is no reason to run slapindex.  Slapindex should only be used
if you've added a /new/ index to your database.  If you are using cn=config
and add a new index, there will be a background thread that will eventually
index the entire DB.

Whether or not you need to add indexing for given attribute(s) can be
discovered by examining the logs generated by slapd, which will file a
complaint if there is a search performed on an unindexed attribute.  You
would need to examine the specific search to see what type(s) of indexing
may be required.

If slapindex is not a tool to use to monitor performance, what do you
recommend?

slapindex has never been a tool to monitor anything.  It is purely for
(re)generating indices on an existing DB.

Hope that helps!

Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

Re: OpenLDAP performance and slapindex

[Michael Ströder]

On 12/14/2016 10:05 PM, Real, Elizabeth (392K) wrote:
> However, we noticed that while on the ldap client and as a regular user
> we type “w” to see who is logged on this machine, it takes a while
> before we get a response with the results. If we type “w” again the
> response is very fast due to caching.
> [..]
> In looking at ways to improve performance of ldap, is slapindex a tool
> to use?

I'm not sure whether changing indexing configuration is the right way to
really improve things. Hint: Adding indexes can make search performance
worse.

So you should first carefully analyzes which LDAP search queries are
sent to the LDAP server.

You also did not provide any information about the LDAP-enabled NSS
client used and its configuration (which is better discussed on the
mailing list of that software though).

Ciao, Michael.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: OpenLDAP performance and slapindex

[Quanah Gibson-Mount]

--On Wednesday, December 14, 2016 9:36 PM + "Real, Elizabeth (392K)" 
 wrote:





Thank you for the quick response.



I have spent a lot of time configuring version 2.4.40. Is there a guide
on how to upgrade to 2.4.44 and switch to back-mdb? we rolled out 2.4.40
today and would need to plan the next upgrade.


As long as it is linked to the same BDB libraries as your 2.4.40 build, it 
is simply a drop in binary upgrade.



What tool do you recommend for monitoring ldap performance?


I suggest applying the operation duration patch:


That will tell you exactly how long each operation is taking in slapd, as 
long as you have "stats" logging enabled.  Then you can tell if the issue 
is at the slapd server, or if there is something else in between that is 
causing the delay that you see.  My suspicion would be that there is 
something else interfering.


As for migrating from hdb to mdb, it's generally straight forward.  You'd 
need to export your data database to LDIF, and then export cn=config, and 
change the HDB specific bits to MDB, and drop the parameters specific to 
HDB that aren't required for MDB.


If you want to see a basic cn=config templated configuration, you can look 
at:




where @BACKEND@ can be any of bdb/hdb/mdb, and tweak as necessary from 
there.


--Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: OpenLDAP performance and slapindex

[Real, Elizabeth (392K)]

Thank you for the quick response.

I have spent a lot of time configuring version 2.4.40. Is there a guide on how 
to upgrade to 2.4.44 and switch to back-mdb? we rolled out 2.4.40 today and 
would need to plan the next upgrade.

I did already have indices defined for the database when I did the import.

What tool do you recommend for monitoring ldap performance?

Thank you,
Liz

From: Quanah Gibson-Mount <qua...@symas.com>
Reply-To: Quanah Gibson-Mount <qua...@symas.com>
Date: Wednesday, December 14, 2016 at 1:21 PM
To: "Real, Elizabeth (392K)" <elizabeth.r...@jpl.nasa.gov>, 
"openldap-technical@openldap.org" <openldap-technical@openldap.org>
Subject: Re: OpenLDAP performance and slapindex

--On Wednesday, December 14, 2016 9:05 PM + "Real, Elizabeth (392K)"
<elizabeth.r...@jpl.nasa.gov<mailto:elizabeth.r...@jpl.nasa.gov>> wrote:



Hello,



I'm running OpenLDAP 2.4.40

Upgrade immediately to 2.4.44.  The 2.4.40 release was seriously flawed and
had numerous critical bugs around replication.

In looking at ways to improve performance of ldap, is slapindex a tool to
use? Meaning after importing users accounts from an older ldap, should
slapindex be ran? Or is this tool is used when you want to modify/add
entries found in the slapd (cn=config.ldif) and database
((olcDatabase={2}hdb.ldif) configuration files not when users are added
to the ldap database.

Stop using back-hdb.  Switch to back-mdb.

<https://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/>

If you already had indices defined for the database when you did the
import, there is no reason to run slapindex.  Slapindex should only be used
if you've added a /new/ index to your database.  If you are using cn=config
and add a new index, there will be a background thread that will eventually
index the entire DB.

Whether or not you need to add indexing for given attribute(s) can be
discovered by examining the logs generated by slapd, which will file a
complaint if there is a search performed on an unindexed attribute.  You
would need to examine the specific search to see what type(s) of indexing
may be required.

If slapindex is not a tool to use to monitor performance, what do you
recommend?

slapindex has never been a tool to monitor anything.  It is purely for
(re)generating indices on an existing DB.

Hope that helps!

Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

Re: OpenLDAP performance and slapindex

[Quanah Gibson-Mount]

--On Wednesday, December 14, 2016 9:05 PM + "Real, Elizabeth (392K)" 
 wrote:





Hello,



I'm running OpenLDAP 2.4.40


Upgrade immediately to 2.4.44.  The 2.4.40 release was seriously flawed and 
had numerous critical bugs around replication.



In looking at ways to improve performance of ldap, is slapindex a tool to
use? Meaning after importing users accounts from an older ldap, should
slapindex be ran? Or is this tool is used when you want to modify/add
entries found in the slapd (cn=config.ldif) and database
((olcDatabase={2}hdb.ldif) configuration files not when users are added
to the ldap database.


Stop using back-hdb.  Switch to back-mdb.



If you already had indices defined for the database when you did the 
import, there is no reason to run slapindex.  Slapindex should only be used 
if you've added a /new/ index to your database.  If you are using cn=config 
and add a new index, there will be a background thread that will eventually 
index the entire DB.


Whether or not you need to add indexing for given attribute(s) can be 
discovered by examining the logs generated by slapd, which will file a 
complaint if there is a search performed on an unindexed attribute.  You 
would need to examine the specific search to see what type(s) of indexing 
may be required.



If slapindex is not a tool to use to monitor performance, what do you
recommend?


slapindex has never been a tool to monitor anything.  It is purely for 
(re)generating indices on an existing DB.


Hope that helps!

Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

OpenLDAP performance and slapindex

[Real, Elizabeth (392K)]

Hello,

I’m running OpenLDAP 2.4.40 on two multi-master servers with replication 
enabled. Today we switched over to this instance, imported all users from the 
old ldap server, and configured an ldap client to use this instance.

Everything is working well, users are authenticating and able to access their 
assigned groups.

However, we noticed that while on the ldap client and as a regular user we type 
“w” to see who is logged on this machine, it takes a while before we get a 
response with the results. If we type “w” again the response is very fast due 
to caching.

In looking at ways to improve performance of ldap, is slapindex a tool to use? 
Meaning after importing users accounts from an older ldap, should slapindex be 
ran? Or is this tool is used when you want to modify/add entries found in the 
slapd (cn=config.ldif) and database ((olcDatabase={2}hdb.ldif) configuration 
files not when users are added to the ldap database.

If slapindex is not a tool to use to monitor performance, what do you recommend?

Thank you,
Liz