RE: Index seems to return wrong amount of candidate causing really poor search performance

[Bliss, Aaron]

First I apologize for posting a non-technical question / follow up to this 
list, however I can speak for the high value add that having official support 
for OpenLDAP that the Symas team offers.  Like most folks on this list, we have 
a great deal of in house expertise on many software stacks including directory 
services in general.  With that said, the Symas team members are great to work 
with, literally write the software that we are asking questions about and as a 
result have many years of experience and expertise that is easy to reach for.  
IMO the support that the Symas org puts forward is a model that vendors should 
be striving for.

Best,
Aaron

-Original Message-
From: Quanah Gibson-Mount  
Sent: Wednesday, September 2, 2020 2:45 PM
To: Howard Chu ; chrichards...@gmail.com; 
openldap-technical@openldap.org
Subject: Re: Index seems to return wrong amount of candidate causing really 
poor search performance


Warning: This email is from outside the company. Be careful clicking links or 
attachments.



--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu 
wrote:

>> The depth is the same, the values are different from the actual that 
>> we use. I cannot share the actual values without disclosing internal 
>> details.
>
> There's not a lot we can do without being able to reproduce the issue 
> and see what's going on.
>
> You could try starting with a mostly empty DB, adding just the 
> offending value, looking at the filter debug output for a lookup on 
> that, and see if it looks sensible first.
>
> is the attribute abc=foo actually unique, or are there multiple 
> occurrences of it in the DB?


I would optionally note that Symas does provide support contracts for OpenLDAP 
and has NDAs if this is a mission critical problem for your company. As Howard 
notes, without a reproduction case, it's virtually impossible for us to help 
here.

One option may be to see if you can reproduce the same problem after renaming 
the attribute to something you can disclose, combined with values that are the 
same exact length but are simillarly obfuscated to see if you can reproduce the 
issue that way.  If so, you should be able to share those details.

Looking at the open bugs, I wonder if it is 


Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:


--
The information contained in this message may be privileged, confidential and 
protected from disclosure. If the reader of this message is not the intended 
recipient, or an employee or agent responsible for delivering this message to 
the intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. If you 
have received this communication in error, please notify your representative 
immediately and delete this message from your computer. Thank you.

Re: Index seems to return wrong amount of candidate causing really poor search performance

[Quanah Gibson-Mount]

--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu  
wrote:



The depth is the same, the values are different from the actual that we
use. I cannot share the actual values without disclosing internal
details.


There's not a lot we can do without being able to reproduce the issue and
see what's going on.

You could try starting with a mostly empty DB, adding just the offending
value, looking at the filter debug output for a lookup on that, and see
if it looks sensible first.

is the attribute abc=foo actually unique, or are there multiple
occurrences of it in the DB?



I would optionally note that Symas does provide support contracts for 
OpenLDAP and has NDAs if this is a mission critical problem for your 
company. As Howard notes, without a reproduction case, it's virtually 
impossible for us to help here.


One option may be to see if you can reproduce the same problem after 
renaming the attribute to something you can disclose, combined with values 
that are the same exact length but are simillarly obfuscated to see if you 
can reproduce the issue that way.  If so, you should be able to share those 
details.


Looking at the open bugs, I wonder if it is 



Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: Index seems to return wrong amount of candidate causing really poor search performance

[Howard Chu]

chrichards...@gmail.com wrote:
> Quanah Gibson-Mount wrote:
>> --On Tuesday, September 1, 2020 11:18 AM + chrichardso27(a)gmail.com 
>> wrote:
>>
>>>
>>> "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc
>>>  tory,dc=example,dc=org))" 
>>
>> foo and bar are the actual real values?  dc=example,dc=org is your domain?
> 
> The depth is the same, the values are different from the actual that we use. 
> I cannot share the actual values without disclosing internal details.

There's not a lot we can do without being able to reproduce the issue and see 
what's going on.

You could try starting with a mostly empty DB, adding just the offending value, 
looking at the
filter debug output for a lookup on that, and see if it looks sensible first.

is the attribute abc=foo actually unique, or are there multiple occurrences of 
it in the DB?

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: Index seems to return wrong amount of candidate causing really poor search performance

[chrichardso27]

Quanah Gibson-Mount wrote:
> --On Tuesday, September 1, 2020 11:18 AM + chrichardso27(a)gmail.com 
> wrote:
> 
> > 
> > "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc
> >  tory,dc=example,dc=org))" 
> 
> foo and bar are the actual real values?  dc=example,dc=org is your domain?

The depth is the same, the values are different from the actual that we use. I 
cannot share the actual values without disclosing internal details.

> 
> > >
> > > b) The schema definition of the attribute in question.
> > 
> >  attributetype (
> >1.3.6.1.4.1.14761.1.26
> >NAME 'abc'
> >DESC 'A description'
> >EQUALITY distinguishedNameMatch
> >SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' ) 
> 
> abc is your actual attribute name?

Same as above, I cannot disclose the actual values but to give it a bit more 
meaning, lets say that the attribute name could be for example "userSystemDN".

> 
> And don't waste time with back-bdb, it's deprecated and removed from 2.5+.

This is known. However, this happens to be a critical issue at the moment with 
BDB. Futhermore, I can reproduce the problem with MDB but the search filter 
execution time drops to around 2 seconds. Still, in a high volume system, this 
is not acceptable. Other values in this same attribute perform in tens of 
milliseconds. The one specific value (which I cannot disclose here) is 
extremely slow compared to similar data, just with different DN value.

> 
> Regards,
> Quanah
> 
> --
> 
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> 

Re: Index seems to return wrong amount of candidate causing really poor search performance

[chrichardso27]

Howard Chu wrote:
> chrichardso27(a)gmail.com wrote:
> >  Quanah Gibson-Mount wrote:
> > > --On Sunday, August 30, 2020 4:48 PM + chrichardso27(a)gmail.com 
> > > wrote:
> > >
> > >>  Search with filter "(abc=cn=foo,dc=bar)" returns close to
> > >> the amount of
> > >>  entries in the database (2M) as candidates, and is somewhat equally slow
> > >>  than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15
> > seconds.
> > >>
> > >>  However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of
> > the
> > >>  index of abc and performs reasonably fast (1-2 seconds).
> > >>
> > >>  This is rather weird and I have no clue on what might be causing the
> > >>  issue. 
> > > Can you provide:
> > >
> > > a) The actual queries (filter, base, etc) 
> Have you looked at the slapd debug output with FILTER output enabled?

Debug log using mdb (which experiences the same poor usage of index)

  begin get_filter
  AND
  begin get_filter_list
  begin get_filter
  EQUALITY
  end get_filter 0
  begin get_filter
  EQUALITY
  end get_filter 0
  end get_filter_list
  end get_filter 0
  => mdb_filter_candidates
OR
  => mdb_list_candidates 0xa1
  => mdb_filter_candidates
EQUALITY
  <= mdb_filter_candidates: id=0 first=0 last=0
  => mdb_filter_candidates
AND
  => mdb_list_candidates 0xa0
  => mdb_filter_candidates
EQUALITY
  <= mdb_filter_candidates: id=-1 first=2 last=2051388
  => mdb_filter_candidates
EQUALITY
  <= mdb_filter_candidates: id=-1 first=39 last=2051956
  <= mdb_list_candidates: id=-1 first=39 last=2051388
  <= mdb_filter_candidates: id=-1 first=39 last=2051388
  <= mdb_list_candidates: id=-1 first=39 last=2051388
  <= mdb_filter_candidates: id=-1 first=39 last=2051388
  => test_filter
  AND
  => test_filter_and
  => test_filter
  EQUALITY
  <= test_filter 6
  => test_filter
  EQUALITY
  <= test_filter 6
  <= test_filter_and 6
  <= test_filter 6
  => test_filter
  AND
  => test_filter_and
  => test_filter
  EQUALITY
  <= test_filter 5
  <= test_filter_and 5
  <= test_filter 5
  => test_filter
  AND
  => test_filter_and
  => test_filter
  EQUALITY

And this repeats on and on.

Furthermore, with -d 1 and -d 32 I get:

   => mdb_equality_candidates (objectClass)
   => key_read
   <= mdb_index_read 2051387 candidates
   <= mdb_equality_candidates: id=-1, first=2, last=2051388
   <= mdb_filter_candidates: id=-1 first=2 last=2051388
   => mdb_filter_candidates
   EQUALITY
   => mdb_equality_candidates (abc)
   => key_read
   <= mdb_index_read 2051918 candidates

Again, almost all entries in the database are considered as candidates for the 
problematic attribute. However, this is not the case as there are only few 
entries having that attribute specified.

> >  
> >  Using ldapsearch
> >  
> >  ldapsearch \
> >  -L \
> >  -x \
> >  -H ldap://localhost:1389 \
> >  -D "cn=admin,cn=directory,dc=example,dc=org" \
> >  -w secret \
> >  -b cn=directory,dc=example,dc=org \
> > 
> > "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
> >  
> >  Note that in this specific case, Rebuilding a database from scratch with a 
> > different data
> > does not reveal to problem. I have a specific LDIF exported using slapcat 
> > (which I sadly
> > cannot share) with which I can reproduce the problem each time.
> >  
> >  One thing that I have not mentioned previously is that we have a two node 
> > system where to
> > other node is a failover node but is continuously kept up to date using 
> > syncrepl.
> >  
> >  I'm just doing random guesses here as I'm running out of ideas:
> >  
> >  - Could this be a syncrepl issue
> >  - Could there be a bug how OpenLDAP decides the candidates when processing 
> > the respective
> > index
> >  
> >  Anyways, again, any ideas would be greatly appreciated! Few more things I 
> > have tried
> > during these few days:
> >  
> >  - Restore backup prior the incident (works of course but not a solution at 
> > this point as
> > the root cause remains to be unknown)
> >  - With bdb, use linearindex (didn't fix the issue)
> >  - With bdb, set dbpagesize for the index files (didn't succeed as the 
> > slapadd runs
> > out of memory for some reason...)
> >  
> > >
> > > b) The schema definition of the attribute in question.
> >  
> >  attributetype (
> >1.3.6.1.4.1.14761.1.26
> >NAME 'abc'
> >DESC 'A description'
> >EQUALITY distinguishedNameMatch
> >SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )

Re: groupOfNames vs. groupOfUniqueNames

[Michael Ströder]

On 9/2/20 6:57 PM, Quanah Gibson-Mount wrote:
> --On Wednesday, September 2, 2020 12:11 PM +0200 Olaf Hopp
>  wrote:
>> we are at the point of reorganising our LDAP.
>> Currently we only have posixGroups, but in future we also want to support
>> groupOfNames or groupOfUniqueNames
>> My question what is the common sense of usage ?
>> groupOfNames or groupOfUniqueNames ?
>>
>> I know your answers, you will say "it depends on your applications"
>> but currently I have no application using it. All my current applications
>> use my posixGroups. I just want to extend my LDAP for future use cases.
> 
> I generally reocommend groupOfNames for LDAP groups, which is a
> different concept than *NIX posix groups.

In opposite to some other LDAP servers OpenLDAP's slapd support
inheriting an object class from multiple parent classes.

This can be used to solve this problem with a hybrid group schema:

https://gitlab.com/ae-dir/ansible-ae-dir-server/-/blob/master/files/schema/ae-dir.schema#L317

groupOfEntries is used to allow empty groups without members.

And of course you have to ensure that attributes 'member' and
'memberUid' are in sync.

Ciao, Michael.

Re: groupOfNames vs. groupOfUniqueNames

[Dieter Klünter]

Am Wed, 2 Sep 2020 11:11:56 +0200
schrieb Olaf Hopp :

> Hi everybody,
> 
> we are at the point of reorganising our LDAP.
> Currently we only have posixGroups, but in future we also want to
> support groupOfNames or groupOfUniqueNames
> My question what is the common sense of usage ?
> groupOfNames or groupOfUniqueNames ?
> 
> I know your answers, you will say "it depends on your applications"
> but currently I have no application using it. All my current
> applications use my posixGroups. I just want to extend my LDAP for
> future use cases.
> 
> So what to take  : groupOf Names or groupOfUniqueNames besides
> posixGroup ?

I would vote for groupOfnames. If you prefer groupOfUniqueNames you
should provide uniqueness.

https://ldapwiki.com/wiki/GroupOfUniqueNames%20vs%20groupOfNames
https://ldapwiki.com/wiki/UniqueMember

The use of posixgroup depends on your requirements.

-Dieter

-- 
Dieter Klünter | Systemberatung
http://sys4.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E

Re: groupOfNames vs. groupOfUniqueNames

[Quanah Gibson-Mount]

--On Wednesday, September 2, 2020 12:11 PM +0200 Olaf Hopp 
 wrote:



Hi everybody,

we are at the point of reorganising our LDAP.
Currently we only have posixGroups, but in future we also want to support
groupOfNames or groupOfUniqueNames
My question what is the common sense of usage ?
groupOfNames or groupOfUniqueNames ?

I know your answers, you will say "it depends on your applications"
but currently I have no application using it. All my current applications
use my posixGroups. I just want to extend my LDAP for future use cases.


I generally reocommend groupOfNames for LDAP groups, which is a different 
concept than *NIX posix groups.


Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: memberof Overlay not showing in base search

[Michal Soltys]

On 9/2/20 3:26 PM, Umar Draz wrote:

Hi,

I am running OpenLDAP server on Ubuntu 18.

*ldapsearch -Y external -H ldapi:/// -b dc=example,dc=com memberOf*

# udraz, Users, example.com 
dn: uid=udraz,ou=Users,dc=example,dc=com
memberOf: cn=developers,ou=Users,dc=example,dc=com

Would you please help me how to solve this



memberOf is an operational attribute; you either have to specify it 
directly or use + to return all operationals.


It's mentioned in ldapsearch manual as well.

Re: memberof Overlay not showing in base search

[Dieter Klünter]

Am Wed, 2 Sep 2020 18:26:52 +0500
schrieb Umar Draz :

> Hi,
> 
> I am running OpenLDAP server on Ubuntu 18.
> 
> The memberOf attribute is not showing in ldap simple search, if I do
> the following then memberOf attribute is hidden.
> 
> *ldapsearch -Y external -H ldapi:/// -b dc=example,dc=com*
> # udraz, Users, example.com 
> dn: uid=udraz,ou=Users,dc=example,dc=com
> objectClass: top
> objectClass: person
> objectClass: organizationalPerson
> objectClass: inetOrgPerson
> objectClass: posixAccount
> objectClass: shadowAccount
> uid: udraz
> sn: Draz
> givenName: Umar
> mail: ud...@example.com
> cn: Umar Draz
> displayName: Umar Draz
> uidNumber: 5000
> gidNumber: 5000
> gecos: Umar Draz
> loginShell: /bin/bash
> homeDirectory: /home/udraz
> 
> But if I do the following then memberOf attribute appear
> 
> *ldapsearch -Y external -H ldapi:/// -b dc=example,dc=com memberOf*
> # udraz, Users, example.com
> dn: uid=udraz,ou=Users,dc=example,dc=com
> memberOf: cn=developers,ou=Users,dc=example,dc=com
> 
> Would you please help me how to solve this

The memberof attribute type is a, on the fly generated, operational
attribute.

-Dieter

-- 
Dieter Klünter | Systemberatung
http://sys4.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E

groupOfNames vs. groupOfUniqueNames

[Olaf Hopp]

Hi everybody,

we are at the point of reorganising our LDAP.
Currently we only have posixGroups, but in future we also want to support
groupOfNames or groupOfUniqueNames
My question what is the common sense of usage ?
groupOfNames or groupOfUniqueNames ?

I know your answers, you will say "it depends on your applications"
but currently I have no application using it. All my current applications
use my posixGroups. I just want to extend my LDAP for future use cases.

So what to take  : groupOf Names or groupOfUniqueNames besides posixGroup ?

Regards,
Olaf


--
Karlsruher Institut für Technologie (KIT)
ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik

Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -

Am Fasanengarten 5, Gebäude 50.34, Raum 009
76131 Karlsruhe
Telefon: +49 721 608-43973
Fax: +49 721 608-46699
E-Mail: olaf.h...@kit.edu
atis.informatik.kit.edu

www.kit.edu

KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft

Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.




smime.p7s
Description: S/MIME Cryptographic Signature

memberof Overlay not showing in base search

[Umar Draz]

Hi,

I am running OpenLDAP server on Ubuntu 18.

The memberOf attribute is not showing in ldap simple search, if I do the
following then memberOf attribute is hidden.

*ldapsearch -Y external -H ldapi:/// -b dc=example,dc=com*
# udraz, Users, example.com 
dn: uid=udraz,ou=Users,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: udraz
sn: Draz
givenName: Umar
mail: ud...@example.com
cn: Umar Draz
displayName: Umar Draz
uidNumber: 5000
gidNumber: 5000
gecos: Umar Draz
loginShell: /bin/bash
homeDirectory: /home/udraz

But if I do the following then memberOf attribute appear

*ldapsearch -Y external -H ldapi:/// -b dc=example,dc=com memberOf*
# udraz, Users, example.com
dn: uid=udraz,ou=Users,dc=example,dc=com
memberOf: cn=developers,ou=Users,dc=example,dc=com

Would you please help me how to solve this