On 09/12/2013 07:39 AM, thierry bordaz wrote:
On 09/10/2013 04:35 PM, Ludwig Krispenz wrote:
On 09/10/2013 04:29 PM, Rich Megginson wrote:
On 09/10/2013 01:47 AM, Ludwig Krispenz wrote:
On 09/09/2013 07:19 PM, Rich Megginson wrote:
On 09/09/2013 02:27 AM, Ludwig Krispenz wrote:
On 09/07/2013 05:02 AM, David Boreham wrote:
On 9/6/2013 8:49 PM, Nathan Kinder wrote:
This is a good idea, and it is something that we discussed
briefly off-list. The only downside is that we need to change
the index format to keep a count of ids for each key.
Implementing this isn't a big problem, but it does mean that
the existing indexes need to be updated to populate the count
based off of the contents (as you mention above).
I don't think you need to do this (I certainly wasn't advocating
doing so). The "statistics" state is much the same as that
proposed in Rich's design. In fact you could probably just use
that same information. My idea is more about where and how you
use the information. All you need is something associated with
each index that says "not much point looking here if you're
after something specific, move along, look somewhere else
instead". This is much the same information as "don't use a high
scan limit here".
In the short term, we are looking for a way to be able to
improve performance for specific search filters that are not
possible to modify on the client side (for whatever reason)
while leaving the index file format exactly as it is. I still
feel that there is potentially great value in keeping a count
of ids per key so we can optimize things on the server side
automatically without the need for complex index configuration
on the administrator's part. I think we should consider this
for an additional future enhancement.
I'm saying the same thing. Keeping a cardinality count per key
is way more than I'm proposing, and I'm not sure how useful that
would be anyway, unless you want to do OLAP in the DS ;)
we have the cardinality of the key in old-idl and this makes some
searches where parts of the filter are allids fast.
I'm late in the discussion, but I think Rich's proposal is very
promising to address all the problems related to allids in new-idl.
We could then eventually rework filter ordering based on these
configurations. Right now we only have a filter ordering based on
index type and try to postpone "<=" or similar filter as they are
known to be costly, but this could be more elaborate.
An alternative would be to have some kind of index lookup
caching. In the example in ticket 47474 the filter is
(&(|(objectClass=organizationalPerson)(objectClass=inetOrgPerson)(objectClass=organization)(objectClass=organizationalUnit)(objectClass=groupOf
Names)(objectClass=groupOfUniqueNames)(objectClass=group))(c3sUserID=EndUser0000078458))"
and probably only the "c3sUserID=xxxxx" part will change, if we
cache the result for the (&(|(objectClass=... part, even if it is
expensive, it would be done only once.
Thanks everyone for the comments. I have added Noriko's suggestion:
http://port389.org/wiki/Design/Fine_Grained_ID_List_Size
David, Ludwig: Does the current design address your concerns,
and/or provide the necessary first step for further refinements?
yes, the topic of filter reordering or caching could be looked at
independently.
Just one concern abou the syntax:
nsIndexIDListScanLimit:
maxsize[:indextype][:flag[,flag...]][:value[,value...]]
since everything is optional, how do you decide if in
nsIndexIDListScanLimit: 6:eq:AND "AND" is a value or a flag ?
and as it defines limits for specific keys, could the attributname
reflect this, eg nsIndexKeyIDListScanLimit or nsIndexKeyScanLimit
or ... ?
Thanks, yes, it is ambiguous.
I think it may have to use keyword=value, so something like this:
nsIndexIDListScanLimit: limit=NNN [type=eq[,sub]] [flags=ADD[,OR]]
[values=val[,val...]]
That should be easy to parse for both humans and machines.
For values, will have to figure out a way to have escapes (e.g. if a
value contains a comma or an escape character). Was thinking of
using LDAP escapes (e.g. \, or \032)
they should be treated as in filters and normalized, in the config it
should be the string representation according to the attributetype
Hi,
I was wondering if this configuration attribute at the index
level, could not also be implemented at the bind-base level.
It could be - it would be more difficult to do - you would have to have
the nsIndexIDListScanLimit attribute specified in the user entry, and it
would have to specify the attribute type e.g.
dn: uid=admin,....
nsIndexIDListScanLimit: limit=xxxx attr=objectclass type=eq
value=inetOrgPerson
Or perhaps a new attribute - nsIndexIDListScanLimit should be not
operational for use in nsIndex, but should be operational for use in a
user entry.
If an application use to bind with a given entry, it could use its
own limitations put for example into operational attribute in the
bound entry itself.
Yes, and we already do this for other limits.
So that two applications, using the same filter component could
have their specific idlist size.
Anyway if it makes sense it could be added later.
Yes, thanks.
best regards
thierry
--
389-devel mailing list
389-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-devel
--
389-devel mailing list
389-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-devel
--
389-devel mailing list
389-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-devel
--
389-devel mailing list
389-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-devel