Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 22.12.2015 16:59, Petr Spacek wrote: On 14.4.2015 19:32, Petr Vobornik wrote: Pushed to master: 11bd9d96f191066f7ba760549f00179c128a9787 Please be so kind and fix naming. AFAIK the patch refers to 'openldap' DN format but in fact it is python-ldap-ishm. It will surely confuse next generation of FreeIPA developers :-) Will be in separate patch. Petr, Santa found out that you did not send the patch! You are risking Christmas without any presents ... :-) Bump for patch. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 14.4.2015 19:32, Petr Vobornik wrote: > > Pushed to master: 11bd9d96f191066f7ba760549f00179c128a9787 > >> >> Please be so kind and fix naming. AFAIK the patch refers to 'openldap' DN >> format but in fact it is python-ldap-ishm. It will surely confuse next >> generation of FreeIPA developers :-) >> > > Will be in separate patch. Petr, Santa found out that you did not send the patch! You are risking Christmas without any presents ... :-) -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 04/14/2015 09:32 AM, Petr Spacek wrote: On 10.4.2015 13:11, Petr Viktorin wrote: ACK Pushed to master: 11bd9d96f191066f7ba760549f00179c128a9787 Please be so kind and fix naming. AFAIK the patch refers to 'openldap' DN format but in fact it is python-ldap-ishm. It will surely confuse next generation of FreeIPA developers :-) Will be in separate patch. -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 04/02/2015 11:54 AM, Petr Viktorin wrote:
On 03/31/2015 12:11 PM, Petr Vobornik wrote:
The only different thing is a lack of utf-8 encoded str support(as
input). I don't know how much important the support is.
I don't think that support is too important (assuming IPA doesn't use
it!). However, the behavior with this patch is dangerous.
It allows unicode and ASCII strings, but fails on non-ASCII strings.
That means things will usually work, but as soon as a non-ASCII
component is introduced at the wrong place, you get an error.
Restoring support for utf-8 encoded str looks easy to do; here's a patch
you can squash in. Or did I miss something?
I also had to fix creation of AVAs to support utf-8 encoded str as input
for attr and value (separately).
maybe it could be attached to ticket
https://fedorahosted.org/freeipa/ticket/4947
-
DN code was optimized to be faster if DNs are created from string. This
is the major use case, since most DNs come from LDAP.
With this patch, DN creation is almost 8-10x faster (with 30K-100K DNs).
Second mojor use case - deepcopy in LDAPEntry is about 20x faster - done
by custom __deepcopy__ function.
The major change is that DN is no longer internally composed of RDNs
and AVAs but it rather keeps the data in open ldap format - the same as
output of str2dn function. Therefore, for immutable DNs, no other
transformations are required on instantiation.
The format is:
DN: [RDN, RDN,...]
RDN: [AVA, AVA,...]
AVA: ['utf-8 encoded str - attr', 'utf-8 encode str -value', FLAG]
FLAG: int
Further indexing of DN object constructs an RDN which is just an
encapsulation of the RDN part of open ldap representation. Indexing of
RDN constructs AVA in the same fashion.
Obtained EditableAVA, EditableRDN from EditableDN shares the respected
lists of the open ldap repr. so that the change of value or attr is
reflected in parent object.
Looks good. A couple of comments:
RDN.to_openldap: _avas always has 3 components, right? I'd prefer
`list(a)` over `[a[0], a[1], a[2]]`. Similarly for tuple in in __add__
and RDN._avas_from_sequence.
Fixed
DN._rdns_from_value: the error message at the end is wrong, RDN is also
accepted. (And, `type(value)` would be more informative than
`value.__class__.__name__`.)
Fixed
You can optimize __deepcopy__ for immutable DNs even further: just
return self!
Fixed, but kept part for EditableDN
In DN.find rfind, RDNs are not accepted but the error message says
they are.
messages fixed
You removed the newline at end of file.
line readded
--
Petr Vobornik
From 6289202ca5c5d24c1b07754d19a292e66cfd5df2 Mon Sep 17 00:00:00 2001
From: Petr Vobornik pvobo...@redhat.com
Date: Wed, 25 Mar 2015 13:39:43 +0100
Subject: [PATCH] performance: faster DN implementation
DN code was optimized to be faster if DNs are created from string. This is
the major use case, since most DNs come from LDAP.
With this patch, DN creation is almost 8-10x faster (with 30K-100K DNs).
Second mojor use case - deepcopy in LDAPEntry is about 20x faster - done by
custom __deepcopy__ function.
The major change is that DN is no longer internally composed of RDNs and
AVAs but it rather keeps the data in open ldap format - the same as output
of str2dn function. Therefore, for immutable DNs, no other transformations
are required on instantiation.
The format is:
DN: [RDN, RDN,...]
RDN: [AVA, AVA,...]
AVA: ['utf-8 encoded str - attr', 'utf-8 encode str -value', FLAG]
FLAG: int
Further indexing of DN object constructs an RDN which is just an encapsulation
of the RDN part of open ldap representation. Indexing of RDN constructs AVA in
the same fashion.
Obtained EditableAVA, EditableRDN from EditableDN shares the respected lists
of the open ldap repr. so that the change of value or attr is reflected in
parent object.
---
ipapython/dn.py| 595 ++---
ipatests/test_ipapython/test_dn.py | 17 +-
2 files changed, 306 insertions(+), 306 deletions(-)
diff --git a/ipapython/dn.py b/ipapython/dn.py
index 834291fbe8696622162efa5193622d74f11f25ca..5b6570770d587937c87380f7ea19e999c3d8867d 100644
--- a/ipapython/dn.py
+++ b/ipapython/dn.py
@@ -497,6 +497,97 @@ def _adjust_indices(start, end, length):
return start, end
+
+def _normalize_ava_input(val):
+if not isinstance(val, basestring):
+val = unicode(val).encode('utf-8')
+elif isinstance(val, unicode):
+val = val.encode('utf-8')
+return val
+
+
+def str2rdn(value):
+try:
+rdns = str2dn(value.encode('utf-8'))
+except DECODING_ERROR:
+raise ValueError(malformed AVA string = \%s\ % value)
+if len(rdns) != 1:
+raise ValueError(multiple RDN's specified by \%s\ % (value))
+return rdns[0]
+
+
+def get_ava(*args, **kwds):
+
+Get AVA from args in open ldap format(raw). Optimized for construction
+from openldap format.
+
+Allowed formats of argument list:
+1) three args - open ldap
Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 03/31/2015 12:11 PM, Petr Vobornik wrote:
The only different thing is a lack of utf-8 encoded str support(as
input). I don't know how much important the support is.
I don't think that support is too important (assuming IPA doesn't use
it!). However, the behavior with this patch is dangerous.
It allows unicode and ASCII strings, but fails on non-ASCII strings.
That means things will usually work, but as soon as a non-ASCII
component is introduced at the wrong place, you get an error.
Restoring support for utf-8 encoded str looks easy to do; here's a patch
you can squash in. Or did I miss something?
maybe it could be attached to ticket
https://fedorahosted.org/freeipa/ticket/4947
-
DN code was optimized to be faster if DNs are created from string. This
is the major use case, since most DNs come from LDAP.
With this patch, DN creation is almost 8-10x faster (with 30K-100K DNs).
Second mojor use case - deepcopy in LDAPEntry is about 20x faster - done
by custom __deepcopy__ function.
The major change is that DN is no longer internally composed of RDNs
and AVAs but it rather keeps the data in open ldap format - the same as
output of str2dn function. Therefore, for immutable DNs, no other
transformations are required on instantiation.
The format is:
DN: [RDN, RDN,...]
RDN: [AVA, AVA,...]
AVA: ['utf-8 encoded str - attr', 'utf-8 encode str -value', FLAG]
FLAG: int
Further indexing of DN object constructs an RDN which is just an
encapsulation of the RDN part of open ldap representation. Indexing of
RDN constructs AVA in the same fashion.
Obtained EditableAVA, EditableRDN from EditableDN shares the respected
lists of the open ldap repr. so that the change of value or attr is
reflected in parent object.
Looks good. A couple of comments:
RDN.to_openldap: _avas always has 3 components, right? I'd prefer
`list(a)` over `[a[0], a[1], a[2]]`. Similarly for tuple in in __add__
and RDN._avas_from_sequence.
DN._rdns_from_value: the error message at the end is wrong, RDN is also
accepted. (And, `type(value)` would be more informative than
`value.__class__.__name__`.)
You can optimize __deepcopy__ for immutable DNs even further: just
return self!
In DN.find rfind, RDNs are not accepted but the error message says
they are.
You removed the newline at end of file.
--
Petr Viktorin
From 1461bc75748d5bdcf301242c3e31c044542f5bdd Mon Sep 17 00:00:00 2001
From: Petr Viktorin pvikt...@redhat.com
Date: Thu, 2 Apr 2015 10:40:15 +0200
Subject: [PATCH] Restore support for creating DNs from utf-8 encoded strings
---
ipapython/dn.py| 4 ++-
ipatests/test_ipapython/test_dn.py | 60 +++---
2 files changed, 33 insertions(+), 31 deletions(-)
diff --git a/ipapython/dn.py b/ipapython/dn.py
index 5d1f15c..dc536ce 100644
--- a/ipapython/dn.py
+++ b/ipapython/dn.py
@@ -1225,7 +1225,9 @@ def _copy_rdns(self, rdns=None):
def _rdns_from_value(self, value):
if isinstance(value, basestring):
try:
-rdns = str2dn(value.encode('utf-8'))
+if isinstance(value, unicode):
+value = value.encode('utf-8')
+rdns = str2dn(value)
if self.is_mutable:
self._copy_rdns(rdns) # AVAs to be list instead of tuple
except DECODING_ERROR:
diff --git a/ipatests/test_ipapython/test_dn.py b/ipatests/test_ipapython/test_dn.py
index 2a7739b..394be5b 100644
--- a/ipatests/test_ipapython/test_dn.py
+++ b/ipatests/test_ipapython/test_dn.py
@@ -1864,11 +1864,11 @@ def test_i18n(self):
self.assertEqual(ava1.attr, self.arabic_hello_unicode)
self.assertEqual(str(ava1), self.arabic_hello_utf8+'=foo')
-# ava1 = AVA_class(self.arabic_hello_utf8, 'foo')
-# self.assertIsInstance(ava1.attr, unicode)
-# self.assertIsInstance(ava1.value, unicode)
-# self.assertEqual(ava1.attr, self.arabic_hello_unicode)
-# self.assertEqual(str(ava1), self.arabic_hello_utf8+'=foo')
+ava1 = AVA_class(self.arabic_hello_utf8, 'foo')
+self.assertIsInstance(ava1.attr, unicode)
+self.assertIsInstance(ava1.value, unicode)
+self.assertEqual(ava1.attr, self.arabic_hello_unicode)
+self.assertEqual(str(ava1), self.arabic_hello_utf8+'=foo')
# test value i18n
ava1 = AVA_class('cn', self.arabic_hello_unicode)
@@ -1877,11 +1877,11 @@ def test_i18n(self):
self.assertEqual(ava1.value, self.arabic_hello_unicode)
self.assertEqual(str(ava1), 'cn='+self.arabic_hello_utf8)
-# ava1 = AVA_class('cn', self.arabic_hello_utf8)
-# self.assertIsInstance(ava1.attr, unicode)
-# self.assertIsInstance(ava1.value, unicode)
-# self.assertEqual(ava1.value, self.arabic_hello_unicode)
-# self.assertEqual(str(ava1),
Re: [Freeipa-devel] [PATCH] 811 performance: faster DN implementation
On 31.3.2015 12:11, Petr Vobornik wrote: The major change is that DN is no longer internally composed of RDNs and AVAs but it rather keeps the data in open ldap format - the same as output of str2dn function. Therefore, for immutable DNs, no other transformations are required on instantiation. Note: I guess that this is an python-ldap format rather than OpenLDAP format. It would be handy to fix commands for further generations to save them some banging with their heads against a wall of confusion. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
[Freeipa-devel] [PATCH] 811 performance: faster DN implementation
The only different thing is a lack of utf-8 encoded str support(as
input). I don't know how much important the support is.
maybe it could be attached to ticket
https://fedorahosted.org/freeipa/ticket/4947
-
DN code was optimized to be faster if DNs are created from string. This
is the major use case, since most DNs come from LDAP.
With this patch, DN creation is almost 8-10x faster (with 30K-100K DNs).
Second mojor use case - deepcopy in LDAPEntry is about 20x faster - done
by custom __deepcopy__ function.
The major change is that DN is no longer internally composed of RDNs
and AVAs but it rather keeps the data in open ldap format - the same as
output of str2dn function. Therefore, for immutable DNs, no other
transformations are required on instantiation.
The format is:
DN: [RDN, RDN,...]
RDN: [AVA, AVA,...]
AVA: ['utf-8 encoded str - attr', 'utf-8 encode str -value', FLAG]
FLAG: int
Further indexing of DN object constructs an RDN which is just an
encapsulation of the RDN part of open ldap representation. Indexing of
RDN constructs AVA in the same fashion.
Obtained EditableAVA, EditableRDN from EditableDN shares the respected
lists of the open ldap repr. so that the change of value or attr is
reflected in parent object.
--
Petr Vobornik
From 1de87815183b9e9ddef5710eab1e83ba36127986 Mon Sep 17 00:00:00 2001
From: Petr Vobornik pvobo...@redhat.com
Date: Wed, 25 Mar 2015 13:39:43 +0100
Subject: [PATCH] performance: faster DN implementation
DN code was optimized to be faster if DNs are created from string. This is
the major use case, since most DNs come from LDAP.
With this patch, DN creation is almost 8-10x faster (with 30K-100K DNs).
Second mojor use case - deepcopy in LDAPEntry is about 20x faster - done by
custom __deepcopy__ function.
The major change is that DN is no longer internally composed of RDNs and
AVAs but it rather keeps the data in open ldap format - the same as output
of str2dn function. Therefore, for immutable DNs, no other transformations
are required on instantiation.
The format is:
DN: [RDN, RDN,...]
RDN: [AVA, AVA,...]
AVA: ['utf-8 encoded str - attr', 'utf-8 encode str -value', FLAG]
FLAG: int
Further indexing of DN object constructs an RDN which is just an encapsulation
of the RDN part of open ldap representation. Indexing of RDN constructs AVA in
the same fashion.
Obtained EditableAVA, EditableRDN from EditableDN shares the respected lists
of the open ldap repr. so that the change of value or attr is reflected in
parent object.
---
ipapython/dn.py| 558 ++---
ipatests/test_ipapython/test_dn.py | 74 ++---
2 files changed, 306 insertions(+), 326 deletions(-)
diff --git a/ipapython/dn.py b/ipapython/dn.py
index 834291fbe8696622162efa5193622d74f11f25ca..5d1f15c27abcea11bc3acafa5e4297692b236e98 100644
--- a/ipapython/dn.py
+++ b/ipapython/dn.py
@@ -497,6 +497,89 @@ def _adjust_indices(start, end, length):
return start, end
+
+def str2rdn(value):
+try:
+rdns = str2dn(value.encode('utf-8'))
+except DECODING_ERROR:
+raise ValueError(malformed AVA string = \%s\ % value)
+if len(rdns) != 1:
+raise ValueError(multiple RDN's specified by \%s\ % (value))
+return rdns[0]
+
+
+def get_ava(*args, **kwds):
+
+Get AVA from args in open ldap format(raw). Optimized for construction
+from openldap format.
+
+Allowed formats of argument list:
+1) three args - open ldap format (attr and value have to be utf-8 encoded):
+a) ['attr', 'value', 0]
+2) two args:
+a) ['attr', 'value']
+3) one arg:
+a) [('attr', 'value')]
+b) [['attr', 'value']]
+c) [AVA(..)]
+d) ['attr=value']
+
+ava = None
+l = len(args)
+if l == 3: # raw values - constructed FROM RDN
+if kwds.get('mutable', False):
+ava = args
+else:
+ava = (args[0], args[1], args[2])
+elif l == 2: # user defined values
+ava = [unicode(args[0]).encode('utf-8'), unicode(args[1]).encode('utf-8'), 0]
+elif l == 1: # slow mode, tuple, string,
+arg = args[0]
+if isinstance(arg, AVA):
+ava = arg.to_openldap()
+elif isinstance(arg, (tuple, list)):
+if len(arg) != 2:
+raise ValueError(tuple or list must be 2-valued, not \%s\ % (arg))
+ava = [unicode(arg[0]).encode('utf-8'), unicode(arg[1]).encode('utf-8'), 0]
+elif isinstance(arg, basestring):
+rdn = str2rdn(arg)
+if len(rdn) 1:
+raise TypeError(multiple AVA's specified by \%s\ % (arg))
+ava = list(rdn[0])
+else:
+raise TypeError(with 1 argument, argument must be str, unicode, tuple or list, got %s instead %
+arg.__class__.__name__)
+else:
+raise TypeError(invalid number of arguments. 1-3 allowed)
+return ava
+
+
+def
