Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[Jan Cholasta]

On 18.8.2016 11:06, David Kupka wrote:

On 17/08/16 14:17, Jan Cholasta wrote:

On 17.8.2016 13:21, David Kupka wrote:

On 08/08/16 13:26, Jan Cholasta wrote:

On 4.8.2016 16:32, David Kupka wrote:

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and
commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema
from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client.
That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for
lists of
topics and command and displaying individual topics. This is
simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are
already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can
generate
basic bits (list of all topics, list of all commands, list of
commands
for each topic) from schema and store it in cache. Then we need
to go
through all client plugins and get similar bits for client
plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it
even
can't be. Also first time you display particular topic or list
takes
longer because it must be freshly generated and stored in cache
for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins
and
remote plugins.
The main idea of this approach is to avoid creating instances of
the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to
access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for
non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__()
rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise
.summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix
them.


Shame on me.

Anyway,

1) When schema() raises SchemaUpToDate, the current code explodes:

ipa: ERROR: KeyError: 'fingerprint'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in
run
api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707,
in finalize
self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422,
in __do_if_not_done
getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585,
in load_plugins
for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919,
in packages
ipaclient.remote_plugins.get_package(self),
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py",

line 92, in get_package
plugins = schema.get_package(api, server_info, client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 558, in get_package
fingerprint = str(schema['fingerprint'])
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 483, in __getitem__
return self._dict[key]
KeyError: 'fingerprint'


2) We read server info every time get_package() is called. It should be
cache similarly to how the schema is cached in the api object.


3) Some of the commands are still fully initialized during "ipa help
commands".


Honza


Updated patches attached.


Thanks, ACK.

Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e



When locale is not available setlocale() explodes. Handling this
exception in the same way as in ipalib/rpc.py to behave consistently.


Works for me, ACK.

Pushed to master: b6d5ed139b261b5db078ab652d22ea1d3b8092d3

--
Jan Cholasta

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[David Kupka]

On 17/08/16 14:17, Jan Cholasta wrote:

On 17.8.2016 13:21, David Kupka wrote:

On 08/08/16 13:26, Jan Cholasta wrote:

On 4.8.2016 16:32, David Kupka wrote:

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and
commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema
from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client.
That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for
lists of
topics and command and displaying individual topics. This is
simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are
already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can
generate
basic bits (list of all topics, list of all commands, list of
commands
for each topic) from schema and store it in cache. Then we need
to go
through all client plugins and get similar bits for client
plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it
even
can't be. Also first time you display particular topic or list
takes
longer because it must be freshly generated and stored in cache
for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of
the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to
access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for
non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__()
rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise
.summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix
them.


Shame on me.

Anyway,

1) When schema() raises SchemaUpToDate, the current code explodes:

ipa: ERROR: KeyError: 'fingerprint'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in
run
api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707,
in finalize
self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422,
in __do_if_not_done
getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585,
in load_plugins
for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919,
in packages
ipaclient.remote_plugins.get_package(self),
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py",
line 92, in get_package
plugins = schema.get_package(api, server_info, client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 558, in get_package
fingerprint = str(schema['fingerprint'])
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 483, in __getitem__
return self._dict[key]
KeyError: 'fingerprint'


2) We read server info every time get_package() is called. It should be
cache similarly to how the schema is cached in the api object.


3) Some of the commands are still fully initialized during "ipa help
commands".


Honza


Updated patches attached.


Thanks, ACK.

Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e



When locale is not available setlocale() explodes. Handling this 
exception in the same way as in ipalib/rpc.py to behave consistently.


--
David Kupka
From 083ad6016999eb244dc9915ec5fe6988f9053b0d Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Thu, 18 Aug 2016 10:59:09 +0200
Subject: [PATCH] schema cache: Fallback to 'en_us' when locale is not
 available

https://fedorahosted.org/freeipa/ticket/6204
---
 ipaclient/remote_plugins/schema.py | 12 +---
 1 file changed, 9 insertions(+), 3 

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[Jan Cholasta]

On 17.8.2016 13:21, David Kupka wrote:

On 08/08/16 13:26, Jan Cholasta wrote:

On 4.8.2016 16:32, David Kupka wrote:

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client.
That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for
lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are
already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can
generate
basic bits (list of all topics, list of all commands, list of
commands
for each topic) from schema and store it in cache. Then we need
to go
through all client plugins and get similar bits for client plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it
even
can't be. Also first time you display particular topic or list
takes
longer because it must be freshly generated and stored in cache for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to
access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for
non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise
.summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix
them.


Shame on me.

Anyway,

1) When schema() raises SchemaUpToDate, the current code explodes:

ipa: ERROR: KeyError: 'fingerprint'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in
run
api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707,
in finalize
self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422,
in __do_if_not_done
getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585,
in load_plugins
for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919,
in packages
ipaclient.remote_plugins.get_package(self),
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py",
line 92, in get_package
plugins = schema.get_package(api, server_info, client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 558, in get_package
fingerprint = str(schema['fingerprint'])
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 483, in __getitem__
return self._dict[key]
KeyError: 'fingerprint'


2) We read server info every time get_package() is called. It should be
cache similarly to how the schema is cached in the api object.


3) Some of the commands are still fully initialized during "ipa help
commands".


Honza


Updated patches attached.


Thanks, ACK.

Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e

--
Jan Cholasta

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[David Kupka]

On 08/08/16 13:26, Jan Cholasta wrote:

On 4.8.2016 16:32, David Kupka wrote:

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client.
That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can
generate
basic bits (list of all topics, list of all commands, list of
commands
for each topic) from schema and store it in cache. Then we need
to go
through all client plugins and get similar bits for client plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it
even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to
access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise
.summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix them.


Shame on me.

Anyway,

1) When schema() raises SchemaUpToDate, the current code explodes:

ipa: ERROR: KeyError: 'fingerprint'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run
api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707,
in finalize
self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422,
in __do_if_not_done
getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585,
in load_plugins
for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919,
in packages
ipaclient.remote_plugins.get_package(self),
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py",
line 92, in get_package
plugins = schema.get_package(api, server_info, client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 558, in get_package
fingerprint = str(schema['fingerprint'])
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py",
line 483, in __getitem__
return self._dict[key]
KeyError: 'fingerprint'


2) We read server info every time get_package() is called. It should be
cache similarly to how the schema is cached in the api object.


3) Some of the commands are still fully initialized during "ipa help
commands".


Honza


Updated patches attached.

--
David Kupka
From 48c908746459f31b44a09045d53653ab503698ad Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Thu, 4 Aug 2016 16:02:24 +0200
Subject: [PATCH 01/10] schema cache: Do not reset ServerInfo dirty flag

Once dirty flag is set to True it must not be set back to False.
Otherwise changes are not written back to file.

https://fedorahosted.org/freeipa/ticket/6048
---
 ipaclient/remote_plugins/__init__.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ipaclient/remote_plugins/__init__.py b/ipaclient/remote_plugins/__init__.py
index 444651d30fd0cd96299fecb7ee7b5e4532b0abd4..976d6968724088d8e1fe8d3615990accf585ffeb 100644
--- 

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[Jan Cholasta]

On 4.8.2016 16:32, David Kupka wrote:

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client.
That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can generate
basic bits (list of all topics, list of all commands, list of
commands
for each topic) from schema and store it in cache. Then we need to go
through all client plugins and get similar bits for client plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to
access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise
.summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix them.


Shame on me.

Anyway,

1) When schema() raises SchemaUpToDate, the current code explodes:

ipa: ERROR: KeyError: 'fingerprint'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run
api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, 
in finalize

self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, 
in __do_if_not_done

getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, 
in load_plugins

for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, 
in packages

ipaclient.remote_plugins.get_package(self),
  File 
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", 
line 92, in get_package

plugins = schema.get_package(api, server_info, client)
  File 
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", 
line 558, in get_package

fingerprint = str(schema['fingerprint'])
  File 
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", 
line 483, in __getitem__

return self._dict[key]
KeyError: 'fingerprint'


2) We read server info every time get_package() is called. It should be 
cache similarly to how the schema is cached in the api object.



3) Some of the commands are still fully initialized during "ipa help 
commands".



Honza

--
Jan Cholasta

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[David Kupka]

On 03/08/16 16:33, Jan Cholasta wrote:

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client. That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in
the
schema cache so we can skip this part and generate help from it,
right?
Not so fast!

There are client plugins with commands and topics. So we can generate
basic bits (list of all topics, list of all commands, list of commands
for each topic) from schema and store it in cache. Then we need to go
through all client plugins and get similar bits for client plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise .summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96



I've made and reviewer overlooked some errors. Attached patches fix them.

--
David Kupka
From ae1a9d024e37a153b6e9e4ada657f0e1e78300ef Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Thu, 4 Aug 2016 16:02:24 +0200
Subject: [PATCH 1/3] schema cache: Do not reset ServerInfo dirty flag

Once dirty flag is set to True it must not be set back to False.
Otherwise changes are not written back to file.

https://fedorahosted.org/freeipa/ticket/6048
---
 ipaclient/remote_plugins/__init__.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ipaclient/remote_plugins/__init__.py b/ipaclient/remote_plugins/__init__.py
index 444651d30fd0cd96299fecb7ee7b5e4532b0abd4..976d6968724088d8e1fe8d3615990accf585ffeb 100644
--- a/ipaclient/remote_plugins/__init__.py
+++ b/ipaclient/remote_plugins/__init__.py
@@ -59,7 +59,8 @@ class ServerInfo(collections.MutableMapping):
 return self._dict[key]
 
 def __setitem__(self, key, value):
-self._dirty = key not in self._dict or self._dict[key] != value
+if key not in self._dict or self._dict[key] != value:
+self._dirty = True
 self._dict[key] = value
 
 def __delitem__(self, key):
-- 
2.7.4

From a6e9b17e0ad109f3237f4417828f7a3cfb5aa743 Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Thu, 4 Aug 2016 16:07:08 +0200
Subject: [PATCH 2/3] schema cache: Write format information in cache

When format is not in cache '0' is assumed and client expecting higher
format refuses to load such cache.

https://fedorahosted.org/freeipa/ticket/6048
---
 ipaclient/remote_plugins/schema.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py
index a215452ea0d2c1278a6121b3806b0daee02abd6e..13b1c365b111faae673c5ed78a7e5439a869c330 100644
--- a/ipaclient/remote_plugins/schema.py
+++ b/ipaclient/remote_plugins/schema.py
@@ -524,6 +524,7 @@ class Schema(object):
 
 schema.writestr('_help',
 json.dumps(self._generate_help(self._dict)))
+schema.writestr('format', json.dumps(FORMAT))
 
 def _read(self, path):
 with self._open_schema(self._fingerprint, 'r') as zf:
-- 
2.7.4

From c57a1b98ca518db8909d4d1176c99c4b2daec606 Mon Sep 17 

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[Jan Cholasta]

On 3.8.2016 16:23, David Kupka wrote:

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only
logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client. That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in
the
schema cache so we can skip this part and generate help from it, right?
Not so fast!

There are client plugins with commands and topics. So we can generate
basic bits (list of all topics, list of all commands, list of commands
for each topic) from schema and store it in cache. Then we need to go
through all client plugins and get similar bits for client plugins.
Then
we can merge and print.

Still the client response is not as fast as before and I this it even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for
next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise .summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.


Thanks, ACK.

Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96

--
Jan Cholasta

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[David Kupka]

On 21/07/16 10:12, Jan Cholasta wrote:

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client. That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in the
schema cache so we can skip this part and generate help from it, right?
Not so fast!

There are client plugins with commands and topics. So we can generate
basic bits (list of all topics, list of all commands, list of commands
for each topic) from schema and store it in cache. Then we need to go
through all client plugins and get similar bits for client plugins. Then
we can merge and print.

Still the client response is not as fast as before and I this it even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member,
because Schema is where all the state is.

(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent
keys? It looks like it doesn't.)


Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather
that getattr()-ing it in _ensure_cached()?


Patch 0116:

ClientCommand.doc should be a class property as well, otherwise .summary
won't work on it correctly.

_SchemaCommand.doc should not be a property, as it's not needed for
.summary to work on it correctly.


Otherwise works fine for me.

Honza



Updated patches attached.

--
David Kupka
From bae93c2a57b51c8b5c11d481f9df2a62f330fb81 Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Wed, 27 Jul 2016 10:46:40 +0200
Subject: [PATCH 1/6] schema: Speed up schema cache

Check presence of schema in cache (and download it if necessary) on
__init__ instead of with each __getitem__ call. Prefill internal
dictionary with empty record for each command to be able to quickly
determine if requested command exist in schema or not. Rest of schema
data are read from cache on first attempt to retrive them.

https://fedorahosted.org/freeipa/ticket/6048
https://fedorahosted.org/freeipa/ticket/6069
---
 ipaclient/remote_plugins/schema.py | 297 ++---
 1 file changed, 175 insertions(+), 122 deletions(-)

diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py
index cd1d5d607978899254325f634ccec91d2c92f59b..70cd7536d836ec113236bfdae8bbabb2d843725d 100644
--- a/ipaclient/remote_plugins/schema.py
+++ b/ipaclient/remote_plugins/schema.py
@@ -5,10 +5,8 @@
 import collections
 import errno
 import fcntl
-import glob
 import json
 import os
-import re
 import sys
 import time
 import types
@@ -65,8 +63,6 @@ USER_CACHE_PATH = (
 '.cache'
 )
 )
-SCHEMA_DIR = os.path.join(USER_CACHE_PATH, 'ipa', 'schema')
-SERVERS_DIR = os.path.join(USER_CACHE_PATH, 'ipa', 'servers')
 
 logger = log_mgr.get_logger(__name__)
 
@@ -274,15 +270,6 @@ class _SchemaObjectPlugin(_SchemaPlugin):
 schema_key = 'classes'
 
 
-def _ensure_dir_created(d):
-try:
-os.makedirs(d)
-except OSError as e:
-if e.errno != errno.EEXIST:
-raise RuntimeError("Unable to create cache directory: {}"
-   "".format(e))
-
-
 class _LockedZipFile(zipfile.ZipFile):
 """ Add locking to zipfile.ZipFile
 Shared lock is used with read mode, exclusive with write mode.
@@ -308,7 +295,10 @@ class _SchemaNameSpace(collections.Mapping):
 self._schema = schema
 
 def __getitem__(self, key):
-return self._schema.read_namespace_member(self.name, key)
+try:
+return self._schema.read_namespace_member(self.name, key)
+except KeyError:
+raise KeyError(key)
 
 def __iter__(self):
 for key in self._schema.iter_namespace(self.name):
@@ -322,6 +312,62 @@ class NotAvailable(Exception):
 pass
 
 
+class ServerInfo(collections.MutableMapping):
+_DIR = 

Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help

[Jan Cholasta]

Hi,

On 20.7.2016 14:32, David Kupka wrote:

On 15/07/16 12:53, David Kupka wrote:

Hello!

After Honza introduced thin client that builds plugins and commands
dynamically from schema client became much slower. This is only logical,
instead of importing a module client now must fetch the schema from
server, parse it and instantiate the commands using the data.

First step to speed it up was addition of schema cache to client. That
removed the RTT and download time of fetching schema every time.

Now the most time consuming task became displaying help for lists of
topics and command and displaying individual topics. This is simply
because of the need to instantiate all the commands to find the
relations between topics and commands.

All the necessary bits for server commands and topics are already in the
schema cache so we can skip this part and generate help from it, right?
Not so fast!

There are client plugins with commands and topics. So we can generate
basic bits (list of all topics, list of all commands, list of commands
for each topic) from schema and store it in cache. Then we need to go
through all client plugins and get similar bits for client plugins. Then
we can merge and print.

Still the client response is not as fast as before and I this it even
can't be. Also first time you display particular topic or list takes
longer because it must be freshly generated and stored in cache for next
use. And this is what the attached patches do.

https://fedorahosted.org/freeipa/ticket/6048


Reimplemented so there is no need to distinguish client plugins and
remote plugins.
The main idea of this approach is to avoid creating instances of the
commands just to get the information about topic, name and summary
needed for displaying help. Instead class properties are used to access
the information directly in schema.


Patch 0112:

I think this would better be done in Schema.read_namespace_member, 
because Schema is where all the state is.


(BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent 
keys? It looks like it doesn't.)



Patch 0113:

How about setting _schema_cached to False in Schema.__init__() rather 
that getattr()-ing it in _ensure_cached()?



Patch 0116:

ClientCommand.doc should be a class property as well, otherwise .summary 
won't work on it correctly.


_SchemaCommand.doc should not be a property, as it's not needed for 
.summary to work on it correctly.



Otherwise works fine for me.

Honza

--
Jan Cholasta

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code