On 05/05/2016 03:44 PM, Petr Vobornik wrote:
On 05/04/2016 02:20 PM, thierry bordaz wrote:
Hello,
I have been doing some tests/measures using
https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
The tool creates a set of typical users/hosts/groups... to import with a
ldapadd.
I wrote down some finding in
http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
I still have to do some cleanup around the performance but the basic of a
possible improvement is to do provisioning in several steps (disabling
plugins, provisioning, enabling plugin, running fixup tasks).
Before going further in the design I wanted to share those ideas and know
if
it raise any concern.
thanks
thierry
Hi Thierry,
Thanks for the analysis. Very nice.
Knowing this will help us suggesting workarounds also for old releases.
Couple questions:
Have you tested retrCL disabled with memberOf enabled. It seems that it
would eliminate 550K adds and 0.8M searches. What would be the time
improvement?
Basically provisioning/migrate does not scale well because of memberOf.
Each time you add a member to a group you need to reevalute its
membership (15M search) and finally update the member (that is a MOD for
the member + ADD in retroCL). If a same entry is added to a group and
later added to an other group, the first evaluation/updates are useless.
Disabling memberof drops SRCH from 15M -> ~1M. You can get an additional
gain ~1M -> ~0.1M if you disable retroCL.
I tested retroCL disabled / memberOf enabled but on a different dataset.
It gaves low improvement in term of duration.
I think MODs are here more expensive that ADD.
I will test your suggestion to confirm the numbers.
Do you know what is the time when memberof is enabled but slapi-nis and
retroCL are disabled?
No. I will test it.
From the text it was not clear to me, if you find or investigate
possible improvements in memberof plugin which would improve the
performance without stopping and starting DS.
memberof does not scale well because of #SRCH and #MOD.
Ludwig suggested a group caching that is looking close to other
in-memory implementations of memberof.
Such cache would reduce the #SRCH possibly from 15M->1M, but the
implementation is not immediate.
Also note that filling the cache has a cost that can delay startup (like
we saw in slapi-nis) and a memory footprint.
Regarding the MODs we can make memberof attribute a virtual attribute
and stop updating the entries. This is also not immediate to implement:
rewrite memberof plugin.
From 7.3 schedule, I think we can get an important improvement if DS
start-stop is acceptable.
Improving memberof is more a longer term option.
thanks
thierry
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
