On 05/31/2016 02:02 PM, Petr Vobornik wrote:
On 05/04/2016 02:20 PM, thierry bordaz wrote:
Hello,

     I have been doing some tests/measures using
     https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.
     The tool creates a set of typical users/hosts/groups... to import with a
     ldapadd.

     I wrote down some finding in
     
http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.
     I still have to do some cleanup around the performance but the basic of a
     possible improvement is to do provisioning in several steps (disabling
     plugins, provisioning, enabling plugin, running fixup tasks).

     Before going further in the design I wanted to share those ideas and know 
if
     it raise any concern.

     thanks
     thierry

Let me conclude the previous sub-thread and also continue with
discussion we had over phone.

The subthread ended with proposal to go with offline provisioning by
disabling LDAP ports similar to ipa-server-upgrade tool and then using
LDAPI to add records by using ipalib in server mode e.g., as in
ipa_otptoken_import.py

So these are the tasks to solve/investigate:

1. Provide guidance/write script which would disable memberof plugin and
other plugins. Disable ldap and ldaps port

I started describing the operations http://www.freeipa.org/page/V4/Performance_Improvements#Algorithm. It needs to be updated with disabling regular ports and accessing through ldapi

2. Provide guidance/script how to use ipalib in server mode and how to
import date. This could be even script which would load file in some
format(e.g. JSON) and executed commands from the file. Basically what
was Alexander proposing and I was against it. After some thought, I
agree that the tool could be easy to write but I'd rather avoid adding
it to 4.4 release maybe even to future releases. Because:
  - It's almost the same as `[RFE] run multiple CLI commands in a batch`
<https://fedorahosted.org/freeipa/ticket/5821> With the distinction that
it connects directly LDAPI and not public API. First I'd like to see
#5821 and then we can think of using same logic(input) to work in
"migration" mode.
- I don't want to include a quickly baked tool so late in 4.4
development. It will have design flaws which will be harder to fix
later. I don't want to loose time with discussion about design of the
tool in this phase of 4.4.
Investigating a ldap bulk load, there is a quite limited set of operations to prepare the instance, run a ldap bulk load and run fixup. However, starting yesterday looking at ipa-otptoken-import I am still unable to connect through ldapi and do those really simple operations... so even it could be easy to write tool my progress start slowly

Regarding the batch commands, it looks quite different than bulk import because batch commands (like user-add) run several ldap ops (SRCH/ADD/MOD) and also batch commands does expect that DS plugins like memberof are enabled.


3. Provide guidance/script to revert #1 and run memberof fixup task.

4. Investigate how replication of so many records with member attributes
will affect other replicas. I.e. if it would not brick entire topology.

*Right now #4 seems to be the most important*.

This was shortly described in http://www.freeipa.org/page/V4/Performance_Improvements#Memberof_plugin and http://www.freeipa.org/page/V4/Performance_Improvements#Replication_being_late.

The topology will slowly converge and eventually all provisioned entries will be available everywhere. But slowly can mean hours or even days before it converges. During that period, a provisioned rules can grant some rights on one replica (where it was imported) but will not grant it on an other where the rule is not yet replicated.

If the topology is a production topology, it can be better to rely on total init of the replicas than on replication.




Then #1, #2, #3 could be delivered as sample script included in
documentation and/or blog post/github. It would allow us to change it
later. I would not focus on #2 before other core 4.4 items are finished.
A lot of guidance is already written in tree design page but it will
need to be transform to easily consumable form for admins. The scripts
can be prepared even when 4.4 is out.

In other words I would not create any official new ipa utility yet.

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to