[389-users] Re: Master-slave replication procedure

[Thomas E Lackey]

Terraform is excellent for provisioning the machine and network infrastructure, 
but it is not a very good tool for managing LDAP replication.  Adding a new DS 
to the replication cluster involves not just changes to that new instance 
(creating the replication account, replica, possibly changelog, etc.) but also 
to all of its replication partners.  And of course, Terraform does not really 
fix the configuration management issue, since not only the Terraform 
configuration needs to be managed, but also its state.  (Replform does not need 
to store state, since it inspects the servers over LDAP to determine its plan.)

 

If you run it globally (‘--global’ ) you only need to maintain one 
configuration file at all.

 

If you run it on each host (our preferred way) there are number of ways you 
might manage the replform configuration across all the LDAP hosts (eg, Puppet) 
but our normal practice is simple: we keep the ‘replform’ configuration in 
source control and have a cron job on each LDAP host that periodically checks 
for updates to the configuration and executes ‘replform’.  This is all setup 
automatically when the host is provisioned by Terraform.  That works equally 
well for a new host that needs to configure replication from scratch and for 
all the existing LDAP hosts which just need to create a replication agreement 
to the new one.

>From past experience, it is quite possible to bring Terraform, Vault, and 
>‘replform’ together to create an entire LDAP cluster, including issuing SSL 
>certs and configuring MMR replication, completely automated.

 

I’ll add an example cron script to GitHub on Monday for the replform part.

 

Cheers,

 

-- 

Thomas E Lackey

 

From: Michal Medvecky  
Sent: Friday, June 22, 2018 4:07 AM
To: General discussion list for the 389 Directory server project. 
<389-users@lists.fedoraproject.org>
Subject: [389-users] Re: Master-slave replication procedure

 

Hello,





19. 6. 2018 v 23:43, Thomas E Lackey mailto:telac...@bozemanpass.com> >:

 

By happy timing, we (Bozeman Pass) just added one of our in-house tools for 
configuring replication to GitHub:  <https://github.com/bozemanpass/replform> 
https://github.com/bozemanpass/replform.

 

I had a look at this but I don’t like the fact you need to statically define 
the configuration. I have variable number of masters and variable number of 
slaves and if I understand this correctly, adding a new backend server would 
need copypasting the replform config.

 

Have you considered creating LDAP providers for Terraform itself?

 

Michal

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/UQWBNKMNC23EWS7HKUZ2GH2JLYEICM6M/

[389-users] Re: Master-slave replication procedure

[Michal Medvecky]

Hello,

> 19. 6. 2018 v 23:43, Thomas E Lackey :
> 
> By happy timing, we (Bozeman Pass) just added one of our in-house tools for 
> configuring replication to GitHub: https://github.com/bozemanpass/replform 
> .

I had a look at this but I don’t like the fact you need to statically define 
the configuration. I have variable number of masters and variable number of 
slaves and if I understand this correctly, adding a new backend server would 
need copypasting the replform config.

Have you considered creating LDAP providers for Terraform itself?

Michal___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/BJGMW7P4GB7A6U3EVK7N354USFD37763/

[389-users] Re: Master-slave replication procedure

[Michal Medvecky]

So I experimented with that and the most reliable option is to write an entry 
to one of masters and then wait until entries appear on all replicas.

Btw the other thing I changed is that I ran tests over VMs instead of 
containers (bcz. I was unable to run syslog in my container env) - for real 
services we use VMs only, but for running automated tests, I was always using 
containers.

Michal

> 20. 6. 2018 v 20:51, Mark Reynolds :
> 
> Michal,
> 
> You can check these attributes in the agmt:
> 
> nsds5replicalastinitend
> 
> nsds5replicalastinitstatus
> 
> These are probably more accurate for what you are trying to do.
> 
> Regards,
> 
> Mark
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/DHXZKMPQMOJTD35JNLQBNUF6PMXRI3R7/

[389-users] Re: Master-slave replication procedure

[Mark Reynolds]

Michal,

You can check these attributes in the agmt:

    nsds5replicalastinitend

    nsds5replicalastinitstatus

These are probably more accurate for what you are trying to do.

Regards,

Mark


On 06/20/2018 07:02 AM, Michal Medvecky wrote:

Adding a 60s sleep between replication setup (step 5) and tests (step 6) helped.

I am not sure waiting for nsds5BeginReplicaRefresh disappearance is enough…

Michal
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/OPJIWGBETMCNXFIRF3DXPB4TJAH65GDX/

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/QBBKHVW4N4FOS2GUN5AKDA3EIUPQ5JQI/

[389-users] Re: Master-slave replication procedure

[Michal Medvecky]

Adding a 60s sleep between replication setup (step 5) and tests (step 6) 
helped. 

I am not sure waiting for nsds5BeginReplicaRefresh disappearance is enough…

Michal
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/OPJIWGBETMCNXFIRF3DXPB4TJAH65GDX/

[389-users] Re: Master-slave replication procedure

[Mark Reynolds]

On 06/19/2018 05:45 PM, Michal Medvecky wrote:


19. 6. 2018 v 23:08, Mark Reynolds >:



2) create LDAP entry:
dn: cn=replica,cn=“dc=test,dc=com”,cn=mapping tree,cn=config;
      nsds5replicaroot: “dc=test,dc=com"
      nsds5replicaid: "{{ range(1,65530) | random }}"
If these are read-only consumers the replica ID must be 65535 (for 
all of them) - not a random number.  Only masters get unique replica  
IDs. This is probably your problem.  Fix this first, and if you still 
have problems then turn on replication error logging and share the 
results.


Changing to 65535 did not help. The results are the same.
Then please turn on replication error logging and share the logging 
results.




Michal


___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/EHCXWRBBCUTTLB57DGK7OAEP635DF3AT/

[389-users] Re: Master-slave replication procedure

[Thomas E Lackey]

By happy timing, we (Bozeman Pass) just added one of our in-house tools for
configuring replication to GitHub: https://github.com/bozemanpass/replform.

 

We call it `replform` as a slight nod towards Hashicorp Terraform.
Obviously they are very different beasts, but what they have in common is
that you specify the infrastructure / topology that you want, and the tool
figures out what changes need to be made to make it happen (eg, create
replication accounts, replica entries, changelogs, agreements, initialize
the replicas, etc.)

 

Running `replform plan` will tell you what it intends to do and `replform
apply` will make it happen.  It is safe to run over again, can handle adding
new hosts, etc.

 

It sounds like it might work for your case.  There is more documentation and
a few examples at: https://github.com/bozemanpass/replform

 

-- 

Thomas E Lackey

 

P.S. Some additional notes and warnings:  It supports build MMR or
supplier/consumer replication topologies (or some mix of the two), but not
"chaining" replication with hubs.  The other is that while it handles the
removal of a consumer automatically, it does not do the RUV cleaning
necessary when removing a supplier.  Lastly, `replform` also supports
pulling credentials for 'cn=repman' and 'cn=Directory Manager' out of
Hashicorp Vault, but that require some Vault tools we created that haven't
made their way into GitHub yet (though they likely will).

 

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/DF4GRUKDISAN4QGICISQNDOOAOFCJ3DF/

[389-users] Re: Master-slave replication procedure

[Mark Reynolds]

On 06/19/2018 04:47 PM, Michal Medvecky wrote:

Hello,

I’m trying hard to figure out the right (ansible-automated) procedure 
for setting up master-slave replication, but I often get RUV errors on 
agreements pointing to already initialized replicas.


My scenario is with 4 master servers (with multimaster replication 
working correctly) and 4 (independent) slave servers.


List of steps:

0) setup master-master replication between master servers (works OK)

1) create replication user cn=myreplicationusername,cn=config on all 
slaves


2) create LDAP entry:
dn: cn=replica,cn=“dc=test,dc=com”,cn=mapping tree,cn=config;
      nsds5replicaroot: “dc=test,dc=com"
      nsds5replicaid: "{{ range(1,65530) | random }}"
If these are read-only consumers the replica ID must be 65535 (for all 
of them) - not a random number.  Only masters get unique replica  IDs.  
This is probably your problem. Fix this first, and if you still have 
problems then turn on replication error logging and share the results.


Thanks,
Mark

      nsds5replicatype: “2"
      nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
      nsds5flags: “0”

3) create ro agreement from every master to every slave
on every master server, create LDAP entry
for every slave:
    dn: “cn=ro-to-{{ one of slaves 
}},cn=replica,cn=“dc=test,dc=com",cn=mapping tree,cn=config"

    objectClass:
      - nsds5replicationagreement
      - top
    attributes:
      nsds5replicahost: "{{ one of slaves }}"
      nsds5replicaport: “389"
      nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
      nsds5replicabindmethod: “SIMPLE"
      nsds5ReplicaTransportInfo: “LDAP"
      nsds5replicaroot: “dc=test,dc=com"
      description: "Agreement between {{ me }} and {{ one of slaves }}"
      nsds5replicaupdateschedule: "0001-2359 0123456"
      nsds5replicatedattributelist: "(objectclass=*) $ EXCLUDE 
authorityRevocationList"

      nsds5replicacredentials: “unbreakable"

4) refresh replicas (Created in 2)) on all hosts except the first master

on {{ first master server }} update all agreements with 
nsds5BeginReplicaRefresh: “start”


5) wait until nsds5BeginReplicaRefresh attribute disappears

6) run tests.

And this is the pain point and the reason I’m emailing the list - I 
add a dummy record to every master server and check it on all slaves.


But tests often fail on a random server.

# ./test.sh
Testing master-slave replication ...
---
Adding entry to ldap-master01.test.com 
adding new entry "uid=slave-repl-test-1,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-1 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master01
deleting entry "uid=slave-repl-test-1,dc=test,dc=com"

---
Adding entry to ldap-master02.test.com 
adding new entry "uid=slave-repl-test-2,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-2 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master02
deleting entry "uid=slave-repl-test-2,dc=test,dc=com"

---
Adding entry to ldap-master03.test.com 
adding new entry "uid=slave-repl-test-3,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-3 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master03
deleting entry "uid=slave-repl-test-3,dc=test,dc=com"

---
Adding entry to ldap-master04.test.com 
adding new entry "uid=slave-repl-test-4,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-4 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master04
deleting entry "uid=slave-repl-test-4,dc=test,dc=com”

List agreement update status on ldap-master01:

ldap-master01:

dn: cn=ro-to-ldap-slave01.test.com 
,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config

cn: ro-to-ldap-slave01.test.com 
nsds5replicaLastUpdateStatus: Error (1) Can't acquire busy replica

dn: cn=ro-to-ldap-slave02.test.com 
,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config

cn: ro-to-ldap-slave02.test.com