[389-users] Re: Master-slave replication procedure

2018-06-19 Thread Mark Reynolds 



On 06/19/2018 05:45 PM, Michal Medvecky wrote:


19. 6. 2018 v 23:08, Mark Reynolds >:



2) create LDAP entry:
dn: cn=replica,cn=“dc=test,dc=com”,cn=mapping tree,cn=config;
      nsds5replicaroot: “dc=test,dc=com"
      nsds5replicaid: "{{ range(1,65530) | random }}"
If these are read-only consumers the replica ID must be 65535 (for 
all of them) - not a random number.  Only masters get unique replica  
IDs. This is probably your problem.  Fix this first, and if you still 
have problems then turn on replication error logging and share the 
results.


Changing to 65535 did not help. The results are the same.
Then please turn on replication error logging and share the logging 
results.




Michal


___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/EHCXWRBBCUTTLB57DGK7OAEP635DF3AT/

[389-users] Re: Master-slave replication procedure

2018-06-19 Thread Thomas E Lackey 
By happy timing, we (Bozeman Pass) just added one of our in-house tools for
configuring replication to GitHub: https://github.com/bozemanpass/replform.

 

We call it `replform` as a slight nod towards Hashicorp Terraform.
Obviously they are very different beasts, but what they have in common is
that you specify the infrastructure / topology that you want, and the tool
figures out what changes need to be made to make it happen (eg, create
replication accounts, replica entries, changelogs, agreements, initialize
the replicas, etc.)

 

Running `replform plan` will tell you what it intends to do and `replform
apply` will make it happen.  It is safe to run over again, can handle adding
new hosts, etc.

 

It sounds like it might work for your case.  There is more documentation and
a few examples at: https://github.com/bozemanpass/replform

 

-- 

Thomas E Lackey

 

P.S. Some additional notes and warnings:  It supports build MMR or
supplier/consumer replication topologies (or some mix of the two), but not
"chaining" replication with hubs.  The other is that while it handles the
removal of a consumer automatically, it does not do the RUV cleaning
necessary when removing a supplier.  Lastly, `replform` also supports
pulling credentials for 'cn=repman' and 'cn=Directory Manager' out of
Hashicorp Vault, but that require some Vault tools we created that haven't
made their way into GitHub yet (though they likely will).

 

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/DF4GRUKDISAN4QGICISQNDOOAOFCJ3DF/

[389-users] Re: Master-slave replication procedure

2018-06-19 Thread Mark Reynolds 



On 06/19/2018 04:47 PM, Michal Medvecky wrote:

Hello,

I’m trying hard to figure out the right (ansible-automated) procedure 
for setting up master-slave replication, but I often get RUV errors on 
agreements pointing to already initialized replicas.


My scenario is with 4 master servers (with multimaster replication 
working correctly) and 4 (independent) slave servers.


List of steps:

0) setup master-master replication between master servers (works OK)

1) create replication user cn=myreplicationusername,cn=config on all 
slaves


2) create LDAP entry:
dn: cn=replica,cn=“dc=test,dc=com”,cn=mapping tree,cn=config;
      nsds5replicaroot: “dc=test,dc=com"
      nsds5replicaid: "{{ range(1,65530) | random }}"
If these are read-only consumers the replica ID must be 65535 (for all 
of them) - not a random number.  Only masters get unique replica  IDs.  
This is probably your problem. Fix this first, and if you still have 
problems then turn on replication error logging and share the results.


Thanks,
Mark

      nsds5replicatype: “2"
      nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
      nsds5flags: “0”

3) create ro agreement from every master to every slave
on every master server, create LDAP entry
for every slave:
    dn: “cn=ro-to-{{ one of slaves 
}},cn=replica,cn=“dc=test,dc=com",cn=mapping tree,cn=config"

    objectClass:
      - nsds5replicationagreement
      - top
    attributes:
      nsds5replicahost: "{{ one of slaves }}"
      nsds5replicaport: “389"
      nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
      nsds5replicabindmethod: “SIMPLE"
      nsds5ReplicaTransportInfo: “LDAP"
      nsds5replicaroot: “dc=test,dc=com"
      description: "Agreement between {{ me }} and {{ one of slaves }}"
      nsds5replicaupdateschedule: "0001-2359 0123456"
      nsds5replicatedattributelist: "(objectclass=*) $ EXCLUDE 
authorityRevocationList"

      nsds5replicacredentials: “unbreakable"

4) refresh replicas (Created in 2)) on all hosts except the first master

on {{ first master server }} update all agreements with 
nsds5BeginReplicaRefresh: “start”


5) wait until nsds5BeginReplicaRefresh attribute disappears

6) run tests.

And this is the pain point and the reason I’m emailing the list - I 
add a dummy record to every master server and check it on all slaves.


But tests often fail on a random server.

# ./test.sh
Testing master-slave replication ...
---
Adding entry to ldap-master01.test.com 
adding new entry "uid=slave-repl-test-1,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-1 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master01
deleting entry "uid=slave-repl-test-1,dc=test,dc=com"

---
Adding entry to ldap-master02.test.com 
adding new entry "uid=slave-repl-test-2,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-2 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master02
deleting entry "uid=slave-repl-test-2,dc=test,dc=com"

---
Adding entry to ldap-master03.test.com 
adding new entry "uid=slave-repl-test-3,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-3 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master03
deleting entry "uid=slave-repl-test-3,dc=test,dc=com"

---
Adding entry to ldap-master04.test.com 
adding new entry "uid=slave-repl-test-4,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-4 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master04
deleting entry "uid=slave-repl-test-4,dc=test,dc=com”

List agreement update status on ldap-master01:

ldap-master01:

dn: cn=ro-to-ldap-slave01.test.com 
,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config

cn: ro-to-ldap-slave01.test.com 
nsds5replicaLastUpdateStatus: Error (1) Can't acquire busy replica

dn: cn=ro-to-ldap-slave02.test.com 
,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config

cn: ro-to-ldap-slave02.test.com

[389-users] Master-slave replication procedure

2018-06-19 Thread Michal Medvecky 
Hello,

I’m trying hard to figure out the right (ansible-automated) procedure for 
setting up master-slave replication, but I often get RUV errors on agreements 
pointing to already initialized replicas.

My scenario is with 4 master servers (with multimaster replication working 
correctly) and 4 (independent) slave servers.

List of steps:

0) setup master-master replication between master servers (works OK)

1) create replication user cn=myreplicationusername,cn=config on all slaves 

2) create LDAP entry:
dn: cn=replica,cn=“dc=test,dc=com”,cn=mapping tree,cn=config; 
  nsds5replicaroot: “dc=test,dc=com"
  nsds5replicaid: "{{ range(1,65530) | random }}"
  nsds5replicatype: “2"
  nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
  nsds5flags: “0”

3) create ro agreement from every master to every slave
on every master server, create LDAP entry
for every slave:
dn: “cn=ro-to-{{ one of slaves }},cn=replica,cn=“dc=test,dc=com",cn=mapping 
tree,cn=config"
objectClass:
  - nsds5replicationagreement
  - top
attributes:
  nsds5replicahost: "{{ one of slaves }}"
  nsds5replicaport: “389"
  nsds5ReplicaBindDN: “cn=myreplicationusername,cn=config"
  nsds5replicabindmethod: “SIMPLE"
  nsds5ReplicaTransportInfo: “LDAP"
  nsds5replicaroot: “dc=test,dc=com"
  description: "Agreement between {{ me }} and {{ one of slaves }}"
  nsds5replicaupdateschedule: "0001-2359 0123456"
  nsds5replicatedattributelist: "(objectclass=*) $ EXCLUDE 
authorityRevocationList"
  nsds5replicacredentials: “unbreakable"

4) refresh replicas (Created in 2)) on all hosts except the first master 

on {{ first master server }} update all agreements with 
nsds5BeginReplicaRefresh: “start” 

5) wait until nsds5BeginReplicaRefresh attribute disappears

6) run tests. 

And this is the pain point and the reason I’m emailing the list - I add a dummy 
record to every master server and check it on all slaves.

But tests often fail on a random server.

# ./test.sh
Testing master-slave replication ...
---
Adding entry to ldap-master01.test.com
adding new entry "uid=slave-repl-test-1,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-1 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-1 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master01
deleting entry "uid=slave-repl-test-1,dc=test,dc=com"

---
Adding entry to ldap-master02.test.com
adding new entry "uid=slave-repl-test-2,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-2 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-2 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master02
deleting entry "uid=slave-repl-test-2,dc=test,dc=com"

---
Adding entry to ldap-master03.test.com
adding new entry "uid=slave-repl-test-3,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-3 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-3 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master03
deleting entry "uid=slave-repl-test-3,dc=test,dc=com"

---
Adding entry to ldap-master04.test.com
adding new entry "uid=slave-repl-test-4,dc=test,dc=com"

Checking entry on slave servers
Checking uid=slave-repl-test-4 on ldap-slave01 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave02 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave03 ... 1 results ✓
Checking uid=slave-repl-test-4 on ldap-slave04 ... 0 results ☠
Removing entry from ldap-master04
deleting entry "uid=slave-repl-test-4,dc=test,dc=com”

List agreement update status on ldap-master01:

ldap-master01: 

dn: 
cn=ro-to-ldap-slave01.test.com,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config
cn: ro-to-ldap-slave01.test.com
nsds5replicaLastUpdateStatus: Error (1) Can't acquire busy replica

dn: 
cn=ro-to-ldap-slave02.test.com,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config
cn: ro-to-ldap-slave02.test.com
nsds5replicaLastUpdateStatus: Error (1) Can't acquire busy replica

dn: 
cn=ro-to-ldap-slave03.test.com,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config
cn: ro-to-ldap-slave03.test.com
nsds5replicaLastUpdateStatus: Error (1) Can't acquire busy replica

dn: 
cn=ro-to-ldap-slave04.test.com,cn=replica,cn=dc\3Dtest\2Cdc\3Dcom,cn=mapping 
tree,cn=config
cn: ro-to-ldap-slave04.test.com
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: 
Replica has different database generation ID, remote replica may need to be 
initialized (RUV error)


The fourth agreement seems unin

[389-users] Announcing 389 Directory Server 1.4.0.11

2018-06-19 Thread Mark Reynolds 


   389 Directory Server 1.4.0.11

The 389 Directory Server team is proud to announce 389-ds-base version 
1.4.0.11


Fedora packages are available on Fedora 28 and 29(rawhide).

Rawhide(F29)

https://koji.fedoraproject.org/koji/taskinfo?taskID=27738498 



F28

https://koji.fedoraproject.org/koji/taskinfo?taskID=27738822 



https://bodhi.fedoraproject.org/updates/FEDORA-2018-b3f2f29e65 



The new packages and versions are:

 * 389-ds-base-1.4.0.11-1

Source tarballs are available for download at Download 
389-ds-base Source 




 Highlights in 1.4.0.11

Bug fixes


 Installation and Upgrade

See Download  for 
information about setting up your yum repositories.


To install, use *dnf install 389-ds-base*, then run *dscreate*. For 
Cockput UI plugin use “dnf install cockpit-389-ds”


See Install_Guide 
 for 
more information about the initial installation, setup, and upgrade


See Source  
for information about source tarballs and SCM (git) access.



 Feedback

We are very interested in your feedback!

Please provide feedback and comments to the 389-users mailing list: 
https://lists.fedoraproject.org/admin/lists/389-users.lists.fedoraproject.org


If you find a bug, or would like to see a new feature, file it in our 
Pagure project: https://pagure.io/389-ds-base


 * Bump version to 1.4.0.11
 * Ticket 49788 - Add test for ticket #49788
 * Ticket 49788 - Fixing 4-byte UTF-8 character validation
 * Ticket 49777 - add config subcommand to dsconf
 * Ticket 49712 - lib389 CLI tools should return a result code on failures
 * Ticket 49588 - Add py3 support for tickets : part-2
 * Remove old RHEL/fedora version checking from upstream specfile
 * Ticket 48204 - remove python2 from scripts
 * Ticket 49576 - ds-replcheck: fix certificate directory verification
 * Bug 1591761 - 389-ds-base: Remove jemalloc exports

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/HFBKOP5AUSIJP2DFQGY4F23RM4TG5L3W/