Hi Oleg,
the problem seems to be on replica2, when it logs this error:
[16/Jun/2015:10:18:34 -0400] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: retry (49) the transaction
(csn=55802fcf000300040000) failed (rc=-30993 (BDB0068 DB_LOCK_DEADLOCK:
Locker killed to resolve a deadlock))
[16/Jun/2015:10:18:34 -0400] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: failed to write entry with csn
(55802fcf000300040000); db error - -30993 BDB0068 DB_LOCK_DEADLOCK:
Locker killed to resolve a deadlock
[16/Jun/2015:10:18:34 -0400] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=onmaster,cn=users,cn=accounts,dc=bagam,dc=net (uniqid:
b8242e18-143111e5-b1d0d0c3-ae5854ff, optype: 32) to changelog csn
55802fcf000300040000
[16/Jun/2015:10:18:34 -0400] - SLAPI_PLUGIN_BE_TXN_POST_DELETE_FN plugin
returned error code but did not set SLAPI_RESULT_CODE
but replication seems to continue and not to repeat this:
[16/Jun/2015:10:18:27 -0400] conn=8 op=6 DEL
dn="uid=onmaster,cn=users,cn=accounts,dc=bagam,dc=net"
[16/Jun/2015:10:18:35 -0400] conn=8 op=6 RESULT err=1 tag=107 nentries=0
etime=8 csn=55802fcf000300040000
[16/Jun/2015:10:18:35 -0400] conn=8 op=7 MOD
dn="cn=ipausers,cn=groups,cn=accounts,dc=bagam,dc=net"
[16/Jun/2015:10:18:36 -0400] conn=8 op=7 RESULT err=0 tag=103 nentries=0
etime=1 csn=55802fcf000400040000
[16/Jun/2015:10:18:36 -0400] conn=8 op=8 DEL
dn="cn=onmaster,cn=groups,cn=accounts,dc=bagam,dc=net"
[16/Jun/2015:10:18:37 -0400] conn=8 op=8 RESULT err=0 tag=107 nentries=0
etime=1 csn=55802fcf000700040000
[16/Jun/2015:10:18:37 -0400] conn=8 op=9 MOD
dn="cn=ipausers,cn=groups,cn=accounts,dc=bagam,dc=net"
[16/Jun/2015:10:18:37 -0400] conn=8 op=9 RESULT err=0 tag=103 nentries=0
etime=0 csn=55802fd0000000060000
I don't see why there is a deadlock ?
Is it reproducable every time ?
On 06/16/2015 04:49 PM, Oleg Fayans wrote:
Hi all,
I've bumped into a strange problem with only a part of changes
implemented on master during replica outage get replicated after
replica recovery.
Namely: when I delete an existing user on the master while the node is
offline, these changes do not get to the node when it's back online.
User creation, however, gets replicated as expected.
Steps to reproduce:
1. Create the following tolopogy:
replica1 <-> master <-> replica2 <-> replica3
2. Create user1 on master, make sure it appears on all replicas
3. Turn off replica2
4. On master delete user1 and create user2, make sure the changes get
replicated to replica1
5. Turn on replica2
Expected results:
A minute or so after repica2 is back up,
1. user1 does not exist neither on replica2 nor on replica3
2. user2 exists both on replica2 and replica3
Actual results:
1. user1 coexist with user2 on replica2 and replica3
2. master and replica1 have only user2
In my case, though, the topology was as follows:
$ ipa topologysegment-find realm
------------------
3 segments matched
------------------
Segment name: f22master.bagam.net-to-f22replica3.bagam.net
Left node: f22master.bagam.net
Right node: f22replica3.bagam.net
Connectivity: both
Segment name: replica1-to-replica2
Left node: f22replica1.bagam.net
Right node: f22replica2.bagam.net
Connectivity: both
Segment name: replica2-to-master
Left node: f22replica2.bagam.net
Right node: f22master.bagam.net
Connectivity: both
----------------------------
Number of entries returned 3
----------------------------
And I was turning off replica2, leaving replica1 offline, but that
does not really matter.
The dirsrv error message, most likely to be relevant is:
-----------------------------------------------------------------------------------------------------------------------------------------------------
Consumer failed to replay change (uniqueid
b8242e18-143111e5-b1d0d0c3-ae5854ff, CSN 55802fcf000300040000):
Operations error (1). Will retry later
-----------------------------------------------------------------------------------------------------------------------------------------------------
I attach dirsrv error and access logs from all nodes, in case they
could be useful
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
