Howard Chu wrote:
Howard Chu wrote:
In at least one case I'm seeing a valid
update being rejected because the incoming cookie seems to have been
confused
with another one. This happens when a NEW_COOKIE message is received.
I'll
note that sending NEW_COOKIE messages is a recent change (ITS#5972),
and there
is no valid case for them to be occurring in test050. I.e., NEW_COOKIE
should
be sent in a partial replication situation, where an entry changed in the
naming context but it's not within the consumer's scope of interest. In
test050, the consumer's scope of interest is the entire naming
context. So
this at least gives me one area to look for a fix.
I agree, in a MMR configuration NEW_COOKIE messages should not have been
sent, except possibly when the entire csn set is updated at the end of a
refresh phase. But is looks more and more to me as if the fact that
test050 do show these messages is a symptom of some entry updates being
ignored by syncprov, or not passed to syncprov by syncrepl.
This piece of the ITS#5972 patch is part of the problem
--- syncprov.c 5 Mar 2009 16:53:01 -0000 1.266
+++ syncprov.c 12 Mar 2009 08:42:54 -0000
@@ -1245,7 +1245,7 @@
} else if ( !saveit && found ) {
/* send DELETE */
syncprov_qresp( opc, ss, LDAP_SYNC_DELETE );
- } else if ( !saveit ) {
+ } else if ( !saveit && !fc.fscope ) {
syncprov_qresp( opc, ss, LDAP_SYNC_NEW_COOKIE );
}
if ( !saveit && found ) {
My diff above is also not the correct fix, which is why I haven't
committed it yet.
The current operation may not have been caught by the previous if
conditions for 3 reasons:
1) the change is out of the consumer's scope
2) the change doesn't match the consumer's filter
3) the change is older than the consumer's cookie
The NEW_COOKIE message must only be sent for conditions 1 and 2, but
it's currently also being sent for 3. Since the cookie comparison is
tacked onto the consumer's filter, an additional comparison is needed to
weed this out.
(Normally 3 can't be true, but this is MMR where the consumer might have
already received this change from some other provider.)
Syncprov generally doesn't know the exact state of its consumers in MMR
configurations, since the consumers CSNs could have been updated by one
of the other providers. So, the NEW_COOKIE messages should be sent in
all three cases, leaving the job of filtering out the too old CSNs to
the one that have enough information to do so, namely the consumer.
I haven't looked yet, but I suspect there is a corresponding bug in the
consumer where it acts on a NEW_COOKIE message whether it's valid or not.
No, the consumer silently ignores updates to CSN values older (or equal)
to the values it already knows about.
I'm also inclined to back out #5972 and its related patches (#5973,
#6001) for
this release. We were looking for bug fixes and stability, and they've
been quite destabilizing.
To me it looks more as the extended test050 have triggered race
conditions that already was there, and that especially the syncprov half
of ITS#5973 have added to the likelihood that they should be shown.
I have run the current test050 script with the 2.4.15 source (which
didn't include these patches), and with RE24 (as of two days ago)
without ITS#5973, and have seen the same type of failures there. Also,
had the problems been triggered by the consumers receiving NEW_COOKIE
messages then I would have expected to see "too old" messages on the
consumers when it ignores entries. Instead, I find no trace of the
missing entries ever being passed on from the provider. But where the
update is lost I haven't found out yet. The problem seem to occur when
the server where entries are missing receives its updates from one of
the other consumers (i.e, not directly from server1). But whether it is
syncrepl on this intermediate server that fails to pass it on to
syncprov, or syncprov that looses them, I don't know.
Also, I now have around 30 core files similar to the one in ITS#5999,
and I have also had a number of cases where I had to kill -9 a slapd
running in a tight unlock, yield, lock loop at the same place in
syncprov_op_mod(). These loops have all happened when slapd should be
stopping, and the mt structure looks equally invalid as with the seg.
fault cases. I have no idea as to whether this has anything to do with
the test050 failures or not.
Btw, all of the test050 failures I have seen due to missing replications
have taken place immediately after the initial loading of the consumers
from server1. This could be a coincident, but I have had enough or them
to start wondering...
Rein
