Hi!
we have tried to use event / timestamp based synchronization from
an external user database to repository. It seems that Syncope
calls sync and getLastSyncToken connector framework operations in
a following manner when time goes on:
t1.1 sync, token = null
t1.2 getLastSyncToken
t2.1 sync, token = value got from t1.2
t2.2 getLastSyncToken
t3.1 sync, token = value got from t2.2
t3.2 getLastSyncToken
in ti.j i means task execution iteration number and j means
called ICF operation number in current task execution.
Why getLastSyncToken is used at all? Could it just continue from
the time which was processed in an earlier run, or use
getLastSyncToken to test if there is something newer and then
call sync only if something is available?
There seems to be quite a serious problem in the current
implementation. Let's suppose that we are going to process step
t2.1 and there are three new events e1, e2 and e3 available. And
let us further suppose that two new items e4 and e5 will happen
during t2.1. In t2.2 token will be set to e5 which means that in
the step t3.1 we will continue from e5 and e4 will be missed!
We have tested this with database table connector and scripted
sql connector and the problem exists in both implementations.
Hi Timo, yes, you are right, there is a big problem.
I'm quite sure it wasn't in the past: order of calls should be inverted.
I mean:
1. get new sync token
2. execute sync with the oldest one
3. replace the old one whit the new one
In this case we risk to perform twise the same operation but, at
least, it shouldn be a problem.
Does it make sense?
Yes. I suppose that this depens on the comparison opetaror used,
">" or ">=". DB seems to use ">" and scripted SQL examples ">=".
Why don't you open a new issue for this problem? The fix will be
released into the 1.1.5 and 1.2.0.
I will!
Hi,
I've seen you've opened SYNCOPE-440, thanks!
Would you be able to provide a patch? The change should be likely
located in SyncJob class [1].
You can find some guidelines about how to submit a patch [2].
Sorry but we don't have any possibility to patch it just now. We are
doing quite deep POC testing, which should be finnished in a couple of
weeks. After that we will decide whether we start a pilot project. If
this is still open we could try to fix it then. However we are not
very familiar with internal desing of syncope, so it could be much
easier for you!
Regards,
Timo
