[
https://issues.apache.org/jira/browse/OAK-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Parvulescu updated OAK-3436:
---------------------------------
Attachment: OAK-3436-part2.patch
attached part deux, 'reindexing': [^OAK-3436-part2.patch]
there were some unexpected changes in behavior here because reindexing is
allowed to bypass the lease check not sure why ( see [0] when {{checkpoint ==
null}}.
removing the 'reindex' clause makes reindexing at like a normal index, but it
changes the {{testPrePrepareRexindex}} test. An interesting side-efect is that
it allows the second indexer ('special' one) to fail early on, whereas before
it kinda looks like it would finish in parallel (there are some index barriers
that check the lease still on indexUpdate events. It also makes the test &
expectations more consistent with the {{testPrePrepare}} test.
For reference, the actual test that surfaced this nuisance is
{{testPostPrepareReindexLeaseExpired}}.
As I can't directly remember the story behind the reindex bypass, I'm going to
assume it was some sort of optimization from before the lease days, which may
not play well with the current setup (I would expect a reindex to respect the
lease).
I'm attaching the patch here for now, to collect feedback.
[0]
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/AsyncIndexUpdate.java#L505
> Prevent missing checkpoint due to unstable topology from causing complete
> reindexing
> ------------------------------------------------------------------------------------
>
> Key: OAK-3436
> URL: https://issues.apache.org/jira/browse/OAK-3436
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
> Reporter: Chetan Mehrotra
> Assignee: Alex Parvulescu
> Labels: resilience
> Fix For: 1.3.13, 1.0.26, 1.2.10
>
> Attachments: AsyncIndexUpdateClusterTest.java, OAK-3436-0.patch,
> OAK-3436-part2.patch, OAK-3436-tests.patch, OAK-3436-v2.patch
>
>
> Async indexing logic relies on embedding application to ensure that async
> indexing job is run as a singleton in a cluster. For Sling based apps it
> depends on Sling Discovery support. At times it is being seen that if
> topology is not stable then different cluster nodes can consider them as
> leader and execute the async indexing job concurrently.
> This can cause problem as both cluster node might not see same repository
> state (due to write skew and eventual consistency) and might remove the
> checkpoint which other cluster node is still relying upon. For e.g. consider
> a 2 node cluster N1 and N2 where both are performing async indexing.
> # Base state - CP1 is the checkpoint for "async" job
> # N2 starts indexing and removes changes CP1 to CP2. For Mongo the
> checkpoints are saved in {{settings}} collection
> # N1 also decides to execute indexing but has yet not seen the latest
> repository state so still thinks that CP1 is the base checkpoint and tries to
> read it. However CP1 is already removed from {{settings}} and this makes N1
> think that checkpoint is missing and it decides to reindex everything!
> To avoid this topology must be stable but at Oak level we should still handle
> such a case and avoid doing a full reindexing. So we would need to have a
> {{MissingCheckpointStrategy}} similar to {{MissingIndexEditorStrategy}} as
> done in OAK-2203
> Possible approaches
> # A1 - Fail the indexing run if checkpoint is missing - Checkpoint being
> missing can have valid reason and invalid reason. Need to see what are valid
> scenarios where a checkpoint can go missing
> # A2 - When a checkpoint is created also store the creation time. When a
> checkpoint is found to be missing and its a *recent* checkpoint then fail the
> run. For e.g. we would fail the run till checkpoint found to be missing is
> less than an hour old (for just started take startup time into account)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
