[
https://issues.apache.org/jira/browse/OAK-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738603#comment-14738603
]
Vikas Saurabh commented on OAK-3380:
------------------------------------
{quote}
If there are multiple cluster nodes, then which one would do the pruning? How
would the others inform the one that does the pruning?
We could do some kind of "garbage collection" process, but that would take some
time as it would have to process the whole index.
{quote}
I thought that we can think along the lines of async indexers (well pruner in
this case) which would hence run only on leader nodes. SInce, the diff gets
calculated anyway for other async indexers, this one could just look for nodes
being removed under property index trees and then inspect those paths for
further pruning).
Having a marker node like {:trash} could also be an interesting idea - I'm
assuming if the pruner finds a trash node but it has got some siblings by the
time it runs it just removes the trash node.
{quote}
instead of pruning while updating the index, we might be able to prune while
reading from the index. I'm not sure if that's really a good idea (write while
reading), but anyway: when traversing a empty node, remove it with a
probability of let's say 5%
{quote}
While I don't have very strong feelings towards not writing while reading
(specially when the structure is oak's implementation detail and is not exposed
at JCR) - but I feel it removes a bit of determinism which I feel uncomfortable
with.
> Property index pruning should happen asynchronously
> ---------------------------------------------------
>
> Key: OAK-3380
> URL: https://issues.apache.org/jira/browse/OAK-3380
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
> Affects Versions: 1.3.5
> Reporter: Vikas Saurabh
> Priority: Minor
> Labels: resilience
>
> Following up on this (a relatively old) thread \[1], we should do pruning of
> property index structure asynchronously. The thread was never concluded..
> here are a couple of ideas picked from the thread:
> * Move pruning to an async thread
> * Throttle pruning i.e. prune only once in a while
> ** I'm not sure how that would work though -- an unpruned part would remain
> as is until another index happens on that path.
> Once we can move pruning to some async thread (reducing concurrent updates),
> OAK-2673 + OAK-2929 can take care of add-add conflicts.
> ----
> h6. Why is this an issue despite merge retries taking care of it?
> A couple of cases which have concurrent updates hitting merge conflicts in
> our product (Adobe AEM):
> * Some index are very volatile (in the sense that indexed property switches
> its values very quickly) e.g. sling job status, AEM workflow status.
> * Multiple threads take care of jobs. Although sling maintains a bucketed
> structure for job storage to reduce conflicts... but inside index tree the
> bucket structure, at times, gets pruned and needs to be created in the next
> job status change
> While retries do take care of these conflict a lot of times and even when
> they don't, AEM workflows has it's own retry to work around. But, retrying,
> IMHO, is just a waste of time -- more importantly in paths where application
> doesn't really have a control.
> h6. Would this add to cost of traversing index structure?
> Yes, there'd be some left over paths in index structure between asynchronous
> prunes. But, I think the cost of such wasted traversals would be covered up
> with time saved in avoiding the concurrent update conflict.
> ----
> (cc [~tmueller], [~mreutegg], [~alex.parvulescu], [~chetanm])
> \[1]:
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201506.mbox/%3ccadichf66u2vh-hlrjunansytxfidj2mt3vktr4ybkngpzy9...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
