[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[ https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993 ] Stefan Egli edited comment on OAK-4581 at 10/20/16 9:55 AM: h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more enclosed in the ChangeProcessor. The disadvantage is that if a listener would block it could not prevent the queue from growing (but maybe supporting an indefinitely blocking listener is not required). I'll look into how such an approach could be implemented next. UPDATE: Added a prototype of this second variant too (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581-type2]). h4. Comparison of the approaches The two approaches basically represent _'head of queue persisting'_ (PersistingEventListener) versus _'tail of queue persisting'_ (SwitchingObserver). h5. Falling out of the documentMk's diff cache * 'tail of queue persisting' has the advantage that it supports any type of storm coming in. It would immediately start persisting newly enqueued commits. (At the moment those diffs already in the queue are worked off 'slowly' at onEvent time, but this could be changed to be force-persisted too. EDIT: but basically 'tail of queue persisting' is a fail-safe for newly added commits which would allow the listener to have infinite time for working off the inmemory queue) * 'head of queue persisting' is less favorable when falling out of the diff cache, as it would basically start persisting the head at max speed (ie without onEvent cost), but perhaps that max speed is slower than new commits coming in. h5. Storm of commits coming in * 'tail of queue persisting' again is deterministic for this case as it just routes incoming commits to persistence for good - no possibility of further queue growth. And the remaining missing feature of not-yet-force-flushing the inmemory queue is not a problem here as we remain in the cache. * 'head of queue persisting' would likely be overwhelmed here: it depends on how rapidly it can filter/generate/persist off the queue vs how quickly new commits come in. h4. Conclusion While the head-of-queue-persisting/PersistingEventListener seems to be the more KISS/elegant solution, it has downsides under heavy load. It seems thus better to do tail-of-queue-persisting with eg the SwitchingObserver. What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon], [~tmueller] was (Author: egli): h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[ https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993 ] Stefan Egli edited comment on OAK-4581 at 10/20/16 9:50 AM: h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more enclosed in the ChangeProcessor. The disadvantage is that if a listener would block it could not prevent the queue from growing (but maybe supporting an indefinitely blocking listener is not required). I'll look into how such an approach could be implemented next. UPDATE: Added a prototype of this second variant too (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581-type2]). h4. Comparison of the approaches The two approaches basically represent _'head of queue persisting'_ (PersistingEventListener) versus _'tail of queue persisting'_ (SwitchingObserver). h5. Falling out of the documentMk's diff cache * 'tail of queue persisting' has the advantage that it supports any type of storm coming in. It would immediately start persisting newly enqueued commits. (At the moment those diffs already in the queue are worked off 'slowly' at onEvent time, but this could be changed to be force-persisted too) * 'head of queue persisting' is less favorable when falling out of the diff cache, as it would basically start persisting the head at max speed (ie without onEvent cost), but perhaps that max speed is slower than new commits coming in. h5. Storm of commits coming in * 'tail of queue persisting' again is deterministic for this case as it just routes incoming commits to persistence for good - no possibility of further queue growth. And the remaining missing feature of not-yet-force-flushing the inmemory queue is not a problem here as we remain in the cache. * 'head of queue persisting' would likely be overwhelmed here: it depends on how rapidly it can filter/generate/persist off the queue vs how quickly new commits come in. h4. Conclusion While the head-of-queue-persisting/PersistingEventListener seems to be the more KISS/elegant solution, it has downsides under heavy load. It seems thus better to do tail-of-queue-persisting with eg the SwitchingObserver. What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon], [~tmueller] was (Author: egli): h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more enclosed in the ChangeProcessor. The disadvantage is that if a listener would block it could not prevent the queue from growing (but maybe supporting an indefinitely blocking
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[ https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993 ] Stefan Egli edited comment on OAK-4581 at 10/20/16 9:49 AM: h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more enclosed in the ChangeProcessor. The disadvantage is that if a listener would block it could not prevent the queue from growing (but maybe supporting an indefinitely blocking listener is not required). I'll look into how such an approach could be implemented next. UPDATE: Added a prototype of this second variant too (in [this branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581-type2]). h4. Comparison of the approaches The two approaches basically represent _'head of queue persisting'_ (PersistingEventListener) versus _'tail of queue persisting'_ (SwitchingObserver). h5. Falling out of the documentMk's diff cache * 'tail of queue persisting' has the advantage that it supports any type of storm coming in. It would immediately start persisting newly enqueued commits. (At the moment those diffs already in the queue are worked off 'slowly' at onEvent time, but this could be changed to be force-persisted too) * 'head of queue persisting' is less favorable when falling out of the diff cache, as it would basically start persisting the head at max speed (ie without onEvent cost), but perhaps that max speed is slower than new commits coming in. h5. Storm of commits coming in * 'tail of queue persisting' again is deterministic for this case as it just routes incoming commits to persistence for good - no possibility of further queue growth. And the remaining missing feature of not-yet-force-flushing the inmemory queue is not a problem here as we remain in the cache. * 'head of queue persisting' would likely be overwhelmed here: it depends on how rapidly it can filter/generate/persist off the queue vs how quickly new commits come in. h4. Conclusion While the head-of-queue-persisting/PersistingEventListener seems to be the more KISS/elegant solution, it has downsides under heavy load. It seems thus better to do tail-of-queue-persisting with eg the SwitchingObserver. What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon], [~tmueller] was (Author: egli): h4. Prototype based on a SwitchingObserver [pushed a prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7] to my github fork which tries to scetch an approach that would use a SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this approach is that there are quite a few new classes in play. The advantage is that it clearly distinguishes between normal (live) mode - by using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new persistent mode. Switching between these modes is delicate and is thus explicitly handled with extra switching modes. This conceptually works fine - added a test that illustrates the switch. h4. Prototype based on persistence embedded in the ChangeProcessor An alternative to the above would be to embedd the persistence directly in the ChangeProcessor. The contentChanged method there would inspect the queue size and if too large, not deliver events to the listener anymore, but instead go via a persistent queue. The advantage is that it's more enclosed in the ChangeProcessor. The disadvantage is that if a listener would block it could not prevent the queue from growing (but maybe supporting an indefinitely blocking listener is not required). I'll look into how such an approach could be implemented
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993
]
Stefan Egli edited comment on OAK-4581 at 10/19/16 3:24 PM:
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork which tries to scetch an approach that would use a
SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair
or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of
this approach is that there are quite a few new classes in play. The advantage
is that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.
h4. Prototype based on persistence embedded in the ChangeProcessor
An alternative to the above would be to embedd the persistence directly in the
ChangeProcessor. The contentChanged method there would inspect the queue size
and if too large, not deliver events to the listener anymore, but instead go
via a persistent queue. The advantage is that it's more enclosed in the
ChangeProcessor. The disadvantage is that if a listener would block it could
not prevent the queue from growing (but maybe supporting an indefinitely
blocking listener is not required). I'll look into how such an approach could
be implemented next.
was (Author: egli):
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork which tries to scetch an approach that would use a
SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair
or a PersistingBackgroundObserver/PersistingEventQueue pair. The disadvantage
of this approach is that there are quite a few new classes in play. The
advantage is that it clearly distinguishes between normal (live) mode - by
using the exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a
new persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.
h4. Prototype based on persistence embedded in the ChangeProcessor
An alternative to the above would be to embedd the persistence directly in the
ChangeProcessor. The contentChanged method there would inspect the queue size
and if too large, not deliver events to the listener anymore, but instead go
via a persistent queue. The advantage is that it's more enclosed in the
ChangeProcessor. The disadvantage is that if a listener would block it could
not prevent the queue from growing (but maybe supporting an indefinitely
blocking listener is not required). I'll look into how such an approach could
be implemented next.
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526714#comment-15526714
]
Stefan Egli edited comment on OAK-4581 at 9/27/16 5:02 PM:
---
[~cziegeler], re
bq. Now, an additional reason at least for Sling was that the event bridge we
have was reading all added/changed nodes to get the resource type property.
-I can't find this dependency in the code, do you know where that's coded in
JcrResourceListener? IIUC then the reason for having these
addedEvents/changedEvents/removedEvents maps in onEvent is to be able to build
a JcrResourceChanged obj that contains all changed properties (unrelated to
resource type - but perhaps that's hidden somewhere down deep..)-
EDIT: Are you referring to
[OsgiObservationBridge.sendOsgiEvent:131|https://github.com/apache/sling/blob/9c424dfca6a802a6b66b4b7981a313c5856a0e1f/bundles/resourceresolver/src/main/java/org/apache/sling/resourceresolver/impl/observation/OsgiObservationBridge.java#L131]
? IIUC that reads the current resource explicitly to get the resourceType. So
sure, having the resource type as part of the event (as OAK-4853 would provide)
would be a handy thing, even if slighlty unexpected. However, I fail to see how
this justifies the addedEvents/changedEvents/removedEvents in
JcrResourceListener. IIUC the reason for building these maps is to be able to
generate _correct_ JcrResourceChange objs - as they must contain all properties
of a particular node - and those come in separate {{Events}}. So if the goal is
to prevent those maps to become huge (or to not have them at all), then this
has nothing to do with the resource type in my view. If we shouldn't rely on
the breadth-first traversal, then the only alternative would be to get all
properties that have been changed/added/removed on the corresponding {{Event}}
of the node (which is slightly different from OAK-4853). wdyt?
was (Author: egli):
[~cziegeler], re
bq. Now, an additional reason at least for Sling was that the event bridge we
have was reading all added/changed nodes to get the resource type property.
I can't find this dependency in the code, do you know where that's coded in
JcrResourceListener? IIUC then the reason for having these
addedEvents/changedEvents/removedEvents maps in onEvent is to be able to build
a JcrResourceChanged obj that contains all changed properties (unrelated to
resource type - but perhaps that's hidden somewhere down deep..)
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522963#comment-15522963
]
Vikas Saurabh edited comment on OAK-4581 at 9/26/16 12:53 PM:
--
Btw, just to note (I'm unable to concretely argue if we really want to do this
or not) - no matter how we proceed, the storage would be cluster-id specific.
So, do we want to cater the case where some cluster node gets obliterated (with
pending stored events) - would/should some other node pick that up? Adding the
disclaimer again that it's a case that would hit us once we start to advertise
"stronger guarantee of delivery of observation events" - not that we must fix
it (we can as document it that the guarantee is a best-effort thing).
If we go for it, then we'd also have to handle cases such as "cluster node
crashed and burnt to never come up again", "cluster node crashed for 2 hours
but got back up again", "some other cluster node waited for lease to expire and
acquired the same cluster id as the one which got crashed recently", etc. The
more we go ahead with solving this - it seems we'd start to invade into what
sling jobs do today (have concrete notion of assignee etc with re-assignments
as required)
was (Author: catholicon):
Btw, just to note (I'm unable to concretely argue if we really want to do this
or not) - no matter how we proceed, the storage would be cluster-id specific.
So, do we want to cater the case where some cluster node gets obliterated (with
pending stored events) - would/should some other node pick that up? Adding the
disclaimer again that it's a case that would hit us once we start to advertise
"stronger guarantee delivery of observation events" - not that we must fix it
(we can as document it that the guarantee is a best-effort thing).
If we go for it, then we'd also have to handle cases such as "cluster node
crashed and burnt to never come up again", "cluster node crashed for 2 hours
but got back up again", "some other cluster node waited for least and acquired
the same cluster id as the one which got crashed", etc. The more we go ahead
with solving this - it seems we'd start to invade into what sling jobs do today
(have concrete notion of assignee etc with re-assignments as required)
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522963#comment-15522963
]
Vikas Saurabh edited comment on OAK-4581 at 9/26/16 12:52 PM:
--
Btw, just to note (I'm unable to concretely argue if we really want to do this
or not) - no matter how we proceed, the storage would be cluster-id specific.
So, do we want to cater the case where some cluster node gets obliterated (with
pending stored events) - would/should some other node pick that up? Adding the
disclaimer again that it's a case that would hit us once we start to advertise
"stronger guarantee delivery of observation events" - not that we must fix it
(we can as document it that the guarantee is a best-effort thing).
If we go for it, then we'd also have to handle cases such as "cluster node
crashed and burnt to never come up again", "cluster node crashed for 2 hours
but got back up again", "some other cluster node waited for least and acquired
the same cluster id as the one which got crashed", etc. The more we go ahead
with solving this - it seems we'd start to invade into what sling jobs do today
(have concrete notion of assignee etc with re-assignments as required)
was (Author: catholicon):
Btw, just to note (I'm unable to concretely argue if we really want to do this
or not) - no matter how we proceed, the storage would be cluster-id specific.
So, do we want to cater the case where some cluster node gets obliterated (with
pending stored events) - would/should some other node pick that up? Adding the
disclaimer again that it's a case that would hit us once we start to advertise
"stronger guarantee delivery of observation events" - not that we must fix it
(we can as document it that the guarantee is a best-effort thing)
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
>
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522942#comment-15522942
]
Vikas Saurabh edited comment on OAK-4581 at 9/26/16 12:43 PM:
--
bq. The advantage of persisting events would have been that it wouldn't have
needed such a cut-off..
Umm... do you mean that we'd be ok with infinite storage with persisted events?
Or, that it de-couples storage concern with GC and hence the cut-off metric
could be something else (say, storage size v/s must-have time boundaries if we
remain coupled)
was (Author: catholicon):
bq. The advantage of persisting events would have been that it wouldn't have
needed such a cut-off..
Umm... do you mean that we'd be ok with infinite storage with persisted events?
Or, that it de-couples storage concern with GC and hence the cut-off metric
could be storage size (as against must-have time boundaries if we remain
coupled)
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3]
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459618#comment-15459618
]
Michael Dürig edited comment on OAK-4581 at 9/2/16 9:18 PM:
bq. One approach that we currently target is to checkpoint the oldest entry,
such that we prevent gc from removing it (assuming checkpoints are respected).
No it doesn't on {{oak-segment-tar}]. We moved from reachability to retention
time for gc as the former didn't work.
was (Author: mduerig):
bq. One approach that we currently target is to checkpoint the oldest entry,
such that we prevent gc from removing it (assuming checkpoints are respected).
No it doesn't on {{oak-segment-tar}. We moved from reachability to retention
time for gc as the former didn't work.
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3]
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation
[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459618#comment-15459618
]
Michael Dürig edited comment on OAK-4581 at 9/2/16 9:19 PM:
bq. One approach that we currently target is to checkpoint the oldest entry,
such that we prevent gc from removing it (assuming checkpoints are respected).
No it doesn't on {{oak-segment-tar}}. We moved from reachability to retention
time for gc as the former didn't work.
was (Author: mduerig):
bq. One approach that we currently target is to checkpoint the oldest entry,
such that we prevent gc from removing it (assuming checkpoints are respected).
No it doesn't on {{oak-segment-tar}]. We moved from reachability to retention
time for gc as the former didn't work.
> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
> Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple
> drawbacks. Quite a bit of work is done to make diff generation faster.
> However there are still chances of event queue getting filled up.
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous
> observer
> # Observors which are meant to handle such events in async way (by virtue of
> being wrapped in BackgroundObserver) would instead pull the events from this
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form.
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of
> SegmentNodeStore root state looks like{color} - Possible the RecordId of
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote
> journal to determine the affected paths. Which reduces the need for
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains
> information about affected path. Similar to what is current being stored in
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching.
> [~mreutegg] suggested that for persisted local journal we can also utilize a
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis.
> This would allow each Listener to "pull" content changes and generate diff as
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2]
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3]
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
