Re: [Observation] Question on local events reported as external
On Fri, Jun 26, 2015 at 1:49 AM, Marius Petria wrote: > The observation queue limit is per EventListener, meaning that I have to > optimize only my listener in order to make it less likely for the limit to be > hit. Yes > At least in theory (and with the exception of collapsing a creation and a > deletion) the events are never dropped, right? The line in [1] even if it > skips the current NodeState it will include it in a later revision, right? Yes Chetan Mehrotra
Re: [Observation] Question on local events reported as external
Thanks Chetan for the detailed explanation. I have a couple more questions :). 1. The observation queue limit is per EventListener, meaning that I have to optimize only my listener in order to make it less likely for the limit to be hit. 2. At least in theory (and with the exception of collapsing a creation and a deletion) the events are never dropped, right? The line in [1] even if it skips the current NodeState it will include it in a later revision, right? Marius [1] https://github.com/apache/jackrabbit-oak/blob/80985b51a18ef61015d263d2358f5c3713fea873/oak-core/src/main/java/org/apache/jackrabbit/oak/spi/commit/BackgroundObserver.java#L278 On 6/25/15, 9:49 PM, "Chetan Mehrotra" wrote: >On Thu, Jun 25, 2015 at 10:49 PM, Marius Petria wrote: >> AFAIU because the local events are changed to external events it means that >> they can also be dropped completly under load, is that true? > >Well they are not dropped. Observation in Oak works on basis of diff >of NodeState. So lets say you have an Observation queue of size 3 with >3 local events > >1. [ns1, ns2, ci1] >2. [ns2, ns3, ci2] >3. [ns3, ns4, ci3] > >Each tuple is [base root nodestate, nodeState post change, commit >info]. Now if 4 change event [ns4, ns5, ci4] comes the >BackgroundObserver has following options > >1. Block on put >2. Have a indefinite length queue then just add to queue >3. Pull out the last event and replace it with merged > >So current logic in Oak goes for #3. > >1. [ns1, ns2, ci1] >2. [ns2, ns3, ci2] >3. [ns3, ns5, null] > >Now when it merged/collapsed the content change then you cannot >associate any specific commit info to it. In such cases > >1. You cannot determine which user has made that change. >2. Some changes might not get visible. For example if a foo property >is added in E3 and removed in E4. Then Merged content change would not >provide any indication that any such change happened > >So such merged changes are shown as external i.e. >JackrabbitEvent#isExternal returns true for them currently. Problem is >currently its not possible to distinguish such collapsed events from >truely external events. May be we make that distinction so that >component which just rely on *some local change* to react can continue >to work. Though there is no gurantee that they see *each* local >change. > >Chetan Mehrotra
Re: [Observation] Question on local events reported as external
On Thu, Jun 25, 2015 at 10:49 PM, Marius Petria wrote: > AFAIU because the local events are changed to external events it means that > they can also be dropped completly under load, is that true? Well they are not dropped. Observation in Oak works on basis of diff of NodeState. So lets say you have an Observation queue of size 3 with 3 local events 1. [ns1, ns2, ci1] 2. [ns2, ns3, ci2] 3. [ns3, ns4, ci3] Each tuple is [base root nodestate, nodeState post change, commit info]. Now if 4 change event [ns4, ns5, ci4] comes the BackgroundObserver has following options 1. Block on put 2. Have a indefinite length queue then just add to queue 3. Pull out the last event and replace it with merged So current logic in Oak goes for #3. 1. [ns1, ns2, ci1] 2. [ns2, ns3, ci2] 3. [ns3, ns5, null] Now when it merged/collapsed the content change then you cannot associate any specific commit info to it. In such cases 1. You cannot determine which user has made that change. 2. Some changes might not get visible. For example if a foo property is added in E3 and removed in E4. Then Merged content change would not provide any indication that any such change happened So such merged changes are shown as external i.e. JackrabbitEvent#isExternal returns true for them currently. Problem is currently its not possible to distinguish such collapsed events from truely external events. May be we make that distinction so that component which just rely on *some local change* to react can continue to work. Though there is no gurantee that they see *each* local change. Chetan Mehrotra
Re: [Observation] Question on local events reported as external
AFAIU because the local events are changed to external events it means that they can also be dropped completly under load, is that true? On 6/25/15, 6:57 PM, "Chetan Mehrotra" wrote: >This is most likely due to rate at which event gets generated and thus >causing Observation queue to fill up (1000 default size). If the >backing JCR listener is slow in processing the queue would get filled >and BackgroundObserver would start compacting/merging the diff thus >converting local events to external. > >There are two solutions >1. Throttle the commits - CommitRateLimiter >2. Have a non limiting queue - Then you end up with OOM if gap in >processing rate is large >Chetan Mehrotra > > >On Thu, Jun 25, 2015 at 8:47 PM, Marius Petria wrote: >> Hi, >> >> I understand that under high load local events can be reported as external. >> My question is why does this happen also for tarmk, where there is a single >> instance active (receiving external events on a single instance seems odd)? >> Also, is there a way to disable this functionality, meaning on tarmk to >> always receive the local events with their associated data (specifically the >> userData)? >> >> Regards, >> Marius >>
Re: [Observation] Question on local events reported as external
This is most likely due to rate at which event gets generated and thus causing Observation queue to fill up (1000 default size). If the backing JCR listener is slow in processing the queue would get filled and BackgroundObserver would start compacting/merging the diff thus converting local events to external. There are two solutions 1. Throttle the commits - CommitRateLimiter 2. Have a non limiting queue - Then you end up with OOM if gap in processing rate is large Chetan Mehrotra On Thu, Jun 25, 2015 at 8:47 PM, Marius Petria wrote: > Hi, > > I understand that under high load local events can be reported as external. > My question is why does this happen also for tarmk, where there is a single > instance active (receiving external events on a single instance seems odd)? > Also, is there a way to disable this functionality, meaning on tarmk to > always receive the local events with their associated data (specifically the > userData)? > > Regards, > Marius >
[Observation] Question on local events reported as external
Hi, I understand that under high load local events can be reported as external. My question is why does this happen also for tarmk, where there is a single instance active (receiving external events on a single instance seems odd)? Also, is there a way to disable this functionality, meaning on tarmk to always receive the local events with their associated data (specifically the userData)? Regards, Marius