Re: Oak JCR Observation scalability aspects and concerns

2013-10-30 Thread Alexander Klimetschek
I created OAK-1133 for the "Observation listener PLUS" as I proposed it, to separate it from the Sling-specific case. https://issues.apache.org/jira/browse/OAK-1133 Cheers, Alex On 27.10.2013, at 23:57, Thomas Mueller wrote: > Hi, > >> I've created OAK-1120 to start with the simple case., >

Re: Oak JCR Observation scalability aspects and concerns

2013-10-27 Thread Thomas Mueller
Hi, > I've created OAK-1120 to start with the simple case., OAK-1120 doesn't address scalability aspects. Regards, Thomas On 10/27/13 10:44 AM, "Carsten Ziegeler" wrote: >I've created OAK-1120 to start with the simple case., > >Thanks >Carsten > > >2013/10/25 Alexander Klimetschek > >> On 25

Re: Oak JCR Observation scalability aspects and concerns

2013-10-27 Thread Carsten Ziegeler
I've created OAK-1120 to start with the simple case., Thanks Carsten 2013/10/25 Alexander Klimetschek > On 25.10.2013, at 14:11, Alexander Klimetschek wrote: > > Maybe it would be useful to additionally allow a generic "matches" > function that can be passed upon listener registration that co

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Alexander Klimetschek
On 25.10.2013, at 14:11, Alexander Klimetschek wrote: > Maybe it would be useful to additionally allow a generic "matches" function > that can be passed upon listener registration that could check whatever it > wants, working on the diff or change set directly. Actually it needs to be able to w

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Alexander Klimetschek
Hi, IMHO it would be nice to support more matching functionality for events down in Oak. See my mail on the Sling thread [0]. The general advantage would be that for non-matches there would be no event created at all, no thread started and what else is involved. I assume that could happen most

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Angela Schreiber
hi bertrand +1 for everything you said. in general i think that we have split that section in the docu: a) summary from OAK point of view (for those that know the very details) b) summary from a JCR/Jackrabbit point of view. IMHO we can't expect our user community to understand how these subt

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Bertrand Delacretaz
Hi Angela, On Fri, Oct 25, 2013 at 11:21 AM, Angela Schreiber wrote: > Bertrand wrote: >>The OSGi events that Sling rebroadcasts are less granular than JCR >>events, so this might not be a problem for that case. > > do you know it or are you guessing?... I'm making an educated guess. To get har

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Angela Schreiber
hi bertrand >On Fri, Oct 25, 2013 at 8:53 AM, Angela Schreiber >wrote: >> ...the biggest challenge i see in terms of backwards compatibility is >> that the diff-mechanism in OAK doesn't allow a 1:1 translation >> to JCR events as they used to be generated in jackrabbit-core... > >The OSGi events

Re: Oak JCR Observation scalability aspects and concerns

2013-10-25 Thread Bertrand Delacretaz
Hi, On Fri, Oct 25, 2013 at 8:53 AM, Angela Schreiber wrote: > ...the biggest challenge i see in terms of backwards compatibility is > that the diff-mechanism in OAK doesn't allow a 1:1 translation > to JCR events as they used to be generated in jackrabbit-core... The OSGi events that Sling rebr

Re: Oak JCR Observation scalability aspects and concerns

2013-10-24 Thread Angela Schreiber
hi carsten IMO that should be feasible as the oak editor/commithook interfaces (and the diff they may make use of) will provide you with the required information (e.g. childNodeChanged takes both the nodestate before and after the modification as params). the biggest challenge i see in terms of b

Re: Oak JCR Observation scalability aspects and concerns

2013-10-24 Thread Carsten Ziegeler
Rethinking this, I think it's clear that the current way the jcr listener works in Sling is not optimal as it reads each and every changed node. So I think it really would be great if we could directly get the information from Oak without the need to additionally process. The listener currently: 1

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Dominik Süß
I just opened a thread at sling-dev for further discussion about api and implementation changes on sling side [0] For discussions around usage of this api within sling please use this linked thread [0]. Best regards Dominik [0] markmail.org/thread/plb7ledhsna33r3g On Tue, Oct 22, 2013 at 4:54

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Jukka Zitting
Hi, On Tue, Oct 22, 2013 at 10:39 AM, Carsten Ziegeler wrote: > Just to reiterate :) if we go with 3 or 4, someone has to do the work in > Sling (and other places) and adapt the code. As obviously as soon as a > single listener is using the old pattern, the whole mechanism is mood. I think we ca

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Felix Meschberger
Again: can we please have sling debates on dev@sling ? Thanks Felix Von meinem iPad gesendet > Am 22.10.2013 um 16:39 schrieb "Carsten Ziegeler" : > > Just to reiterate :) if we go with 3 or 4, someone has to do the work in > Sling (and other places) and adapt the code. As obviously as soon as

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Thomas Mueller
Hi, >I really would like to have a constructive discussion here. Sure. If we want a fully scalable solution, Sling needs to be changed. > 1. we leave everything as is > 2. we maintain the API as-is in Sling and try to make the implementation >as fast as possible This will limit scalability so I

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Carsten Ziegeler
Just to reiterate :) if we go with 3 or 4, someone has to do the work in Sling (and other places) and adapt the code. As obviously as soon as a single listener is using the old pattern, the whole mechanism is mood. Carsten 2013/10/22 Dominik Süß > +1 on 4 since I fear 3 will create some overhe

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Dominik Süß
+1 on 4 since I fear 3 will create some overhead for existing solutions that won't need this kind of scalabilty (and therefore create uncessary efforts for migration). This is the old "compat" pattern seen so often. IMHO this should be an extension that "can" be installed but is not available by

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Bertrand Delacretaz
On Tue, Oct 22, 2013 at 4:30 PM, Carsten Ziegeler wrote: > ...4. same as 3. but keep the old Sling API with a bold marker when it's used > that this does not scale... That's my favorite choice - along with providing a way for users of those Sling events to specify more precisely what they actuall

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Dominik Süß
Hi :) Speaking as developer using the Sling eventing I just wanted to add that in most cases there are restrictions on Paths (most times not just one but multiple "searchpaths") and on a resourceType (not just exact match but a set or pattern to identify a set of resourceTypes) and in some occasio

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Carsten Ziegeler
I really would like to have a constructive discussion here. I think the Sling use case is pretty well explained now - that's an api Sling offers and which is used by a lot of code out there (a great part of Sling is based on the OSGi events and layers on top of Sling are using it as well). That's a

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Bertrand Delacretaz
On Tue, Oct 22, 2013 at 4:19 PM, Thomas Mueller wrote: >>This is the contract we have to maintain in Sling. > Well, we can't maintain this contract, because it blocks scalability I tend to agree, and OTOH not everybody will need the kind of scalability that we're discussing here. Many of the

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Jukka Zitting
Hi, On Tue, Oct 22, 2013 at 9:59 AM, Felix Meschberger wrote: > That's one Event object per event -- not one event per listener per event. > This is completely different to JCR. You're mistaking the problem here, it's not the number of listeners, it's the number of events *per listener*. What w

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Thomas Mueller
Hi, >This is the contract we have to maintain in Sling. Well, we can't maintain this contract, because it blocks scalability. Regards, Thomas

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Thomas Mueller
Hi, >That's one Event object per event -- not one event per listener per >event. This is completely different to JCR. Well, this still doesn't scale, if every cluster node needs all events. It doesn't matter how many observation listeners you have. To block scalability, it's enough to have one o

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Jukka Zitting
Hi, On Tue, Oct 22, 2013 at 9:43 AM, Carsten Ziegeler wrote: > This is the contract we have to maintain in Sling. I repeat from my earlier post: > Right, it just means that a deployment with such an observer will have > a built-in scalability limit as at some point the listener will no > longer

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Felix Meschberger
Hi Am 22.10.2013 um 15:52 schrieb Jukka Zitting: > Hi, > > On Tue, Oct 22, 2013 at 9:41 AM, Felix Meschberger wrote: >> The JcrResourceListener just gets JCR Observation events, creates the OSGi >> Event objects >> and hands them over for distribution by the OSGi EventAdmin service. The >> la

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Jukka Zitting
Hi, On Tue, Oct 22, 2013 at 9:41 AM, Felix Meschberger wrote: > The JcrResourceListener just gets JCR Observation events, creates the OSGi > Event objects > and hands them over for distribution by the OSGi EventAdmin service. The > latter service is > then responsible for dispatching taking the

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Carsten Ziegeler
Sling's jcr listener provides an API/contract - so this is an infrastructure component used by application code (or other infrastructure code). And as the listener is delegating the promotion of events to the EventAdmin, the jcr listener does not know if there are event listeners at all or what kin

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Felix Meschberger
Hi Am 22.10.2013 um 15:27 schrieb Jukka Zitting: > Hi, > > On Tue, Oct 22, 2013 at 5:21 AM, Felix Meschberger wrote: >> Am 22.10.2013 um 11:17 schrieb Chetan Mehrotra: >>> I think in Sling case it would make sense for it to be implemented as >>> an Observer. And I had a look at implementation o

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Jukka Zitting
Hi, On Tue, Oct 22, 2013 at 5:21 AM, Felix Meschberger wrote: > Am 22.10.2013 um 11:17 schrieb Chetan Mehrotra: >> I think in Sling case it would make sense for it to be implemented as >> an Observer. And I had a look at implementation of some of the >> listener implementations of [1] and I think

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Angela Schreiber
hi felix from what i see Sling heavily relies on jackrabbit-core functionality... i would be very pleased if it would just rely on public API such as JCR, Jackrabbit API and in the future OAK; it doesn't and this is causing a lot of troubles. kind regards angela On 10/22/13 11:21 AM, "Felix Mesc

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Felix Meschberger
Hi Am 22.10.2013 um 11:17 schrieb Chetan Mehrotra: > On Mon, Oct 21, 2013 at 11:39 PM, Jukka Zitting > wrote: >> 3) The Observer mechanism allows a listener to look at repository >> changes in variable granularity and frequency depending on application >> needs and current repository load. Thus

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Chetan Mehrotra
On Mon, Oct 21, 2013 at 11:39 PM, Jukka Zitting wrote: > 3) The Observer mechanism allows a listener to look at repository > changes in variable granularity and frequency depending on application > needs and current repository load. Thus an Oak Observer can > potentially process orders of magnitud

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Chetan Mehrotra
On Mon, Oct 21, 2013 at 6:47 PM, Jukka Zitting wrote: > -1 This introduces the problem where a single JCR event listener can > block or slow down all other listeners. That can be mitigated upto an extent by using some sort of Black List (OAK-1084). However current approach of each listener pullin

Re: Oak JCR Observation scalability aspects and concerns

2013-10-22 Thread Thomas Mueller
Hi, I think it's quite clear that "global" observation listeners are not scalable. (Observation listeners that listen for all events below root, from all cluster nodes.) It is needed for backward compatibility, but we need to find a solution to make this obsolete. It's not enough to just reduce th

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Angela Schreiber
hi just one more comment regarding caches from a security perspective >> However, there are many use cases where a local cache is used on >> each instance which would make such an approach useless. > >Oak has a variety of solutions that could be used to make such code >more scalable (ordered by i

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Jukka Zitting
Hi, On Mon, Oct 21, 2013 at 12:03 PM, Carsten Ziegeler wrote: > This Sling listener is providing higher level application support as it > creates resource events out of jcr observation events. A lot of code in > Sling now relies on this functionality and even more higher level code > based on Sli

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Carsten Ziegeler
This Sling listener is providing higher level application support as it creates resource events out of jcr observation events. A lot of code in Sling now relies on this functionality and even more higher level code based on Sling does so. Usually application listeners register for "/" and then filt

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Jukka Zitting
Hi, On Mon, Oct 21, 2013 at 9:47 AM, Bertrand Delacretaz wrote: > On Mon, Oct 21, 2013 at 3:17 PM, Jukka Zitting > wrote: >> ...Instead of an repository problem (like diffing, event creation, etc.), >> this analysis tells me that the bottleneck here is the application >> that tries to listen to

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Bertrand Delacretaz
On Mon, Oct 21, 2013 at 3:17 PM, Jukka Zitting wrote: > ...Instead of an repository problem (like diffing, event creation, etc.), > this analysis tells me that the bottleneck here is the application > that tries to listen to so many events... FWIW, by default Sling does have a listener that catch

Re: Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Jukka Zitting
Hi, On Mon, Oct 21, 2013 at 6:38 AM, Chetan Mehrotra wrote: > Marcel - Basic concern raised is that listeners without any filter > would cause lots of reads on the repository. these kind of listeners > would pull in modifications of all sessions performing distributed > writes. In our view this w

Oak JCR Observation scalability aspects and concerns

2013-10-21 Thread Chetan Mehrotra
Below are the details about discussion held [0] so far on factors which possibly hinder scaling JCR Observation. It also touches upon area where usage pattern of applications running on JCR (like Sling) effects performance Concerns - Marcel - Basic concern raised is that listeners wit