On Tue, Jan 12, 2010 at 5:59 PM, Reinier van den Born <[email protected]> wrote: > Hi Ard, > > Shoot, I am blind. The refresher is already dropping requests for keys that > are already in the queue. > So then I don't think there is a serious problem left :-).
Great! It just works then? I like it when people actually use the async stuff I wrote back then, it wasn't easy :-)) Thanks for letting me know it works, Cheers > > > Reinier > > > On Tue, Jan 12, 2010 at 5:50 PM, Reinier van den Born < > [email protected]> wrote: > >> Hi Ard, >> >> This is becoming interesting :-). >> >> On Tue, Jan 12, 2010 at 1:16 PM, Ard Schrijvers <[email protected] >> > wrote: >> >>> On Tue, Jan 12, 2010 at 11:40 AM, Reinier van den Born >>> <[email protected]> wrote: >>> > Hello Ard, >>> > >>> > I followed your proposal to take the outside (http:) route. >>> > So I have two pipeline matchers now: "lazysource" and >>> "lazysource_direct", >>> > where "lazysource" generates "async:http://host/lazysource_direct"; and >>> the >>> > latter does the original work. >>> > >>> > This seems to work, ie. I get cached results in return until it expires, >>> > while the _direct returns an updated result instantly. >>> > The only odd thing is that when I log the matchers, "lazysource_direct" >>> > appears to be invoked every time lazysource is. >>> >>> this is to compute the cachekey. The cached result should be returned >>> still. So, that the call is done is correct >>> >> >> Still a bit confusing (to me) that the logging action is also called when >> only the key is generated. >> I would expect it to be only called when entering the pipeline to >> generate(), >> Maybe I should configure the actions differently (I am not very experienced >> with Cocoon). >> >> >> > So it seems the cached result is used, but still gets the result to be >> > generated. Sounds odd, because that would defy the purpose of the whole >> >> No, the cachekey is generated, see >>> http://cocoon.apache.org/2.2/core-modules/core/2.2/690_1_1.html >> >> >> Interesting, but goes a bit too deep to fully understand with my limited >> knowledge :-) >> The page they claim to make more intelligible is actually easier to >> follow...but is probably outdated. >> But does this one describe really what is happening for an async-ed Source? >> >> Still not really sure why cocoon would need to generate a cachekey for >> "lazysource_direct" as long as "lazysource" is in cache and valid. >> I can see it being necessary for normal caching go down a full cache tree. >> But for an expiring, time-triggered cache that doesn't seem necessary, or? >> Unless it is checking for existence. According to the page you refer to >> that might be it... >> >> > exercise. >>> > >>> > I tried to put lazysource in a "caching", as opposed to "ecaching", >>> pipeline >>> > but that doesn't seem to make a difference. >>> > >>> > Made me wonder further, whether you have foreseen a mechanism that keeps >>> > simultaneous requests from being able to kick off parallel requests for >>> > "_direct" once it is outdated. >>> >>> you can try to create your own generator (configured in the >>> cocoon.xconf to have a pool limit size of 1) and have in here a >>> synchronized method, which blocks other requests >>> >> >> Isn't having pool-size=1 and using synchronized, somehow doing things >> double? >> >> Also it seems to me this would serialize all generate/refresh requests >> (yes, this would prevent us from running out of memory) >> but not suppress them. See below. >> >> >> > Because that is what my original problem was. >> > >> > I made an attempt to locate the source code to take a peek myself. Only >> > found something doing with async in the repository block of cocoon itself >> > (CachingSource.java). >> > Is that where I should be looking? >> >> you should take a look at the hippo cachingsource block, but be >>> warned, Cocoon's caching is a very complex thing. >> >> >> Just looking around the code I ran into refresher, the one that controls >> the actual generating work on asynced stuff. >> What if it were to be modified to drop refresh() requests that are already >> in the queue (match on cacheKey)? >> Since the refresher starts processing no more than one request per second >> (which in cases may be limiting, might want to make a parameter of that?) >> and is threadCount (which is a parameter) limited, unnecessary generation >> will not be completely avoided, but they will be kept under tight control. >> Quick upper limit estimate: number of threads+1 or so?? >> >> Doesn't seem like a risky or complicated modification, but my view of the >> world may be to simplified :-) >> What do you think? >> >> Groeten, >> >> Reinier >> >> >> >>> >>> Regards Ard >>> >>> > >>> > Thanks, >>> > >>> > Reinier >>> > >>> > Reinier van den Born >>> > HintTech B.V. >>> > The Netherlands >>> > >>> > T: +31(0)88 268 25 00 >>> > F: +31(0)88 268 25 01 >>> > M: +31(0)6 494 171 36 >>> > HintTech is a specialist in eBusiness Technology ( .Net, Java platform, >>> > Tridion ) and IT-Projects. >>> > Chamber of Commerce The Hague nr. 27242282 | Sales Tax nr. >>> NL8062.16.396.B01 >>> > >>> > >>> > >>> > >>> > >>> > On Fri, Jan 8, 2010 at 2:45 PM, Ard Schrijvers < >>> [email protected]>wrote: >>> > >>> >> Hello Reinier, >>> >> >>> >> On Fri, Jan 8, 2010 at 2:37 PM, Reinier van den Born >>> >> <[email protected]> wrote: >>> >> > Hi Ard, >>> >> > >>> >> > I have followed the instructions, but it doesn't seem to work. At >>> least >>> >> not >>> >> > for a cocoon: resource. >>> >> > >>> >> > I created a separate pipeline (original was calling a resource, so >>> that >>> >> was >>> >> > easier). >>> >> > >>> >> > Without async: it works like before, but with async the cache seems >>> not >>> >> to >>> >> > expire at all. >>> >> > I tried both with and without a cache-expires argument. >>> >> > This is what I am using: >>> >> > <map:generate src="async:cocoon://lazysource"/> >>> >> > >>> >> > I also tried >>> >> > <map:generate >>> src="async:cocoon://lazysource?cocoon:cache-expires=10"/> >>> >> > >>> >> > When I switch back and forth between with async: and without async: I >>> get >>> >> > the first cached version and the latest version respectively. >>> >> > >>> >> >>> just found out this is not entirely correct: after a real long >>> while - >>> >> at >>> >> > least 20 minutes but more like an hour - the cache is being updated >>> << >>> >> > >>> >> > However when I replace the cocoon: url with a http: one things start >>> to >>> >> work >>> >> > like you described. >>> >> > So for instance >>> >> > <map:generate src="async: >>> >> > http://www.anwb.nl/verkeer/verkeersinformatie_files_nl"/> >>> >> > is updated properly (with the delay). >>> >> > (except that the first request will show the old cached version, like >>> we >>> >> > discussed below, showing it is actually active :-) >>> >> > >>> >> > I hope this is just me overlooking something, because it looks very >>> >> > promosing. >>> >> > Any ideas? >>> >> >>> >> It might have been broken for the cocoon:// protocol. I vaguely >>> >> remember that it was extremely hard to accomplish for the cocoon:// >>> >> protocol. >>> >> >>> >> But can't you just do an http call to the cocoon instance? Thus, >>> instead >>> >> of: >>> >> >>> >> <map:generate src="async:cocoon://lazysource?cocoon:cache-expires=10"/> >>> >> >>> >> use >>> >> >>> >> <map:generate src="async: >>> >> http://www.mydomain.com/lazysource?cocoon:cache-expires=10"/> >>> >> >>> >> Regards Ard >>> >> >>> >> > >>> >> > Reinier >>> >> > >>> >> > >>> >> > >>> >> > On Thu, Jan 7, 2010 at 5:36 PM, Ard Schrijvers < >>> >> [email protected]>wrote: >>> >> > >>> >> >> hello, >>> >> >> >>> >> >> On Thu, Jan 7, 2010 at 5:17 PM, Reinier van den Born >>> >> >> <[email protected]> wrote: >>> >> >> > Hi Ard, >>> >> >> > >>> >> >> > Thanks for your reply. >>> >> >> > >>> >> >> > Your suggestion looks quite like what we need, but I am not sure >>> >> whether >>> >> >> it >>> >> >> > will really work for us. >>> >> >> > >>> >> >> > You are talking about an external resource where the (async) >>> >> regeneration >>> >> >> is >>> >> >> > simply triggered when the cache has expired. >>> >> >> >>> >> >> it works also for internal >>> >> >> >>> >> >> > In our case we are dealing with a repository resource, where cache >>> >> >> > invalidations are triggered by resource changes. >>> >> >> > So normally update events are invalidating all caches upto our >>> final >>> >> >> page. >>> >> >> > If this continues to happen the cache >>> >> >> > will be invalidated in between expires, so the problem would >>> remain. >>> >> >> >>> >> >> I think I addressed this, where the old response is still being >>> served. >>> >> >> >>> >> >> > Or will the cache-expires option protect the "slowpart" cache from >>> >> those >>> >> >> > cache invalidations? >>> >> >> >>> >> >> think so, but it was long time ago i built it >>> >> >> >>> >> >> > >>> >> >> > If not, maybe it is possible to set up an intermediate pipeline to >>> >> >> provide >>> >> >> > the necessary isolation? >>> >> >> >>> >> >> I would just test whether it does what you expect from it...I know >>> the >>> >> >> Cocoon caching is quite complex, and this part was much complexer >>> then >>> >> >> standard Cocoon caching >>> >> >> >>> >> >> > >>> >> >> > Some more general questions: >>> >> >> > >>> >> >> > I've been looking around a bit further and ran into the following >>> >> page: >>> >> >> > >>> >> >> >>> >> >>> http://wiki.onehippo.com/display/CMS/Using+asynchronic+get+for+cached+content >>> >> >> > If I am not mistaken this is using the async option on Cocoon's >>> >> >> > CachingSourceFactory. >>> >> >> > Is this related to what you mention or are these entirely >>> different >>> >> >> things? >>> >> >> >>> >> >> Heey, I wrote that wiki page, great! This is exactly what I meant. I >>> >> >> only forgot I added a seperate source-factory async for it...whow, I >>> >> >> forgot I did all that back then :-)) >>> >> >> >>> >> >> You should follow that page! >>> >> >> >>> >> >> >> >>> >> >> > Btw, if I understand the async option correctly, the regeneration >>> is >>> >> >> still >>> >> >> > initiated by an incoming request. >>> >> >> >>> >> >> yes... (you can have some cron script calling it if you want?) >>> >> >> >>> >> >> > Does that mean that if a request arrives first after an expire it >>> >> still >>> >> >> gets >>> >> >> > the old content served? >>> >> >> >>> >> >> Exactly.... >>> >> >> >>> >> >> > In other words, when page traffic is sufficiently low, the site >>> will >>> >> >> always >>> >> >> > produce expired content? >>> >> >> >>> >> >> Tja, that's what you get with a async fetch...iirc, I did not add a >>> >> >> cron job checking the cache for async expired entries and refetch >>> >> >> them... >>> >> >> >>> >> >> > >>> >> >> > This shouldn't be a problem for the case we're experiencing our >>> >> problems, >>> >> >> > but it would be something to take into consideration for different >>> >> cases. >>> >> >> >>> >> >> Well....yes, I see your point, but it was quite hard already :-)) >>> >> >> >>> >> >> Just try it I think is the best! >>> >> >> >>> >> >> Regards Ard >>> >> >> >>> >> >> > >>> >> >> > Regards, >>> >> >> > >>> >> >> > Reinier >>> >> >> > >>> >> >> > Reinier van den Born >>> >> >> > HintTech B.V. >>> >> >> > >>> >> >> > T: +31(0)88 268 25 00 >>> >> >> > F: +31(0)88 268 25 01 >>> >> >> > M: +31(0)6 494 171 36 >>> >> >> > >>> >> >> > Delftechpark 37i | 2628 XJ Delft | The Netherlands >>> >> >> > www.hinttech.com<javascript:void('http://www.hinttech.com');> >>> >> >> > >>> >> >> > HintTech is a specialist in eBusiness Technology ( .Net, Java >>> >> platform, >>> >> >> > Tridion ) and IT-Projects. >>> >> >> > Chamber of Commerce The Hague nr. 27242282 | Sales Tax nr. >>> >> >> NL8062.16.396.B01 >>> >> >> > >>> >> >> > >>> >> >>> >> Deleted tail of the conversation here. >>> > ******************************************** >>> > Hippocms-dev: Hippo CMS development public mailinglist >>> > >>> > Searchable archives can be found at: >>> > MarkMail: http://hippocms-dev.markmail.org >>> > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >>> > >>> > >>> ******************************************** >>> Hippocms-dev: Hippo CMS development public mailinglist >>> >>> Searchable archives can be found at: >>> MarkMail: http://hippocms-dev.markmail.org >>> Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >>> >>> >> > ******************************************** > Hippocms-dev: Hippo CMS development public mailinglist > > Searchable archives can be found at: > MarkMail: http://hippocms-dev.markmail.org > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > ******************************************** Hippocms-dev: Hippo CMS development public mailinglist Searchable archives can be found at: MarkMail: http://hippocms-dev.markmail.org Nabble: http://www.nabble.com/Hippo-CMS-f26633.html
