Hello Reinier, On Fri, Jan 8, 2010 at 2:37 PM, Reinier van den Born <[email protected]> wrote: > Hi Ard, > > I have followed the instructions, but it doesn't seem to work. At least not > for a cocoon: resource. > > I created a separate pipeline (original was calling a resource, so that was > easier). > > Without async: it works like before, but with async the cache seems not to > expire at all. > I tried both with and without a cache-expires argument. > This is what I am using: > <map:generate src="async:cocoon://lazysource"/> > > I also tried > <map:generate src="async:cocoon://lazysource?cocoon:cache-expires=10"/> > > When I switch back and forth between with async: and without async: I get > the first cached version and the latest version respectively. > >>> just found out this is not entirely correct: after a real long while - at > least 20 minutes but more like an hour - the cache is being updated << > > However when I replace the cocoon: url with a http: one things start to work > like you described. > So for instance > <map:generate src="async: > http://www.anwb.nl/verkeer/verkeersinformatie_files_nl"/> > is updated properly (with the delay). > (except that the first request will show the old cached version, like we > discussed below, showing it is actually active :-) > > I hope this is just me overlooking something, because it looks very > promosing. > Any ideas?
It might have been broken for the cocoon:// protocol. I vaguely remember that it was extremely hard to accomplish for the cocoon:// protocol. But can't you just do an http call to the cocoon instance? Thus, instead of: <map:generate src="async:cocoon://lazysource?cocoon:cache-expires=10"/> use <map:generate src="async:http://www.mydomain.com/lazysource?cocoon:cache-expires=10"/> Regards Ard > > Reinier > > > > On Thu, Jan 7, 2010 at 5:36 PM, Ard Schrijvers > <[email protected]>wrote: > >> hello, >> >> On Thu, Jan 7, 2010 at 5:17 PM, Reinier van den Born >> <[email protected]> wrote: >> > Hi Ard, >> > >> > Thanks for your reply. >> > >> > Your suggestion looks quite like what we need, but I am not sure whether >> it >> > will really work for us. >> > >> > You are talking about an external resource where the (async) regeneration >> is >> > simply triggered when the cache has expired. >> >> it works also for internal >> >> > In our case we are dealing with a repository resource, where cache >> > invalidations are triggered by resource changes. >> > So normally update events are invalidating all caches upto our final >> page. >> > If this continues to happen the cache >> > will be invalidated in between expires, so the problem would remain. >> >> I think I addressed this, where the old response is still being served. >> >> > Or will the cache-expires option protect the "slowpart" cache from those >> > cache invalidations? >> >> think so, but it was long time ago i built it >> >> > >> > If not, maybe it is possible to set up an intermediate pipeline to >> provide >> > the necessary isolation? >> >> I would just test whether it does what you expect from it...I know the >> Cocoon caching is quite complex, and this part was much complexer then >> standard Cocoon caching >> >> > >> > Some more general questions: >> > >> > I've been looking around a bit further and ran into the following page: >> > >> http://wiki.onehippo.com/display/CMS/Using+asynchronic+get+for+cached+content >> > If I am not mistaken this is using the async option on Cocoon's >> > CachingSourceFactory. >> > Is this related to what you mention or are these entirely different >> things? >> >> Heey, I wrote that wiki page, great! This is exactly what I meant. I >> only forgot I added a seperate source-factory async for it...whow, I >> forgot I did all that back then :-)) >> >> You should follow that page! >> >> >> >> > Btw, if I understand the async option correctly, the regeneration is >> still >> > initiated by an incoming request. >> >> yes... (you can have some cron script calling it if you want?) >> >> > Does that mean that if a request arrives first after an expire it still >> gets >> > the old content served? >> >> Exactly.... >> >> > In other words, when page traffic is sufficiently low, the site will >> always >> > produce expired content? >> >> Tja, that's what you get with a async fetch...iirc, I did not add a >> cron job checking the cache for async expired entries and refetch >> them... >> >> > >> > This shouldn't be a problem for the case we're experiencing our problems, >> > but it would be something to take into consideration for different cases. >> >> Well....yes, I see your point, but it was quite hard already :-)) >> >> Just try it I think is the best! >> >> Regards Ard >> >> > >> > Regards, >> > >> > Reinier >> > >> > Reinier van den Born >> > HintTech B.V. >> > >> > T: +31(0)88 268 25 00 >> > F: +31(0)88 268 25 01 >> > M: +31(0)6 494 171 36 >> > >> > Delftechpark 37i | 2628 XJ Delft | The Netherlands >> > www.hinttech.com<javascript:void('http://www.hinttech.com');> >> > >> > HintTech is a specialist in eBusiness Technology ( .Net, Java platform, >> > Tridion ) and IT-Projects. >> > Chamber of Commerce The Hague nr. 27242282 | Sales Tax nr. >> NL8062.16.396.B01 >> > >> > >> > On Thu, Jan 7, 2010 at 11:37 AM, Ard Schrijvers >> > <[email protected]>wrote: >> > >> >> Hello, >> >> >> >> On Thu, Jan 7, 2010 at 10:51 AM, Reinier van den Born >> >> <[email protected]> wrote: >> >> > Hi all, >> >> > >> >> > We are using Hippo 6 with a Cocoon frontend. A few days ago our >> frontend >> >> > went down because it ran out of memory. >> >> > Analysis pointed in the direction of the following: >> >> > >> >> > 1. There is a page based on quite a big document (thus memory >> intensive) >> >> > that is updated every couple of minutes or so. >> >> > 2. The page gets a timeout of 30 seconds, so browsers request an >> update >> >> with >> >> > that frequency. >> >> > >> >> > The problem occurred when we had a lot of visitors. At some point the >> >> source >> >> > document was updated and flushed from the cache. >> >> > Many requests were coming in and as long as the page wasn't cached >> each >> >> > request resulted in an attempt to regenerate the page. >> >> > The result was a load of threads, each of them trying to build the >> page >> >> from >> >> > the big document and thus together eating up all memory. >> >> > >> >> > In a way it surprised us that neither Apache httpd's mod_cache or >> >> Cocoon's >> >> > caching seems to have some mechanism to prevent these redundant >> parallel >> >> > activities. >> >> >> >> I am not to surprised as it is really hard to have one general solution >> >> >> >> > >> >> > Question is whether anyone can point us to a solution? >> >> > >> >> > Our own thoughts: >> >> > >> >> > First of all we need to assure that at most one thread renders the >> page. >> >> For >> >> > the other threads two possibilities seem to be reasonable: >> >> > 1. Serve the "outdated" page as long as the updated is not ready >> >> > 2. Wait until the updated is available. >> >> > The first solution seems preferable since it prevents the possibility >> of >> >> > having a load of "dangling" requests. >> >> > >> >> > However one of the solutions we are thinking of, is to create our own >> >> Cocoon >> >> > cache class, that only allows the first thread to build a specific >> page >> >> > and, using semaphores or so, has the others waiting for the first to >> >> > finish. >> >> > Obviously it is a solution type 2 but it is attractive because it >> seems >> >> > relatively easy to implement. >> >> > We are basically hesitant to mess more than necessary with Cocoon's >> >> caching >> >> > mechanism. >> >> > >> >> > Any advice/ideas on this? >> >> >> >> I wouldn't try to solve it in cocoon's cache. Are you talking about >> >> one single part that is the same for every request, that takes a lot >> >> of time to build? >> >> >> >> I did add something (quite some time ago) like asynchronous fetching >> >> of sources, where you could configure an expires after which an >> >> asynchronous fetch was done, serving from cache until the asynchronous >> >> background thread was finished, replacing the existing cached part. >> >> Sounds like something you need isn't? It was though designed for >> >> something like fetching an external rss feed, where you don't want to >> >> wait for an external rss feed while the external site being down for >> >> example. >> >> >> >> I am not totally sure it works for non repository sources, but you >> >> might want to try this: >> >> >> >> Suppose your slow pipeline is: >> >> >> >> pattern="myslowpart" >> >> >> >> the place that calls this pipeline, should be change into: >> >> >> >> <map:generate >> >> >> src="cached:cocoon://myslowpart?cocoon:cache-expires=60&async=true"/> >> >> >> >> from the top of my head, it has to be a call to the root sitemap, this >> >> cocoon://. >> >> >> >> Another way of course to achieve something similar, is create your own >> >> generator, which stores the result on filesystem. You use this for all >> >> request, and every now and then, recompute the result and store it >> >> again >> >> >> >> Regards Ard >> >> >> >> > >> >> > Thanks, >> >> > >> >> > Reinier >> >> > ******************************************** >> >> > Hippocms-dev: Hippo CMS development public mailinglist >> >> > >> >> > Searchable archives can be found at: >> >> > MarkMail: http://hippocms-dev.markmail.org >> >> > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >> >> > >> >> > >> >> ******************************************** >> >> Hippocms-dev: Hippo CMS development public mailinglist >> >> >> >> Searchable archives can be found at: >> >> MarkMail: http://hippocms-dev.markmail.org >> >> Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >> >> >> >> >> > ******************************************** >> > Hippocms-dev: Hippo CMS development public mailinglist >> > >> > Searchable archives can be found at: >> > MarkMail: http://hippocms-dev.markmail.org >> > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >> > >> > >> ******************************************** >> Hippocms-dev: Hippo CMS development public mailinglist >> >> Searchable archives can be found at: >> MarkMail: http://hippocms-dev.markmail.org >> Nabble: http://www.nabble.com/Hippo-CMS-f26633.html >> >> > ******************************************** > Hippocms-dev: Hippo CMS development public mailinglist > > Searchable archives can be found at: > MarkMail: http://hippocms-dev.markmail.org > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > ******************************************** Hippocms-dev: Hippo CMS development public mailinglist Searchable archives can be found at: MarkMail: http://hippocms-dev.markmail.org Nabble: http://www.nabble.com/Hippo-CMS-f26633.html
