Hi Ard, I have followed the instructions, but it doesn't seem to work. At least not for a cocoon: resource.
I created a separate pipeline (original was calling a resource, so that was easier). Without async: it works like before, but with async the cache seems not to expire at all. I tried both with and without a cache-expires argument. This is what I am using: <map:generate src="async:cocoon://lazysource"/> I also tried <map:generate src="async:cocoon://lazysource?cocoon:cache-expires=10"/> When I switch back and forth between with async: and without async: I get the first cached version and the latest version respectively. >> just found out this is not entirely correct: after a real long while - at least 20 minutes but more like an hour - the cache is being updated << However when I replace the cocoon: url with a http: one things start to work like you described. So for instance <map:generate src="async: http://www.anwb.nl/verkeer/verkeersinformatie_files_nl"/> is updated properly (with the delay). (except that the first request will show the old cached version, like we discussed below, showing it is actually active :-) I hope this is just me overlooking something, because it looks very promosing. Any ideas? Reinier On Thu, Jan 7, 2010 at 5:36 PM, Ard Schrijvers <[email protected]>wrote: > hello, > > On Thu, Jan 7, 2010 at 5:17 PM, Reinier van den Born > <[email protected]> wrote: > > Hi Ard, > > > > Thanks for your reply. > > > > Your suggestion looks quite like what we need, but I am not sure whether > it > > will really work for us. > > > > You are talking about an external resource where the (async) regeneration > is > > simply triggered when the cache has expired. > > it works also for internal > > > In our case we are dealing with a repository resource, where cache > > invalidations are triggered by resource changes. > > So normally update events are invalidating all caches upto our final > page. > > If this continues to happen the cache > > will be invalidated in between expires, so the problem would remain. > > I think I addressed this, where the old response is still being served. > > > Or will the cache-expires option protect the "slowpart" cache from those > > cache invalidations? > > think so, but it was long time ago i built it > > > > > If not, maybe it is possible to set up an intermediate pipeline to > provide > > the necessary isolation? > > I would just test whether it does what you expect from it...I know the > Cocoon caching is quite complex, and this part was much complexer then > standard Cocoon caching > > > > > Some more general questions: > > > > I've been looking around a bit further and ran into the following page: > > > http://wiki.onehippo.com/display/CMS/Using+asynchronic+get+for+cached+content > > If I am not mistaken this is using the async option on Cocoon's > > CachingSourceFactory. > > Is this related to what you mention or are these entirely different > things? > > Heey, I wrote that wiki page, great! This is exactly what I meant. I > only forgot I added a seperate source-factory async for it...whow, I > forgot I did all that back then :-)) > > You should follow that page! > > >> > > Btw, if I understand the async option correctly, the regeneration is > still > > initiated by an incoming request. > > yes... (you can have some cron script calling it if you want?) > > > Does that mean that if a request arrives first after an expire it still > gets > > the old content served? > > Exactly.... > > > In other words, when page traffic is sufficiently low, the site will > always > > produce expired content? > > Tja, that's what you get with a async fetch...iirc, I did not add a > cron job checking the cache for async expired entries and refetch > them... > > > > > This shouldn't be a problem for the case we're experiencing our problems, > > but it would be something to take into consideration for different cases. > > Well....yes, I see your point, but it was quite hard already :-)) > > Just try it I think is the best! > > Regards Ard > > > > > Regards, > > > > Reinier > > > > Reinier van den Born > > HintTech B.V. > > > > T: +31(0)88 268 25 00 > > F: +31(0)88 268 25 01 > > M: +31(0)6 494 171 36 > > > > Delftechpark 37i | 2628 XJ Delft | The Netherlands > > www.hinttech.com<javascript:void('http://www.hinttech.com');> > > > > HintTech is a specialist in eBusiness Technology ( .Net, Java platform, > > Tridion ) and IT-Projects. > > Chamber of Commerce The Hague nr. 27242282 | Sales Tax nr. > NL8062.16.396.B01 > > > > > > On Thu, Jan 7, 2010 at 11:37 AM, Ard Schrijvers > > <[email protected]>wrote: > > > >> Hello, > >> > >> On Thu, Jan 7, 2010 at 10:51 AM, Reinier van den Born > >> <[email protected]> wrote: > >> > Hi all, > >> > > >> > We are using Hippo 6 with a Cocoon frontend. A few days ago our > frontend > >> > went down because it ran out of memory. > >> > Analysis pointed in the direction of the following: > >> > > >> > 1. There is a page based on quite a big document (thus memory > intensive) > >> > that is updated every couple of minutes or so. > >> > 2. The page gets a timeout of 30 seconds, so browsers request an > update > >> with > >> > that frequency. > >> > > >> > The problem occurred when we had a lot of visitors. At some point the > >> source > >> > document was updated and flushed from the cache. > >> > Many requests were coming in and as long as the page wasn't cached > each > >> > request resulted in an attempt to regenerate the page. > >> > The result was a load of threads, each of them trying to build the > page > >> from > >> > the big document and thus together eating up all memory. > >> > > >> > In a way it surprised us that neither Apache httpd's mod_cache or > >> Cocoon's > >> > caching seems to have some mechanism to prevent these redundant > parallel > >> > activities. > >> > >> I am not to surprised as it is really hard to have one general solution > >> > >> > > >> > Question is whether anyone can point us to a solution? > >> > > >> > Our own thoughts: > >> > > >> > First of all we need to assure that at most one thread renders the > page. > >> For > >> > the other threads two possibilities seem to be reasonable: > >> > 1. Serve the "outdated" page as long as the updated is not ready > >> > 2. Wait until the updated is available. > >> > The first solution seems preferable since it prevents the possibility > of > >> > having a load of "dangling" requests. > >> > > >> > However one of the solutions we are thinking of, is to create our own > >> Cocoon > >> > cache class, that only allows the first thread to build a specific > page > >> > and, using semaphores or so, has the others waiting for the first to > >> > finish. > >> > Obviously it is a solution type 2 but it is attractive because it > seems > >> > relatively easy to implement. > >> > We are basically hesitant to mess more than necessary with Cocoon's > >> caching > >> > mechanism. > >> > > >> > Any advice/ideas on this? > >> > >> I wouldn't try to solve it in cocoon's cache. Are you talking about > >> one single part that is the same for every request, that takes a lot > >> of time to build? > >> > >> I did add something (quite some time ago) like asynchronous fetching > >> of sources, where you could configure an expires after which an > >> asynchronous fetch was done, serving from cache until the asynchronous > >> background thread was finished, replacing the existing cached part. > >> Sounds like something you need isn't? It was though designed for > >> something like fetching an external rss feed, where you don't want to > >> wait for an external rss feed while the external site being down for > >> example. > >> > >> I am not totally sure it works for non repository sources, but you > >> might want to try this: > >> > >> Suppose your slow pipeline is: > >> > >> pattern="myslowpart" > >> > >> the place that calls this pipeline, should be change into: > >> > >> <map:generate > >> > src="cached:cocoon://myslowpart?cocoon:cache-expires=60&async=true"/> > >> > >> from the top of my head, it has to be a call to the root sitemap, this > >> cocoon://. > >> > >> Another way of course to achieve something similar, is create your own > >> generator, which stores the result on filesystem. You use this for all > >> request, and every now and then, recompute the result and store it > >> again > >> > >> Regards Ard > >> > >> > > >> > Thanks, > >> > > >> > Reinier > >> > ******************************************** > >> > Hippocms-dev: Hippo CMS development public mailinglist > >> > > >> > Searchable archives can be found at: > >> > MarkMail: http://hippocms-dev.markmail.org > >> > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > >> > > >> > > >> ******************************************** > >> Hippocms-dev: Hippo CMS development public mailinglist > >> > >> Searchable archives can be found at: > >> MarkMail: http://hippocms-dev.markmail.org > >> Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > >> > >> > > ******************************************** > > Hippocms-dev: Hippo CMS development public mailinglist > > > > Searchable archives can be found at: > > MarkMail: http://hippocms-dev.markmail.org > > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > > > > ******************************************** > Hippocms-dev: Hippo CMS development public mailinglist > > Searchable archives can be found at: > MarkMail: http://hippocms-dev.markmail.org > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > ******************************************** Hippocms-dev: Hippo CMS development public mailinglist Searchable archives can be found at: MarkMail: http://hippocms-dev.markmail.org Nabble: http://www.nabble.com/Hippo-CMS-f26633.html
