Hi Ard, Thanks for your reply.
Your suggestion looks quite like what we need, but I am not sure whether it will really work for us. You are talking about an external resource where the (async) regeneration is simply triggered when the cache has expired. In our case we are dealing with a repository resource, where cache invalidations are triggered by resource changes. So normally update events are invalidating all caches upto our final page. If this continues to happen the cache will be invalidated in between expires, so the problem would remain. Or will the cache-expires option protect the "slowpart" cache from those cache invalidations? If not, maybe it is possible to set up an intermediate pipeline to provide the necessary isolation? Some more general questions: I've been looking around a bit further and ran into the following page: http://wiki.onehippo.com/display/CMS/Using+asynchronic+get+for+cached+content If I am not mistaken this is using the async option on Cocoon's CachingSourceFactory. Is this related to what you mention or are these entirely different things? > Btw, if I understand the async option correctly, the regeneration is still initiated by an incoming request. Does that mean that if a request arrives first after an expire it still gets the old content served? In other words, when page traffic is sufficiently low, the site will always produce expired content? This shouldn't be a problem for the case we're experiencing our problems, but it would be something to take into consideration for different cases. Regards, Reinier Reinier van den Born HintTech B.V. T: +31(0)88 268 25 00 F: +31(0)88 268 25 01 M: +31(0)6 494 171 36 Delftechpark 37i | 2628 XJ Delft | The Netherlands www.hinttech.com<javascript:void('http://www.hinttech.com');> HintTech is a specialist in eBusiness Technology ( .Net, Java platform, Tridion ) and IT-Projects. Chamber of Commerce The Hague nr. 27242282 | Sales Tax nr. NL8062.16.396.B01 On Thu, Jan 7, 2010 at 11:37 AM, Ard Schrijvers <[email protected]>wrote: > Hello, > > On Thu, Jan 7, 2010 at 10:51 AM, Reinier van den Born > <[email protected]> wrote: > > Hi all, > > > > We are using Hippo 6 with a Cocoon frontend. A few days ago our frontend > > went down because it ran out of memory. > > Analysis pointed in the direction of the following: > > > > 1. There is a page based on quite a big document (thus memory intensive) > > that is updated every couple of minutes or so. > > 2. The page gets a timeout of 30 seconds, so browsers request an update > with > > that frequency. > > > > The problem occurred when we had a lot of visitors. At some point the > source > > document was updated and flushed from the cache. > > Many requests were coming in and as long as the page wasn't cached each > > request resulted in an attempt to regenerate the page. > > The result was a load of threads, each of them trying to build the page > from > > the big document and thus together eating up all memory. > > > > In a way it surprised us that neither Apache httpd's mod_cache or > Cocoon's > > caching seems to have some mechanism to prevent these redundant parallel > > activities. > > I am not to surprised as it is really hard to have one general solution > > > > > Question is whether anyone can point us to a solution? > > > > Our own thoughts: > > > > First of all we need to assure that at most one thread renders the page. > For > > the other threads two possibilities seem to be reasonable: > > 1. Serve the "outdated" page as long as the updated is not ready > > 2. Wait until the updated is available. > > The first solution seems preferable since it prevents the possibility of > > having a load of "dangling" requests. > > > > However one of the solutions we are thinking of, is to create our own > Cocoon > > cache class, that only allows the first thread to build a specific page > > and, using semaphores or so, has the others waiting for the first to > > finish. > > Obviously it is a solution type 2 but it is attractive because it seems > > relatively easy to implement. > > We are basically hesitant to mess more than necessary with Cocoon's > caching > > mechanism. > > > > Any advice/ideas on this? > > I wouldn't try to solve it in cocoon's cache. Are you talking about > one single part that is the same for every request, that takes a lot > of time to build? > > I did add something (quite some time ago) like asynchronous fetching > of sources, where you could configure an expires after which an > asynchronous fetch was done, serving from cache until the asynchronous > background thread was finished, replacing the existing cached part. > Sounds like something you need isn't? It was though designed for > something like fetching an external rss feed, where you don't want to > wait for an external rss feed while the external site being down for > example. > > I am not totally sure it works for non repository sources, but you > might want to try this: > > Suppose your slow pipeline is: > > pattern="myslowpart" > > the place that calls this pipeline, should be change into: > > <map:generate > src="cached:cocoon://myslowpart?cocoon:cache-expires=60&async=true"/> > > from the top of my head, it has to be a call to the root sitemap, this > cocoon://. > > Another way of course to achieve something similar, is create your own > generator, which stores the result on filesystem. You use this for all > request, and every now and then, recompute the result and store it > again > > Regards Ard > > > > > Thanks, > > > > Reinier > > ******************************************** > > Hippocms-dev: Hippo CMS development public mailinglist > > > > Searchable archives can be found at: > > MarkMail: http://hippocms-dev.markmail.org > > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > > > > ******************************************** > Hippocms-dev: Hippo CMS development public mailinglist > > Searchable archives can be found at: > MarkMail: http://hippocms-dev.markmail.org > Nabble: http://www.nabble.com/Hippo-CMS-f26633.html > > ******************************************** Hippocms-dev: Hippo CMS development public mailinglist Searchable archives can be found at: MarkMail: http://hippocms-dev.markmail.org Nabble: http://www.nabble.com/Hippo-CMS-f26633.html
