Hiya, That's a very pertinent set of questions - I'll answer as best I can below, the only caveat is that I won't proclaim myself a JCS expert, given that I was really only introduced to it a few days ago ;)
-----Original Message----- From: Geoff Howard [mailto:[EMAIL PROTECTED] Sent: Saturday, 6 March 2004 2:38 p.m. To: [EMAIL PROTECTED] Subject: Re: Turning off default MRU store Corin Moss wrote: >Hi Guys, > >I think the thing to remember here is that it's the Disk Cache >implementation that has this behaviour, and that it's very easy to use >a different store type if the user feels it's necessary. There are >other, "safer" store implementations we can use - the only problem in >this case is that the index is kept in memory (which is what makes it >so fast.) If the user decides that this is inappropriate for them, the >change to a different store type is very easy to make within the >config. > >I think that moving to a filesystem store would be counter-productive, >as the thing that JCS provides is total flexibility of store type. > >Just my thoughts :) > > Ok, I apologize that I just can't look into this myself right now, but let me try to state what I perceive to be the things we really need so we can narrow down whether this is the right direction to be going: 1) - we need a high-performance in-memory cache - we need that cache to be backed by some sort of persistence for two reasons first, so that cached items aren't lost between server starts second so that items bumped off the memory cache can still be preserved as the assumption is that pulling a cached item off disk is still faster than regenerating the entire pipeline. for example, the most popular 100 items are served in memory, but the rest are on disk. (and at shutdown, the 100 in-memory ones go to disk). ** is JCS (properly configured) a good way to meet both of these requirements? CM: Yes. A JCS store can be configured to use an MRU store (the test config provided does just this.) This MRU store is in turn configurable - number of objects catered for, what type of store (memory etc.) The swapping to disk based store is handled in the same way as Cocoon currently does - objects are dropped out of MRU store into diskbased store. The only difference is that the "disk based" store can in fact be any of the several stores supported by JCS (remote server, disk, JISP etc.) At shutdown, things can be configured to store to disk (both store, and index.) The only issue is that in the event of an abnormal shutdown the index can be lost (this is only when the index is kept in memory of course - other store types which use pure disk based index storage wouldn't be effected by this.) -- 2) currently we accomplish both of those transparently with a transient-store configured to use a persistent-store. we have some subsystems which use a transient-store configured _not_ to use persistence because of application requirements. ** is JCS (properly configured) a good way to meet this requirement? CM: Yes. You could very easily configure a separate role for transient store which would use a different group / region (JCS terms.) This region would be configured to use _only_ memory store, with no support for persistence. This could be accomplished as is, by changing roles appropriately, and providing different config in the xconf for the transient store. -- 3) we have one subsystem which goes direct to the persistent-store right at shutdown which could just as easily go to the in-memory front-end as long as the persistence between restarts still happened. ** is JCS (properly configured) a good way to meet this requirement? CM: Yes. Again, this could either be written to the default store, which would go to MRU, and then disk at shutdown, _or_ a different group / region could be created and configured which would go straight to disk, without an in-memory MRU involved. -- I am not following the details well here about the persistence issue. If we don't use the Disk Cache, what other persistent (on disk, survives restarts) options are there? CM: As I mentioned before, the only issue which needs more investigation is abnormal shutdown with an in-memory index, however, this could theoretically be up to the user to decide - if their application requires an absolute guarantee of persistence at all times, they could configure it to use an on-disk index at all times - this would require no class change, just a change in the CCF. Hope that answers your questions properly :) I have a horrible feeling that my email client won't format this nicely ;) -- Geoff >-----Original Message----- >From: Unico Hommes [mailto:[EMAIL PROTECTED] > >Sent: Saturday, 6 March 2004 3:46 a.m. >To: [EMAIL PROTECTED] >Subject: Re: Turning off default MRU store > > >Geoff Howard wrote: > > > >>Unico Hommes wrote: >> >> >> >>>Sylvain Wallez wrote: >>> >>> >>> >>>>Unico Hommes wrote: >>>> >>>> >>>> >>>>>Unico Hommes wrote: >>>>> >>>>> >>>>> >>>>>>Corin Moss wrote: >>>>>> >>>>>> >>>>>> >>>>>>>Hi Guys, >>>>>>> >>>>>>>I might be getting ahead of myself a bit here, but I'm going to >>>>>>>try and turn off the default MRU store, in favour of the JCS >>>>>>> >>>>>>> > > > >>>>>>>based persistent store. I'd like to try some tests on >>>>>>> >>>>>>> > > > >>>>>>>performance without the default MRU - has anyone else tried >>>>>>> >>>>>>> > > > >>>>>>>anything similar? I've simply set the core store's role to point >>>>>>> >>>>>>> > > > >>>>>>>to the JCS store implementation. >>>>>>> >>>>>>> >>>>>>> >>>>>>I guess I already got ahead of you when I renamed >>>>>>JCSPersistentStore JCSStore just now :-) (And merged it with the >>>>>> >>>>>> > > > >>>>>>AbstractJCSStore BTW). It seems to me that JCS is both and it >>>>>> >>>>>> > > > >>>>>>could replace all three stores: DefaultStore, TransientStore and >>>>>> >>>>>> > > > >>>>>>PersistentStore. >>>>>> >>>>>> >>>>>> >>>>>I have to add that this is not the whole story. We do actually need >>>>>to distinguish between persistent and transient storage. Some >>>>> >>>>> > > > >>>>>components explicitly want to persist some data instead of just >>>>> >>>>> > > > >>>>>cache it for speed. But as far as caching is concerned I think it >>>>> >>>>> > > > >>>>>we can leave it completely up to JCS. >>>>> >>>>> >>>> >>>> >>>> >>>>Some components are explicitely using the transient-store to keep >>>>data that shouldn't (or cannot) be serialized. But AFAIK, no >>>> >>>> > > > >>>>component directly uses the persistent-store except the store >>>> >>>> >itself. > > >>>To my knowlegde there is one in eventcache block that uses the >>>PersistentStore to persist some info it needs to recover upon next >>> >>> > > > >>>startup. >>> >>>It does not look to me as though JCS would fit the PersistentStore >>>role very well. I was thinking that perhaps. We could have JCSStore >>> >>> > > > >>>as a replacement for TransientStore and Store roles and use >>> >>> > > > >>>FilesystemStore for the PersistentStore role. >>> >>> >> >>Wait a minute. The original issue that brought JCS to the front as >>that the persistent store was broken and had license problems. Are >> >> > > > >>you saying that JCS isn't a good candidate for persisting the cached >> >> > > > >>info to disk, or that the fact that it by default has an MRU memory >> >> > > > >>front-end makes it not map directly to persistent-store role cleanly? >> >> >> >IIUC, JCS will always use a memory store in front yes. And it suggests > >on the JCS website that the persistence process is not very reliable: > >from http://jakarta.apache.org/turbine/jcs/IndexedDiskAuxCache.html : > >"When the disk cache is properly shutdown, the memory index is written > >to disk and the value file is defragmented. When the cache starts up, > >the disk cache can be configured to read or delete the index file. This > >provides an unreliable persistence mechanism." > > > >>The use of persistent store in eventcache shouldn't be a problem with >>JCS as long as at shutdown, it persists everything it has in its MRU >> >> > > > >>if it hasn't already. If there is some problem it would be much >> >> > > > >>better to just go back to the old serializing persistence for the >> >> > > > >>event cache data because the only benefit to putting it in the store >> >> > > > >>is that you more or less guaranteed that the events and cached >> >> > > > >>responses would be kept together. >> >> > > >Yes, giving up that feature would be too bad. > >Unico > > ================================================================ CAUTION: This e-mail and any attachment(s) contains information that is intended to be read only by the named recipient(s). It may contain information that is confidential, proprietary or the subject of legal privilege. This information is not to be used by any other person and/or organisation. If you are not the intended recipient, please advise us immediately and delete this e-mail from your system. Do not use any information contained in it. ================================================================ For more information on the Television New Zealand Group, visit us online at http://www.tvnz.co.nz ================================================================
