Re: lastModificationDate misuse? - old thread - progress
Carsten Ziegeler wrote: > The "S-xml-1_=NOP Validity" is appended by the xml serializer, so > in fact the cached entry with the "S-xml-1_=NOP Validity" is for > a complete pipeline with serializer. I'm still wondering > why this is the value for prev. For which internal pipeline > does this happen? Any/every pipeline that builds up the xsp. I created a very simple pipeline (see below) to test and I got the same problem, ie "S-xml-1_=NOP Validity" appearing in the prev. I cannot find the code appends the "S-xml-1_=NOP Validity" as follows: prev: {G-file--7461341140568259924_=TimeStamp Validity[1060277818000], T-xslt-2097470703455153761_=TimeStamp Validity[1062494248000], S-xml-1_=NOP Validity} curr: {G-file--7461341140568259924_=TimeStamp Validity[1060277818000], T-xslt-2097470703455153761_=TimeStamp Validity[1062494248000]} >From sitemap my serialiser is: org.apache.cocoon.serialization.XMLSerializer my serverpages generator is: org.apache.cocoon.generation.ServerPagesGenerator >From .xconf my event-pipeline class="org.apache.cocoon.components.pipeline.CachingEventPipeline
RE: lastModificationDate misuse? - old thread - progress
John Williams wrote: > > Carsten Ziegeler wrote: > > > Inspecting the contents of the currValidity and prevValidity > Maps I find > > > that they are equivalent excepting that the prevValidity contains an > entry > > > "S-xml-1_=NOP Validity" so the test fails and code is > > > re-generated. > > > > Huh! That's strange, now as far as I remember the code, the prevValidity > > should > > not contain "S-xml-1_=NOP Validity", as this is the validity for the > > serializer > > which is not used for internal pipeline calls. Hmm. > > I assume you mean src="cocoon://foo" calls and not internal-only > pipelines, > ie one with internal-only="true". Yes. > > I have a fairly deep pipeline underlying the generated XSP "file" > from which > .java is generated. Maybe this depth exposes a problem in discarding > NOPCacheValidity for internal pipeline calls. Can you see from > the following > which class is adding the "S-xml-1_=NOP Validity", I will then > have a poke > around/debug. > > > Below are some details of the case causing the problems: > > create-categories.xsp - which has the "S-xml-1_=NOP Validity" in the > prevValidity is: > { > aggregation of { > > aggregation of {tableDefs/categories.xml, > appDef/categories/UISpecifics.xml and appDef/terminologyMappings.xml} > transformed with stylesheets/buildUIStructure.xsl > and > appDef/categories/aCustomisation.xml > } > transformed with stylesheets/applyCustomisation.xsl > } > transformed with stylesheets/create.xsl > > This is what validities look like: (equal besides the "S-xml-1_=NOP > Validity") > > prev: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973], > T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp > Validity[106198276]:TimeStamp Validity[1061976654000]], S-xml-1_=NOP > Validity} > > curr: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973], > T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp > Validity[106198276]:TimeStamp Validity[1061976654000]]} > > Relevant part of the sitemap is as follows: > > > > > > > > > > > > > > > > > > > > > > > > > > The "S-xml-1_=NOP Validity" is appended by the xml serializer, so in fact the cached entry with the "S-xml-1_=NOP Validity" is for a complete pipeline with serializer. I'm still wondering why this is the value for prev. For which internal pipeline does this happen? Carsten
Re: lastModificationDate misuse? - old thread - progress
Carsten Ziegeler wrote: > > Inspecting the contents of the currValidity and prevValidity Maps I find > > that they are equivalent excepting that the prevValidity contains an entry > > "S-xml-1_=NOP Validity" so the test fails and code is > > re-generated. > > Huh! That's strange, now as far as I remember the code, the prevValidity > should > not contain "S-xml-1_=NOP Validity", as this is the validity for the > serializer > which is not used for internal pipeline calls. Hmm. I assume you mean src="cocoon://foo" calls and not internal-only pipelines, ie one with internal-only="true". I have a fairly deep pipeline underlying the generated XSP "file" from which .java is generated. Maybe this depth exposes a problem in discarding NOPCacheValidity for internal pipeline calls. Can you see from the following which class is adding the "S-xml-1_=NOP Validity", I will then have a poke around/debug. Below are some details of the case causing the problems: create-categories.xsp - which has the "S-xml-1_=NOP Validity" in the prevValidity is: { aggregation of { aggregation of {tableDefs/categories.xml, appDef/categories/UISpecifics.xml and appDef/terminologyMappings.xml} transformed with stylesheets/buildUIStructure.xsl and appDef/categories/aCustomisation.xml } transformed with stylesheets/applyCustomisation.xsl } transformed with stylesheets/create.xsl This is what validities look like: (equal besides the "S-xml-1_=NOP Validity") prev: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973], T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp Validity[106198276]:TimeStamp Validity[1061976654000]], S-xml-1_=NOP Validity} curr: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973], T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp Validity[106198276]:TimeStamp Validity[1061976654000]]} Relevant part of the sitemap is as follows:
RE: lastModificationDate misuse? - old thread - progress
John Williams wrote: > > Joerg Heinicke wrote > > Bug was fixed some days ago: > > http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten > > requested for testers of the patch, so if you can test the behaviour > > with the current Cocoon 2.0 CVS it will be good. > > > If XSP is the result of a pipeline then the ServerPages > Generator can't > > > work out its currency and randomly re-generates and compiles. To make > > > I have inspected the code of 2.0.4 and it is still there, ie code is > > > same as when reported in 2001. Surely a typical use of cocoon is to > > > build XSP using XSLT? > > I have tested (with cocoon-2.0_20030812221701) and believe the problem has > only been partially solved. Instead of the prior situation where > the .java & > .class were re-generated from XSP on a random basis - they now are always > regenerated. This is at least consistent - but very wasteful on > resources & > latency in requests. > > I have investigated the source and found that the test as to > whether or not > regeneration takes place is: >!currValidity.equals(prevValidity) > > Inspecting the contents of the currValidity and prevValidity Maps I find > that they are equivalent excepting that the prevValidity contains an entry > "S-xml-1_=NOP Validity" so the test fails and code is > re-generated. Huh! That's strange, now as far as I remember the code, the prevValidity should not contain "S-xml-1_=NOP Validity", as this is the validity for the serializer which is not used for internal pipeline calls. Hmm. So, I think we should investigate why this validity object is there instead of ignoring it later on. Carsten
Re: lastModificationDate misuse? - old thread - progress
Joerg Heinicke wrote > Bug was fixed some days ago: > http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten > requested for testers of the patch, so if you can test the behaviour > with the current Cocoon 2.0 CVS it will be good. > > If XSP is the result of a pipeline then the ServerPages Generator can't > > work out its currency and randomly re-generates and compiles. To make > > I have inspected the code of 2.0.4 and it is still there, ie code is > > same as when reported in 2001. Surely a typical use of cocoon is to > > build XSP using XSLT? I have tested (with cocoon-2.0_20030812221701) and believe the problem has only been partially solved. Instead of the prior situation where the .java & .class were re-generated from XSP on a random basis - they now are always regenerated. This is at least consistent - but very wasteful on resources & latency in requests. I have investigated the source and found that the test as to whether or not regeneration takes place is: !currValidity.equals(prevValidity) Inspecting the contents of the currValidity and prevValidity Maps I find that they are equivalent excepting that the prevValidity contains an entry "S-xml-1_=NOP Validity" so the test fails and code is re-generated. However, it appears that the "NOP Validity" is simply indicating an always valid condition - so code should not be re-generated. I have added some code to strip the NOPCacheValidity entries and do a comparison and my problem is solved - ie code is only regenerated when the cached objects are out-of-date. I have pasted the diff of my patch at the end of this message. I think the bug http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348 should be re-opened. Your views Joerg , Carsten? John Williams Suggested patch to SiteMapSource.java: @@ -56,12 +56,15 @@ import java.io.InputStream; import java.net.MalformedURLException; import java.util.Map; +import java.util.HashMap; +import java.util.Iterator; import org.apache.avalon.framework.component.ComponentException; import org.apache.avalon.framework.component.ComponentManager; import org.apache.cocoon.ProcessingException; import org.apache.cocoon.Processor; import org.apache.cocoon.caching.PipelineCacheKey; +import org.apache.cocoon.caching.NOPCacheValidity; import org.apache.cocoon.components.CocoonComponentManager; import org.apache.cocoon.components.EnvironmentStack; import org.apache.cocoon.components.pipeline.CacheableEventPipeline; @@ -322,7 +325,7 @@ Map currValidity = cep.generateValidity(this.environment); - if (prevValidity == null || !currValidity.equals(prevValidity)) { + if (prevValidity == null || !equalsIgnoreNOPCacheValidities(currValidity, prevValidity)) { // validity changed, update cached validity, timestamp and last modified this.lastModificationDate = System.currentTimeMillis(); obj = new Object[] {new Long (this.lastModificationDate), currValidity}; @@ -365,6 +368,29 @@ this.needsRefresh = false; } + static boolean equalsIgnoreNOPCacheValidities (Map aMapWithNOPValidities, Map anotherMapWithNOPValidities) { + return stripNOPValidities(aMapWithNOPValidities).toString().equals(stripNOPValiditi es(anotherMapWithNOPValidities).toString()); + } + + + static Map stripNOPValidities(Map withNOPValidities) { + if (withNOPValidities == null) return null; + else { + if (withNOPValidities.containsValue(NOPCacheValidity.CACHE_VALIDITY)) { + Map retVal = new HashMap(); + Iterator iter = withNOPValidities.entrySet().iterator(); + for (; iter.hasNext(); ) { + Map.Entry entry = (Map.Entry)iter.next(); + if (!entry.getValue().equals(NOPCacheValidity.CACHE_VALIDITY)) + retVal.put(entry.getKey(), entry.getValue()); + } + return retVal; + } + else return withNOPValidities; + } + } + + /** * Return a new InputSource object */
Re: lastModificationDate misuse? - old thread - progress
Bug was fixed some days ago: http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten requested for testers of the patch, so if you can test the behaviour with the current Cocoon 2.0 CVS it will be good. Joerg John Williams wrote: I have been mystified by the symptoms of the problem raised by Stuart Roebuck some time ago on cocoon-dev and I can't see if/when it has been fixed or a workaround/hack avail: If XSP is the result of a pipeline then the ServerPages Generator can't work out its currency and randomly re-generates and compiles. To make matters worse if the XSP has the code necessary for resultant objects to be cached then the code generated and compiled is not even used, ie a cached object is used instead. I have inspected the code of 2.0.4 and it is still there, ie code is same as when reported in 2001. Surely a typical use of cocoon is to build XSP using XSLT? John Stuart Roebuck wrote: >>On Tuesday, August 14, 2001, at 04:04 pm, Carsten Ziegeler wrote: >> >>> In SitemapSource.java are the lines: >>> // the event pipeline is cacheable // now calculate a last modification date String hashKey = pck.toString() + validity.toString(); this.lastModificationDate = HashUtil.hash(hashKey); >>> >>> this is used in ProgramGeneratorImpl.java: >>> /* * FIXME: It's the program (not the instance) that must * be queried for changes!!! */ if (programInstance != null && programInstance.modifiedSince(source.getLastModified())) { // Release the component. release(programInstance); >>> >>> consequently, any use of the ServerPages Generator which takes an input >>> from within the sitemap (e.g. using a "cocoon:/" URI) will result >>> in (on a >>> practically random basis) certain XSP (et. al) pages being regenerated >>> over and over again, and will potentially prevent updated XSP (et. al) >>> from being properly regenerated. >>> >> Hm, that is true. The problem is that I don't saw any other way to >> implement the lastModificationDate for the SitemapSource than hashing >> the key. >> Usually this modification date is used in the caching algorithm >> and is there only tested against equal. So there this hack works. >> >> Any solution for this? > >For the caching, I presume there is a mapping of hashKeys to >serializedResults. If this is true, then couldn't these also map to a >date-stamp of when the result was cached and have lastModificationDate = >lastCachedDate?
lastModificationDate misuse? - old thread - progress
I have been mystified by the symptoms of the problem raised by Stuart Roebuck some time ago on cocoon-dev and I can't see if/when it has been fixed or a workaround/hack avail: If XSP is the result of a pipeline then the ServerPages Generator can't work out its currency and randomly re-generates and compiles. To make matters worse if the XSP has the code necessary for resultant objects to be cached then the code generated and compiled is not even used, ie a cached object is used instead. I have inspected the code of 2.0.4 and it is still there, ie code is same as when reported in 2001. Surely a typical use of cocoon is to build XSP using XSLT? John Stuart Roebuck wrote:>>On Tuesday, August 14, 2001, at 04:04 pm, Carsten Ziegeler wrote:> In SitemapSource.java are the lines:>>> // the event pipeline is cacheable // now calculate a last modification date String hashKey = pck.toString() + validity.toString(); this.lastModificationDate = HashUtil.hash(hashKey);>> this is used in ProgramGeneratorImpl.java:>>> /* * FIXME: It's the program (not the instance) that must * be queried for changes!!! */ if (programInstance != null && programInstance.modifiedSince(source.getLastModified())) { // Release the component. release(programInstance);>> consequently, any use of the ServerPages Generator which takes an input>>> from within the sitemap (e.g. using a "cocoon:/" URI) will result>>> in (on a>>> practically random basis) certain XSP (et. al) pages being regenerated>>> over and over again, and will potentially prevent updated XSP (et. al)>>> from being properly regenerated.> Hm, that is true. The problem is that I don't saw any other way to>> implement the lastModificationDate for the SitemapSource than hashing>> the key.>> Usually this modification date is used in the caching algorithm>> and is there only tested against equal. So there this hack works. Any solution for this?>>For the caching, I presume there is a mapping of hashKeys to >serializedResults. If this is true, then couldn't these also map to a >date-stamp of when the result was cached and have lastModificationDate = >lastCachedDate?